Integrating Lua unit tests in CxxTest

I'm currently working on a project that uses both C++ and Lua. In the beginning I looked at several options to do unit tests in C++ and ended up choosing CxxTest. I've managed to integrate it well in my cmake build script and all the tests run with a simple "make test".
Because the Lua code calls the C++ code I started coding it later and only now I'm writing unit tests for it (yeah, I'm not a TDD fan). There are several good options for unit testing in Lua (see lua wiki) but there are two reasons why I didn't use them. The first one is that I wanted to use only one testing framework to have unified testing statistics. The other reason is that they lack a certain features CxxTest has. The thing I like about CxxTest is that its test macros also capture the source text. For example if I write:

int i = 3;
TS_ASSERT_EQUALS(i, 5);
TS_ASSERT(i == 5);

I get:

test.cpp:29: Error: Expected (i == 5), found (3 != 5)
test.cpp:30: Error: Assertion failed: i == 5

When you look at that output you already know what variables are wrong even before opening the code in your text editor. It would be nice to have this feature in Lua as well.
Of course this only work in CxxTest because it uses the argument to string conversion feature of C macros. Now, Lua has no macros but it has a much more powerful feature which is the dynamic loading of code from strings (Actually there is a macro extension for Lua). Its impossible to write

i = 3
TS_ASSERT(i == 5)

But it should be possible to write:

i = 3
TS_ASSERT[[i == 5]]

Our first try is

function TS_ASSERT(code)
    local func = assert(loadstring("return "..code))
    if not func() then
       local info = debug.getinfo(2, "Sl");
       local msg = {"Failed assertion: ", code}
       TS_FAIL(info.short_src, info.currentline, table.concat(msg))
    end
end

This function tries to compile a string, containing an expression, passed as argument and checks if the result evaluates to true or false according to Lua's rules. If the result is true the test passed and life goes on. If not, we find out the source location where our function was called and pass it on to TS_FAIL together with the expression that failed. For now, all we need to know about TS_FAIL is that it calls CxxTest to account for this failure.
The output for the sample above would be:

test.lua:29: Error: Failed assertion: i == 5

This is very nice but we have no clue about the actual value of i. And there is another problem: it only works with global variables because the code compiled with loadstring has no lexical scoping. If we had written

local i = 3
TS_ASSERT[[i == 5]]

the test would always fail because the undefined global variable i would be nil.
Let't turn our attention to the first problem first: we need to find out the values of variables inside a string of code. We could print all the values of _G, the global variable table, but this would clutter the output confusing the programmer instead of helping him. An extreme solution would be to parse the code ourselves to find out the names of the variables and lookup their values in _G to print them. A more practical solution is to use environments and meta-tables. In Lua we can set a special environment for functions. At the same time we can use the meta-method __index to track the access to keys that are not present. With a specially crafted environment table we can track all the variables acesses of a funtion. Adding these ideas our second attempt looks like this:

function TS_ASSERT(code)

    local func = assert(loadstring("return "..code))

    local calling_globals = getfenv(2)
    local referenced = {}
    local env = {}

    setmetatable(env, {__index=
        function(table, name)
            local value = calling_globals[name] end
            referenced[name] = value
            return value
        end
    })

    setfenv(func, env)
    if not func() then
       local info = debug.getinfo(2, "Sl");
       local msg = {"Failed assertion: ", code}

       local first = true
       for k, v in pairs(referenced) do
           if first then
               msg[#msg+1]= ". Variables: "
           else
               msg[#msg+1] = ", "
           end
           msg[#msg+1]= tostring(k).." = "..tostring(v)
           first = false
       end
       TS_FAIL(info.short_src, info.currentline, table.concat(msg))
    end
end

First we get the environment of the calling function. Then we give our newly compiled function an empty environment with an __index meta-method that will forward all variable lookups to the environment of the calling function. So now the output for the sample above would be:

test.lua:29: Error: Failed assertion: i == 5. Variables: i = 3

Much better, but it still won't work for local variables. Unlike globals, local variables are not stored in a table, possibly for performance reasons. For this we must, again, resort to the debug library. The function debug.getlocal an index as argument and returns the name of the corresponding local variable, followed by its value. All we have to do is to copy these values to our fake envirnonment:

...
    local calling_globals = getfenv(2)
    local calling_locals  = {}
    local i = 1
    while true do
        local name, value = debug.getlocal(2, i)
        if not name then break end
        calling_locals[name] = value
        i = i + 1
    end

    local notpresent = {}
    setmetatable(calling_locals, { __index=function() return notpresent end})

    local referenced = {}
    local env = {}

    setmetatable(env, {__index=
        function(table, name)
            local value = calling_locals[name]
            if value == notpresent then value = calling_globals[name] end
            referenced[name] = value
            return value
        end
    })
...

We collect the local variables in a table and we make the intercepting function look in this table before searching in the global environment, respecting the Lua's rules for variable lookup. The only complication is that a table lookup returns nil both if the entry doesn't exists or its value is really nil. To make the distinction we use the empty table "notpresent". The call to getlocal might be slow but we don't want to make the fastest unit test in the world but the most helpful.
As a last addition we can make the following enhancement to TS_ASSERT: if the argument is not a string we simply see if it evaluates to false.

...
    if (type(code) ~= "string") then
        if not code then
            local info = debug.getinfo(2, "Sl");
            local msg = {"Failed assertion"}
            TS_FAIL(info.short_src, info.currentline, table.concat(msg))
        end
        return
    end
...

Now we can also write:

local file = io.open("/dev/null", "w")
TS_ASSERT(file)

This assertion will fail if the io.open call is unsuccessful.
The last thing left to explain is the TS_FAIL function. We have used it passing a custom source location but it would also be nice to use it like the C++ macro to fail a test:

local file, msg = io.open("/dev/null", "w")
if not file then TS_FAIL(msg) end

All we have to do is to write a small binding for the CxxTest API function that the macro calls internally:

int ts_fail(lua_State* L) {
    const int nargs = lua_gettop(L);

    if (nargs == 3) {
        CxxTest::TestTracker::tracker().failedTest(
            luaL_checkstring(L, -3),
            luaL_checkint(L, -2),
            luaL_checkstring(L, -1) );
    } else if (nargs >= 1) {
        lua_Debug debug;
        lua_getstack(L, 1, &debug);
        lua_getinfo(L, "Sl", &debug);
        CxxTest::TestTracker::tracker().failedTest(
            debug.short_src,
            debug.currentline,
            luaL_checkstring(L, -1) );
    } else {
        luaL_error(L, "TS_FAIL called with an illegal number of arguments");
    }
    return 0;
}

If this function is called with three arguments we assume that the first two are the location and the second the message. If only one argument is supplied, we assume that it is the error message and find out the calling location using the C API of the debug library. In both cases the location and the message are passed to the failedTest method of CxxTest's TestTracker singleton.
It's arguable if the name of the variables in the testing output will help you to fix you errors faster, but I think it is interesting to see that it can be done. There are many more features we can add to this framework. One, for example, is measuring the code coverage of the tested code with Lua's debug library. But for now this will already help me a lot to write unit tests in Lua.