Testing And Building LiveCode Builder With Tests

We recently launched the first developer preview (DP) release of LiveCode 8, which provides a first look at LiveCode’s new “LiveCode Builder” language. This is a powerful new compiled language which can be used to quickly and efficiently develop new controls and code libraries for use in LiveCode apps.

I’d like to tell you about some of the techniques we’ve been using to make sure that LiveCode Builder is robust and reliable.

Rigorous Testing

During the LiveCode Builder development process, we’ve had several members of the Engine Development Team working simultaneously on different aspects of the LiveCode Builder platform. We’ve had people developing new widgets using LiveCode Builder, but at the same time major changes and improvements to the LiveCode Builder core library and compiler itself have been taking place.

To help us ensure that compiler changes don’t break the core library, and that library changes don’t break our in-house widgets, we have been doing a lot of testing. This is with a “unit testing” approach: each unit test deals with a very small, clearly-defined element of the core library. At the moment we have 556 individual unit tests.

One of the things that’s made the unit tests so effective is that they are written in LiveCode Builder — and even the test harness (the program that runs each of the tests and checks their output) is written in LiveCode Builder, too. Since the test harness is a fairly non-trivial program which makes use of some of LiveCode Builder’s more advanced capabilities, it gives the compiler and runtime environment a pretty good workout and lets us spot issues early.

We’ve even added some syntax to the language to make writing unit tests easier. Here’s a snippet of some unit tests for operations on “List” values:

public handler TestPush()
	variable tList
	put ["x", 1, true] into tList

	push "y" onto tList
	test "push" when the tail of tList is "y"

	push "z" onto front of tList
	test "push front" when the head of tList is "z"

	push "w" onto back of tList
	test "push back" when the tail of tList is "w"

	test "push (result)" when tList is ["z", "x", 1, true, "y", "w"]
end handler

The “test” syntax lets us quickly and efficiently associate test descriptions (e.g. “”empty (empty list)””) with conditions (e.g. “tList is empty”). Each “test” generates a line of log output in Test Anything Protocol format. For example, part of the output of the “List” tests looks like this:

### stdlib/list TestEmpty 

ok - empty (empty)
ok - empty (empty list)
ok - empty (empty literal)
ok - empty (no elements)

### stdlib/list TestPush 

ok - push
ok - push front
ok - push back
ok - push (result)

The test harness parses all of the log output from all of the tests to generate a summary. For example, as of our latest build the summary looks like:

================================================================
All 556 tests behaved as expected
	10 expected failures
	1 skipped
================================================================

We have the ability to mark tests as expected to fail by writing “broken test” instead of “test”. For example, the 10 expected failures that we currently report are because LiveCode Builder isn’t yet able to tell the difference between integers and real numbers. We can also “skip test” — this is used for tests that are only relevant on specific platforms.

Continuous Integration

Tests are only any good if you run them — so over the last couple of months we’ve been using the Travis continuous integration service to build LiveCode and run all of the LiveCode Builder unit tests for every single change that we make. This has been absolutely invaluable for catching problems before they occur.

At the moment, we only do continuous integration testing for the x86-64 Linux platform. However, it’s been so successful that we’re soon going to move to an in-house solution that’ll let us add Windows and Android support, followed shortly by support for OS X and iOS.

Building on tests

Unit tests have been very useful for us, so we want to bring the benefits of unit testing to everyone writing LiveCode Builder extensions — both widgets and libraries. We currently plan to add easy-to-use unit testing functions to the IDE’s extension builder that’ll use exactly the same “test” syntax that we use for the core library unit tests.

Adding unit tests and running them as part of our continuous integration testing has made a really big impact on the quality of LiveCode Builder and the LiveCode 8 DP 1 release. This is just one of a number of measures we’re taking to further improve our quality assurance processes. I’m sure you’ll be hearing about some of the other steps we’re taking in the future.