Testing 1, 2, 28534

I really enjoy writing tests. It’s great to ensure that a new piece of functionality will never be broken, or that a bug will never be reintroduced. It’s doubly satisfying when something previously untestable becomes testable, when a certain level of struggle or ingenuity is required to work out how to test something, or when a test is particularly (perhaps even maximally) succinct.

LiveCode’s continuous integration had its 2nd birthday recently, and ever since a ‘quick’ test runner was implemented for this purpose, the number of tests and the test infrastructure itself has ballooned.

To celebrate this birthday, in this blog post I’ll be describing all the different sorts of test we run, and picking out a few of my favourite examples.

Engine tests

The engine test runner uses the standalone engine to execute each test handler defined in the test scripts in its own subprocess. These tests – at least those in the ‘core’ section – are intended to ensure that every single variant of syntax, each of which should have received a distinct code-path in the syntax refactoring project, has a test of its functionality. Ideally, every execution and parse error would also have a test.

For example, a simple engine test might be

on TestCreateStack
    create stack "test"
    TestAssert "Stack created", there is a stack "test"
end TestCreateStack

The TestAssert handler takes two arguments,

TestAssert pDescription, pExpectTrue

If the expression passed in pExpectTrue is true, the test with description pDescription passes. If not, it fails.

Cyclic behavior error

on _TestCyclicBehavior pStack
    set the behavior of (the behavior of pStack) to pStack
end _TestCyclicBehavior

on TestCyclicBehavior
    create stack "Behavior"
    create stack
    set the behavior of it to the long id of stack "Behavior"
    TestAssertThrow "cycle in behavior script hierarchy throws", \
        "_TestCyclicBehavior", the long id of me, \
        "EE_PARENTSCRIPT_CYCLICOBJECT", it
end TestCyclicBehavior

What does it test?

This test ensures that it is an error if there is a cycle in the behavior hierarchy.

How does it test?

The TestAssertThrow handler is a test utility that allows you to test that the expected error is thrown when a given handler is executed. The TestAssertThrow handler has parameters as follows:

TestAssertThrow pDescription, pHandlerName, pTarget, pExpectedError, pParam

Its function could be described as: execute handler pHandlerName which is implemented in the script of pTarget, (with an optional parameter pParam); pass the test with description pDescription if the error with code pExpectedError is thrown as a result.

The error in this case is “parentScript: loop in hierarchy”, which you can find by checking the execution errors source.

Why do I like it?

I must admit, my enjoyment of this test is almost exclusively bound to the simplicity of the error-throwing line of code. It’s neat.

Why is it important?

This needs to be an error because otherwise there would be an infinite loop when trying to find a handler in one of the objects’ message paths.

Other good tests:

• The rather complex-looking char tests which mirror the Unicode grapheme cluster boundary rules
• More test cases that ∞ is ∞ than you probably thought possible and/or necessary

IDE tests

In stark contrast to the engine tests, the IDE tests are not at all self-contained. They test things like the high-level IDE API, and functionality of stacks within the IDE.

revXML (and other external) inclusions

What does it test?

This set of tests ensures that apps built with standalone builder can load the externals that are selected (or detected) for inclusion.

How does it test?

The core of this test is creating a standalone application with the following script:

on testInclusion
    get revXMLTrees()
    if the result is empty then
        quit 0
    else
        write the result to stderr
        quit 1
    end if
end testInclusion

on startup
    try
        testInclusion
    catch tError
        write "inclusion not loaded" to stderr
        quit 1
    end try
end startup

The ‘search for inclusions’ feature should see the XML-related function in the stack script when building the standalone and include the revXML external accordingly. Then the test executes the newly built standalone and checks that the exit code is 0. Quitting with exit code 1 if the result is not empty ensures that if revXML external had internal errors when calling revXMLTrees, the test fails. Similarly, the try loop ensures that if revXMLTrees doesn’t exist (because the revXML external is missing), the standalone quits with exit code 1.

Why do I like it?

I enjoy the fact that the standalone engine is running the development engine as a subprocess, which in turn runs the standalone engine as a subprocess. However this is a quirk of how the IDE tests are run, so I’ll also say I like it for the same reason it’s so important.

Why is it important?

Loading dynamic libraries can be tricky, especially where multiple platforms are involved.

• If multiple versions of a library are present on a system, various things like environment variables can affect which version is found and loaded.
• On Mac, both shared libraries and bundles can be loaded dynamically, but in a slightly different way
• Sometimes they can have special requirements (such as location or existence of other libraries or code resources)

Other good tests:

• An ever-growing set of tests for the intricate message box autocomplete features
• Tests for the correspondence between color names and RGB values in the IDE

LCB Standard Library Tests

The core LCB tests are essentially to LiveCode Builder as the engine tests are to LiveCode Script. They use the lc-run program, which can execute LCB bytecode files directly, to run the tests.

List tests

Eg ‘the number of elements in’ test:

public handler TestCount()
    test "count" when the number of elements in [1, 2, ["a", "b"]] is 3
end handler

I didn’t want to pick out an individual test here, just wanted to highlight the List tests in their entirety. TestCount is included as an illustration of the test syntax in LCB.

What do they test?

The clue is in the name – they test various aspects of the syntax associated with Lists.

How do they test?

Most of these tests create a specific list, perform an operation on them and use the LCB test syntax to verify the expected result.

Why do I like them?

I like these tests because I like the LCB List type very much. There are several aspects of string / delimiter based lists that LiveCode Script uses which can be confusing, or appear anomalous, or actually are anomalous. The List type is free of these complications, and more efficient than a string list under the hood.

Why is it important?

The List type is used in almost every widget, library and support module that has so far been implemented!

LCB Virtual Machine Tests

These tests ensure that the fundamental parts of LCB (i.e. the control structures, function calling, etc) which are essentially hard-coded in LCB (as opposed to the modular, plugged-in standard library) work correctly.

Thunks

handler type Thunk()

handler Yes()
    return true
end handler

handler InvokeArgument(in pFunc as Thunk)
    return pFunc()
end handler

public handler TestDynamicInvokeVariable()
    variable pFunc as Thunk
    put Yes into pFunc
    test "dynamic invoke (variable)" when pFunc()
end handler

public handler TestDynamicInvokeArgument()
    test "dynamic invoke (argument)" when InvokeArgument(Yes)
end handler

What does it test?

Invocation of handlers that are stored in variables or passed as arguments to functions.

How does it test?

Handler types can be declared in LCB, and handlers which conform to the type can be put into variables of that type. A handler conforms to a handler type if its parameter and return value types match.

In this case, the handler type Thunk is defined with no parameters and the default return type (optional any), so the Yes handler obviously conforms to the handler type. The Yes handler is passed directly to InvokeArgument, and also put into a variable and passed to InvokeArgument.

Why do I like it?

Partly because the word ‘Thunk’ is great.

Why is it important?

The ability to pass handlers as arguments is one of the most powerful features of LiveCode Builder. One of the most common use cases is for recursively performing some kind of action on an array – there are several examples of this in the Tree View widget.

LCB Compiler Tests

LCB Compiler tests ensure that the lc-compile program, which parses LCB modules, either succeeds when it’s supposed to, or outputs the correct error message, indicating the correct part of the offending line of code.

Line Continuation tests

%TEST ContinuationPreComment
module %{CONTINUATION_BEFORE_COMMENT} \ -- test
    compiler_test
end module
%EXPECT PASS
%ERROR "Illegal token ''" AT CONTINUATION_BEFORE_COMMENT
%ENDTEST

What does it test?

This test ensures that a line continuation character consumes a following newline, and backslash followed by a comment isn’t a line continuation.

How does it test?

The %{MARKER} syntax is used in compiler tests to specify the expected location of a syntax error. So this test simply consists of a module script which would only succeed in compiling if –test were considered a line continuation.

Why do I like it?

I like this test because we believe this is the correct and expected behavior when putting a comment after a line continuation character, but discovered that such a construction has always been valid in LiveCode Script. It’s a neat illustration of the situation.

Why is it important?

Where a decision is made to depart from some of the anomalies of LiveCode Script, it’s doubly important that tests exist for the situation at hand in order to indicate that the departure is not accidental.

Other good tests:

• Tests that ensure the LCB namespace operator ‘.’ works as expected

LCS Parser Tests

LCS Parser tests ensure that LiveCode Script snippets which should compile without error do so, and also that correct error messages are thrown for particular syntax errors.

Object property parsing test

%TEST GetTheObjectPropertyNoObject
on parse_test
 get the width %{AFTER_PROPERTY}
end parse_test
%EXPECT PASS
%ERROR PE_PROPERTY_NOTOF AT AFTER_PROPERTY
%ENDTEST

What does it test?

This test ensures that properties which must have an object target cause a parser error when the object target is missing.

How does it test?

By ensuring that PE_PROPERTY_NOTOF is thrown for the line get the width, as width is not a global property.

Why do I like it?

I like this test because it’s so much neater than the corresponding test in develop-8.1 where the parser test runner does not exist.

Why is it important?

The message box needs to know whether a property is global or not in order to do its intelligence object autocompletion. With this parser error in place, the message box knows not to execute get the width by itself, but instead autocompletes to get the width of stack "Untitled 1" or whatever the currently chose target object is.

Other good tests:

• Line continuation tests which are the LCS counterpart to the LCB compiler line continuation tests mentioned above

C++ tests

C++ tests are (unsurprisingly) written in C++, and are only written for things that are too low-level to test any other way. These tests use the Google test framework and live in a subfolder of the main folder of source code in the engine repository.

Lextable ordering

TEST(lextable, table_pointer)
{
    extern LT *table_pointers[];
    extern const uint4 table_pointers_size; 
 extern uint2 table_sizes[]; 
 extern const uint4 table_sizes_size; 
 ASSERT_EQ(table_pointers_size, table_sizes_size); 
 
 for (uint4 i = 0; i < table_pointers_size; i++) 
 { 
 LT* table = table_pointers[i]; 
 const uint4 table_size = table_sizes[i]; 
 ASSERT_GE(table_size, (unsigned)1); 
 for (uint4 j = 0; j < table_size - 1; j++) 
 { 
 EXPECT_LT(strcmp(table[j].token, table[j+1].token), 0) 
 << "\"" << table[j+1].token << "\"" 
 << " comes before " << "\"" 
 << table[j].token << "\""; 
 }
 } 
}

What does it test?

That the tokens that make up LiveCode script syntax are listed in alphabetical order.

How does it test?

By iterating through the list and ensuring each entry comes before the following entry in the table.

Why do I like it?

See ‘Why is it important?’

Why is it important?

• An annoying bug
• Another annoying bug

Note my irritation at this sort of bug expressed in the latter report.

Syntax tokens are mapped to numeric constants using a binary search of these tables. The best case scenario with a misplaced token is that only the token itself is not found. But if the misplaced token happens to be used in a step of the binary search to decide on the next part of the table to look at, it could cause syntax errors for a whole swathe of tokens.

Other good tests:

Well, there are only three more

• One ensures that the rgb values are in the correct order.
• Another ensures the engine doesn’t crash when memory allocation via the new operator fails.
• Finally one ensures that the object type table is complete and matches the order of defined object chunk types.

Extension Tests

Extension tests are written in LCB or LCS depending on the type of extension. There are currently four types of extension:

• LCS libraries
• Widgets
• LCB Libraries
• LCB Modules

LCS library tests and widget tests are written in LCS, since these both
depend on the engine for their functionality. LCB modules are not directly accessible through LCS, so their tests are written in LCB. LCB library tests can be written in either LCS or LCB (although currently the latter is restricted by the fact that the library must have no module dependencies)

JSON library test suite

What does it test?

That the LCB JSON library can handle a large range of incoming json data correctly

How does it test?

The test loads a series of .json files which are either intended to parse successfully, or fail, and ensures they do just that.

Why do I like it?

The sheer extent of the tests and number of edge cases gives this test suite a certain charm.

Why is it important?

As JSON data is likely to come from outside sources, it is vitally important that the JsonImport handler agrees with the rest of the world on what is valid or not.

Other good tests:

• Script items module, string items tests
• Command line option parsing script library tests

Docs Tests

This is not strictly a subcategory, in that the docs tests are run as part of the engine test suite. But it’s worth mentioning them anyway! At some point they could be run separately.

Dictionary validation

What does it test?

This set of tests ensures that the .lcdoc dictionary files are correctly formatted, that the example scripts compile, and that the syntax specification is correct.

How does it test?

This test use the functions defined in the revDocsParser library to obtain the data from the dictionary in the form of a LiveCode array. It then performs a battery of checks on various aspects of that array.

Why do I like it?

Because it performs almost 30,000 tests on the docs.

Why is it important?

Misleading documentation is bad, for obvious reasons.

Broken Tests

No, this isn’t a perverse subcategory of tests, but a note on an important feature of the tester API. It is possible to assert that a particular feature is broken. For example, there is currently a bug with the behavior of the hilitedButtonName property, and there are tests in place which assert the broken-ness of that particular test case.

The command is

TestAssertBroken <description of test>, <broken test case>, <reason broken>

This is just like the usual TestAssert command, except that the test is expected to fail and a reason must be provided – usually the associated bug number. So if you’re feeling adventurous, submit a broken test next time you report a bug.

So there you have it, an extremely subjective run down of the various types of tests we run on every single pull request to ensure it won’t cause problems when merged. We now ask for tests to be added to code changes whenever it is possible. There is no reason why tests can’t be added on their own however, so if you spot a gap in our coverage please consider submitting a test for it! This is potentially one of the simplest and best ways to contribute to LiveCode, and also getting you Hacktoberfest t-shirt if you contribute this month.