Solving the LiveCode Engine

The number seven is interesting, and not just because there are no uninteresting numbers. For one reason or another the number has taken on a mystical quality in various cultures, not least in the office where it now surely holds the distinction of being the ‘most said number of the year’. Incidentally, we are not in general numerologically inclined, but hexakosioihexekontahexaphobes will be pleased to hear that there will probably not be another maintenance release after 6.6.5, despite what would have been the serendipitous proximity of Hallowe’en, the perfect date to unleash it.

There seems to be a certain memorable quality conferred on any set of seven things, and as such seven has become the de facto target for the quantity of a would-be interesting collection. Science, literature and life are littered with examples – Wonders of the Ancient World, colours of the rainbow, deadly sins, to name but a few. 

I am a fan of puzzles. These can be various sorts of puzzles, although the ones that really get my attention are the sort that are generally only found in a so-called “puzzle hunt”. To the uninitiated, these are strange variations on the traditional puzzle – they contain no instructions, and there is always an answer (usually a single word or phrase) as opposed to just a puzzle in a solved state. It turns out that sets of seven is a relatively common theme for these puzzles. Here are two examples (one from a puzzle hunt that is ongoing):

Li Ven Se Tt Le

Quadratum

The point is, for me working from start to finish on LiveCode 7 was in many ways similar to the process of solving a puzzle hunt. In fact, it probably ought to be similar in precisely seven ways, so lets see if I can shoehorn this section into a thematic format. I present a Buzzfeed style list:

Why working on LiveCode 7 is similar to solving a puzzle hunt. 

1. There is a way in

Since there are no instructions, the best puzzles tend to have an early hook to bring you into the world it inhabits. This can be any number of things, perhaps it has the presentation of a traditional puzzle such as a crossword, or perhaps it has some basic questions to answer, images to identify, facts to gather. Sometimes all you have is a pun in the title.

At the beginning of the refactoring project, my main task was to separate evaluation and execution in LiveCode syntax. Every function, command, operator, control structure, and constant in the language of LiveCode has a syntax class which is responsible for resolving expressions or performing actions in the appropriate way. We wanted the resolution of  subexpressions to be completely distinct from the execution. 

 In the first cases I encountered, this could be as easy as simply creating a new function in the engine, cutting a section of the syntax class implementation, and pasting it into the new function. Of course even in a simple case, one needs to read what is actually happening to make sure it is the correct thing to do. This was my way in; learning about LiveCode syntax and how it works; being brought into its world.

Here is an example of one of the more simple refactoring tasks. At the other extreme, refactoring the chunk object was completed with approximately 20 commits. 

2. It consists of a multitude of multi-stage processes

Another thing that distinguishes puzzle hunt style puzzles is the fact that they are generally multi-stage puzzles. There is rarely just one thing to do to get the answer. Sometimes it is not clear how many things will have to be done to complete the puzzle. The analogy here is clear – the many tasks necessary for the completion of the refactoring project all tended to break down into multiple subtasks. On a number of occasions, the details of the later stages of a task were not revealed to us until we had to do them. Of course, like some puzzles, this means that some stages of the project were fairly epic. Some lasted days, others weeks, still others months. That is to be expected, I suppose, when dealing with something on this scale. But the fact that the work would apparently have taken one person almost a decade to complete still boggles my mind. 

Usually a puzzle hunt will try and have a wide variety of puzzle types, testing many areas of skill: linguistic, logical, mathematical, artistic and musical puzzles abound. There is a certain polymathic requirement when dealing with the LiveCode engine too. While most of the engine is written in C++, parts of it are in Objective C, and parts of it are in Java, and when you are supporting many different platforms that means dealing with different APIs from all the operating systems. Just like a puzzle hunt, you have to be prepared to acquire a new skill just for one specific task. You often find yourself becoming an expert in rather strange areas. Like Roller coasters and knot theory. 

3. Organisation is paramount

The elements of the refactoring project that went the most smoothly were of course those for which we had been able to dedicate a good amount of time to planning and tracking. The trail of spreadsheets and documents in my Google Drive is longer than my virtual arm. This also relates in part to the previous point. The more subdivided the task, the easier the documentation of progress, and the better documented the progress, the more that progress is a boon to the project as a whole.

 I and the rest of my puzzle team have frequently found that a suitable organisation of the information gathered is sufficient to spark the next idea, or to unstick oneself. I’m trying to get the phrase “Sometimes it takes looking at different data on three laptop screens to work out how to solve a 3d maze and apply those directions to a Rubik’s cube (last page)” into general usage, but I’m not sure it will catch on.

Refactoring is in itself just a form of organisation. The codebase now is far more readable than it was before; which of course was one of the motivations for undertaking the project in the first place. It is important for people to be able to contribute to an open source project, and clearly the easier it is to understand what is already there, the easier it is to contribute. 

4. It involves looking at a lot of unfamiliar Unicode codepoints

And sometimes understanding them

And sometimes knowing how they are made up

And sometimes switching to different keyboard layouts

 Changing LiveCode’s internal string representation to accommodate Unicode has taught me untold amounts about the scripts used in various countries around the world. The integration of the text shaping library Harfbuzz (so that scripts such as Arabic, Khmer, and Devanagari look as they should on Android devices), amongst other things, resulted in an input source list that looks like this:

On one occasion, such keyboard switching had me completely locked out of my Ubuntu virtual machine, due to not being able to switch to English layout without logging in, but also not being able to actually type any of the characters in my password without switching to English layout. 

5. It requires concentration and attention to detail

Sometimes this is attention to small details in situations where there is not much information. Or in the case of this puzzle, extreme attention to detail (warning: attempting to solve the latter puzzle is a surefire route to madness). Other times it is a case of not trying to understand everything that you’re looking at, just the relevant parts. Like this. The aforementioned chunk object is so fundamental to LiveCode that even a slight mistake in refactoring it caused the LiveCode IDE equivalent of the blue screen of death (which happens to be a white rectangle with the “revLibURL” button on it).

 When replacing c-strings with MCStringRefs in the code, it was frequently the most efficient strategy to merely ensure that the behaviour was identical to what it was before, without completely understanding what the function was doing as a whole. 

6. It sometimes feels like it will never be finished

Getting stuck is a fact of life. Even more so a fact of puzzle hunts. Puzzles have a phase known as ‘extraction’, whereby the answer is obtained using all of the information that has been gathered and worked out. This can be as simple as using some numbers to index some words, or an acrostic that appears after sorting alphabetically. Other puzzles have more intricate extractions mechanisms, and some of the most beautiful involve iteration of procedures from previous parts of the puzzle. In any case, extraction can be very difficult, and it is therefore a very common place to get stuck.

Likewise, the period of time between being “basically” finished with LiveCode 7 and actually being finished felt like an eternity. I say actually finished, obviously there are some things to clean up, some residual bugs and so on. But we’re putting away the syntax caravan, and work is already underway on the widgets and themes project that will form the basis of LiveCode 8. In fact, I would like to get the phrase “putting away the syntax caravan” into general usage to mean almost finished. “How’s that novel coming along?” “Well, it’s not done, but I’m putting away the syntax caravan, if you know what I mean.” “No, I don’t know what you mean, Ali. Stop saying that.”

7. There are many ‘aha’s

The ‘aha’ or lightbulb moment is probably the reason why people who do these puzzles do them. It can be quite exhilarating, the joy of discovery, and is certainly quite addictive. One of the closest real-world experiences to the idealised puzzle ‘aha’ is solving a programming problem. There is a huge amount of satisfaction that comes from finishing off anything, particularly if it has involved an element of mystery, triumph over adversity, and especially if it is completely and utterly epic in scale. The refactoring project ticks all these boxes as far as I’m concerned. And there were ‘aha’ moments in spades. 

Caveat: Why working on LiveCode 7 is not at all similar to solving a puzzle hunt

I mentioned before that there is always an answer to a puzzle hunt puzzle. There is also usually a ‘metapuzzle’, which involves using all the answers to all of the individual puzzles in the hunt. Well, here’s where the analogy completely breaks down. There is not single phrase answer to the LiveCode engine. Like a more traditional puzzle, the best we can say is that it is in a more ‘solved’ state than it was when we started. But that is still deeply satisfying, and a goal I’m happy to have spent my first two years working towards.

Solutions:

Quadratum

Please Remain Seated

Contortionist

English Expectations

Cryptograf

Symphony from the Old World

Circulus In Probando

Team Statistics

and finally

The LiveCode Engine