Code Health Warning: This article contains Really Important Information for your code. Especially if you are not yet using LiveCode 8.

The number seven is interesting, and not just because there are no uninteresting numbers. For one reason or another the number has taken on a mystical quality in various cultures, not least in the office where it now surely holds the distinction of being the ‘most said number of the year’. Incidentally, we are not in general numerologically inclined, but hexakosioihexekontahexaphobes will be pleased to hear that there will probably not be another maintenance release after 6.6.5, despite what would have been the serendipitous proximity of Hallowe’en, the perfect date to unleash it.

There seems to be a certain memorable quality conferred on any set of seven things, and as such seven has become the de facto target for the quantity of a would-be interesting collection. Science, literature and life are littered with examples – Wonders of the Ancient World, colours of the rainbow, deadly sins, to name but a few. 

I am a fan of puzzles. These can be various sorts of puzzles, although the ones that really get my attention are the sort that are generally only found in a so-called “puzzle hunt”. To the uninitiated, these are strange variations on the traditional puzzle – they contain no instructions, and there is always an answer (usually a single word or phrase) as opposed to just a puzzle in a solved state. It turns out that sets of seven is a relatively common theme for these puzzles. Here are two examples (one from a puzzle hunt that is ongoing):

Li Ven Se Tt Le

Quadratum

The point is, for me working from start to finish on LiveCode 7 was in many ways similar to the process of solving a puzzle hunt. In fact, it probably ought to be similar in precisely seven ways, so lets see if I can shoehorn this section into a thematic format. I present a Buzzfeed style list:

Why working on LiveCode 7 is similar to solving a puzzle hunt. 

1. There is a way in

Since there are no instructions, the best puzzles tend to have an early hook to bring you into the world it inhabits. This can be any number of things, perhaps it has the presentation of a traditional puzzle such as a crossword, or perhaps it has some basic questions to answer, images to identify, facts to gather. Sometimes all you have is a pun in the title.

At the beginning of the refactoring project, my main task was to separate evaluation and execution in LiveCode syntax. Every function, command, operator, control structure, and constant in the language of LiveCode has a syntax class which is responsible for resolving expressions or performing actions in the appropriate way. We wanted the resolution of  subexpressions to be completely distinct from the execution. 

 In the first cases I encountered, this could be as easy as simply creating a new function in the engine, cutting a section of the syntax class implementation, and pasting it into the new function. Of course even in a simple case, one needs to read what is actually happening to make sure it is the correct thing to do. This was my way in; learning about LiveCode syntax and how it works; being brought into its world.

Here is an example of one of the more simple refactoring tasks. At the other extreme, refactoring the chunk object was completed with approximately 20 commits. 

2. It consists of a multitude of multi-stage processes

Another thing that distinguishes puzzle hunt style puzzles is the fact that they are generally multi-stage puzzles. There is rarely just one thing to do to get the answer. Sometimes it is not clear how many things will have to be done to complete the puzzle. The analogy here is clear – the many tasks necessary for the completion of the refactoring project all tended to break down into multiple subtasks. On a number of occasions, the details of the later stages of a task were not revealed to us until we had to do them. Of course, like some puzzles, this means that some stages of the project were fairly epic. Some lasted days, others weeks, still others months. That is to be expected, I suppose, when dealing with something on this scale. But the fact that the work would apparently have taken one person almost a decade to complete still boggles my mind. 

Usually a puzzle hunt will try and have a wide variety of puzzle types, testing many areas of skill: linguistic, logical, mathematical, artistic and musical puzzles abound. There is a certain polymathic requirement when dealing with the LiveCode engine too. While most of the engine is written in C++, parts of it are in Objective C, and parts of it are in Java, and when you are supporting many different platforms that means dealing with different APIs from all the operating systems. Just like a puzzle hunt, you have to be prepared to acquire a new skill just for one specific task. You often find yourself becoming an expert in rather strange areas. Like Roller coasters and knot theory. 

3. Organisation is paramount

The elements of the refactoring project that went the most smoothly were of course those for which we had been able to dedicate a good amount of time to planning and tracking. The trail of spreadsheets and documents in my Google Drive is longer than my virtual arm. This also relates in part to the previous point. The more subdivided the task, the easier the documentation of progress, and the better documented the progress, the more that progress is a boon to the project as a whole.

I found this particular progress spreadsheet irresistible in terms of trying to get to 100%

 I and the rest of my puzzle team have frequently found that a suitable organisation of the information gathered is sufficient to spark the next idea, or to unstick oneself. I’m trying to get the phrase “Sometimes it takes looking at different data on three laptop screens to work out how to solve a 3d maze and apply those directions to a Rubik’s cube (last page)” into general usage, but I’m not sure it will catch on.

 

Disclaimer: I did not actually solve this puzzle. 

Refactoring is in itself just a form of organisation. The codebase now is far more readable than it was before; which of course was one of the motivations for undertaking the project in the first place. It is important for people to be able to contribute to an open source project, and clearly the easier it is to understand what is already there, the easier it is to contribute. 

4. It involves looking at a lot of unfamiliar Unicode codepoints

And sometimes understanding them

And sometimes knowing how they are made up

And sometimes switching to different keyboard layouts

 Changing LiveCode’s internal string representation to accommodate Unicode has taught me untold amounts about the scripts used in various countries around the world. The integration of the text shaping library Harfbuzz (so that scripts such as Arabic, Khmer, and Devanagari look as they should on Android devices), amongst other things, resulted in an input source list that looks like this:

 

On one occasion, such keyboard switching had me completely locked out of my Ubuntu virtual machine, due to not being able to switch to English layout without logging in, but also not being able to actually type any of the characters in my password without switching to English layout. 

5. It requires concentration and attention to detail

Sometimes this is attention to small details in situations where there is not much information. Or in the case of this puzzle, extreme attention to detail (warning: attempting to solve the latter puzzle is a surefire route to madness). Other times it is a case of not trying to understand everything that you’re looking at, just the relevant parts. Like this. The aforementioned chunk object is so fundamental to LiveCode that even a slight mistake in refactoring it caused the LiveCode IDE equivalent of the blue screen of death (which happens to be a white rectangle with the “revLibURL” button on it).

 When replacing c-strings with MCStringRefs in the code, it was frequently the most efficient strategy to merely ensure that the behaviour was identical to what it was before, without completely understanding what the function was doing as a whole. 

6. It sometimes feels like it will never be finished

Getting stuck is a fact of life. Even more so a fact of puzzle hunts. Puzzles have a phase known as ‘extraction’, whereby the answer is obtained using all of the information that has been gathered and worked out. This can be as simple as using some numbers to index some words, or an acrostic that appears after sorting alphabetically. Other puzzles have more intricate extractions mechanisms, and some of the most beautiful involve iteration of procedures from previous parts of the puzzle. In any case, extraction can be very difficult, and it is therefore a very common place to get stuck.

Likewise, the period of time between being “basically” finished with LiveCode 7 and actually being finished felt like an eternity. I say actually finished, obviously there are some things to clean up, some residual bugs and so on. But we’re putting away the syntax caravan, and work is already underway on the widgets and themes project that will form the basis of LiveCode 8. In fact, I would like to get the phrase “putting away the syntax caravan” into general usage to mean almost finished. “How’s that novel coming along?” “Well, it’s not done, but I’m putting away the syntax caravan, if you know what I mean.” “No, I don’t know what you mean, Ali. Stop saying that.”

7. There are many ‘aha’s

The ‘aha’ or lightbulb moment is probably the reason why people who do these puzzles do them. It can be quite exhilarating, the joy of discovery, and is certainly quite addictive. One of the closest real-world experiences to the idealised puzzle ‘aha’ is solving a programming problem. There is a huge amount of satisfaction that comes from finishing off anything, particularly if it has involved an element of mystery, triumph over adversity, and especially if it is completely and utterly epic in scale. The refactoring project ticks all these boxes as far as I’m concerned. And there were ‘aha’ moments in spades. 

Caveat: Why working on LiveCode 7 is not at all similar to solving a puzzle hunt

I mentioned before that there is always an answer to a puzzle hunt puzzle. There is also usually a ‘metapuzzle’, which involves using all the answers to all of the individual puzzles in the hunt. Well, here’s where the analogy completely breaks down. There is not single phrase answer to the LiveCode engine. Like a more traditional puzzle, the best we can say is that it is in a more ‘solved’ state than it was when we started. But that is still deeply satisfying, and a goal I’m happy to have spent my first two years working towards.

Solutions:

Quadratum

Please Remain Seated

Contortionist

English Expectations

Cryptograf

Symphony from the Old World

Circulus In Probando

Team Statistics

and finally

The LiveCode Engine

read more

One of the main goals of the refactoring project was to increase the clarity and extensibility of the LiveCode engine, by cleaning up and modularising the script execution code. The project has made it easier than ever to contribute to the development of LiveCode.

One of the areas of syntax in which this is particularly visible is that of properties. The setting and getting of properties is now largely table-driven. Say you want to add ‘myProperty’ as a global boolean property. Here is how it would be done

1) Add your property to the property enum type

The file ‘parsedef.h’ contains all the enum types associated with livecode syntax. For internal reference, ‘myProperty’ needs a numerical value, so you would just add P_MY_PROPERTY to the Properies enum.

2) Add ‘myProperty’ to the table of syntax tokens.

The file ‘lextable.cpp’ contains all the strings of LiveCode’s reserved words in various contexts. The factor table is where all the tokens for properties are defined. In this case, we would add the line

{"myProperty", TT_PROPERTY, P_MY_PROPERTY},

This associates the token with its type (property) and the enum value added in step 1. Make sure it is in alphabetical order! Script parsing uses the alphabetical ordering to speed lookup, so any mistakes will cause unexpected errors.

3) Add to the property info table

The property info table, kMCPropertyInfoTable, is located at the top of the file ‘property.cpp’. We would add

DEFINE_RW_PROPERTY(P_MY_PROPERTY, Bool, <Module>, MyProperty)

where <Module> is the name of the module in which the property should reside. This adds all the relevant information for the engine to be able to execute the relevant property setters and getters.

4) Implement the functions

The final piece of the puzzle is to implement the following two functions:

MC<Module>GetMyProperty(MCExecContext& ctxt, bool& r_value)
MC<Module>SetMyProperty(MCExecContext& ctxt, bool p_value)

All being well, this is enough to get the line ‘set the myProperty to true’ in LiveCode to execute the C++ code in the body of MC<Module>SetMyProperty(MCExecContext& ctxt, bool p_value), with p_value equal to true.

In the enhancement requests section of the LiveCode forums, Dar Scott recently suggested having properties of “opened things” (i.e. sockets, processes and files). This proved to be relatively simple to add to the engine, and can be seen on GitHub at https://github.com/runrevali/livecode/tree/feature_pseudo-object_props.

I have implemented the following examples:

the rect of screen 
the openmode of file|process 
the name of socket 

Notice that the last example is merely an example and not actually useful at all (it just returns the name you put in, or throws an error if the socket is not found). The first example, although equivalent to ‘line of the screenrects’, is an example of the kind of syntax that will be available with Open Language. However it is not a waste of time to implement pseudo-object properties now – it will save some effort later on when Open Language is implemented. Getting the openmode (read/write/update/append etc) of a process or file is, I believe, not currently possible, at least not directly.

The idea is just to have as a separate type of chunk expression, the pseudo-object, which behaves similarly to object chunks. There are no compound pseudo-objects (yet); the only way they parse correctly is in the form ‘the
of [screen|socket|file|process] ‘.

The framework has been set up so that more pseudo-objects can be added if necessary. Every pseudo-object needs:

– A property table, and function MC<Module>GetPropertyTable() that returns it,

– A ‘get object’ function, MC<Module>GetObject, that returns the relevant pseudo-object from the expression (eg the socket from the socket name, the screen from the screen index)

– A template function which calls the appropriate getters and setters

The implementations of these should be fairly easy to copy, see for example the file exec-network-socket.cpp for the implementations of the first two, and for the third exec.h, (search for MCSocketPropertyThunk) which contains the macros and templates – the complex systems of levers and pulleys that makes it work.

If the infrastructure of your pseudo-object is already there, you can add properties to it in much the same way as for the global properties described above, but adding the appropriate incantation to the pseudo-object’s property table. See MCScreenGetRect for a simple example, or MCProcessGetOpenMode for an example using an enumerated string type.

If you’re thinking of contributing to the engine, go to the Engine Contributors LiveCode forum at forums.runrev.com to discuss your ideas with us and other contributors.

read more

Almost all of the drive towards getting out the first DP of LiveCode 7 consisted of squashing bugs and improving general stability. In other words, we were focused almost entirely on functionality. Now that the list of bugs has dropped to a manageable level, we have turned our attention to improving speed (don’t worry though, we haven’t forgotten about your bugs!)

If you’ve read Fraser’s blog post about storage of unicode strings you’ll know that certain on-screen characters take up more space in memory than others. Most of the awkwardness in manipulating and processing strings comes from having to deal with this possibility. This means that where previously accessing a character of a string was just a constant time array access operation, now it requires mapping the code unit indices to character indices. “You can’t make a Unicode omelette without having to crack a few performance eggs,” if you will.

So one way we’ve improved performance is by keeping track of when the strings have such anomalies. This is what I’m thinking of as the ‘count your chickens before they are hatched’ optimisation. Each code unit is an egg, and the characters are the resulting chicks. If I’ve promised to give you my 20,000th chick, I can simply give you the 20,000th egg, knowing that there is a one-to-one correspondence between them. Otherwise, I would have to go through them one by one inspecting each egg to see if it has hatched before being able to declare a certain chick as the 20,000th. 

You did say number 20,000, right?

Another change that has been made to speed up the engine is what might be called the ‘don’t put all your eggs in one basket’ optimisation. The basket, in this case, is the array of 16-bit code units which is used to store the Unicode string. Now we have two baskets, the second being an array of 8-bit code units. We have all our large eggs in one basket, and small in another. We’re not so much concerned about dropping the basket and losing the eggs, but more about being able to use shortcuts – sorting them out in this way should make it much easier for the eggs to be examined and put in appropriate boxes, for example. Similarly there are many more efficient methods we can use for string processing with arrays of 8-bit code units.

Speaking of boxes, another major slowdown in LiveCode 7 was related to the storage of LiveCode variables. All LiveCode variables were being stored using an internal concept of ‘value’, and never as native C types. In particular, this meant that numeric variables were stored as MCNumberRef, essentially just a wrapper round a native type. Adding to a number involved the unboxing from and re-boxing into an MCNumberRef, a relatively expensive operation. The change to allowing certain native types is roughly the equivalent of having a built-in egg holder in your fridge versus having the actual box in there.

If you thought that analogy was a stretch, then you perhaps ought to stop here. I’ll be honest, I thought there were more chicken and/or egg phrases that I could adapt for my purposes.

Benjamin Franklin once said “An egg today is better than a hen tomorrow.” How remarkably prescient of him, about 250 years before the fact, to envisage how we would be approaching the task of improving our Unicode string comparison operations. For he must surely have been referring to the following example:

put "a" before tLongUnicodeString
if tLongUnicodeString begins with "a" then
            // do something
end if

Unfortunately checking in general if tLongUnicodeString begins with another string is distinctly non-trivial if the formSensitive is false. This is because there is no way of knowing in advance what displayable characters it consists of, and how many code units might make up those characters; it may be that the whole string is only one displayed character, if it has an absurd amount of combining characters. So even if the first displayed character of the string is in fact a single code unit, we would have to normalize the entirety of tLongUnicodeString simply in order to perform the check. To use Franklin’s rather bizarrely chosen fowl metaphor, the normalized form of tLongUnicodeString is the ‘hen’ which takes a (metaphorical) day to normalize, whereas the first displayed character of it is the ‘egg’. If we could normalize on the fly, then we would only have to wait for the ‘hen’ if the strings were in fact equal. If not, we could use the ‘egg’ today to return a result very quickly.

These are just some of the optimisations we have implemented for LiveCode 7, so we think you’ll notice significant speed improvements in DP 3.

read more