Since we’ve finished the refactoring of the engine, with every functionality working as it should, a performance issue was raised by several community members, on different parts of the new engine. I have been working closely with Ali on this aspect, and here is my take on it, with fewer eggs but a nice musical clef.

We were of course aware of this slowdown, which was mainly caused by the fact the engine was working in a uniform way, regardless of the kind of characters in the strings in use. This consideration of the strings was the first part of the global refactoring plan, to allow us to bring modifications impacting the whole engine instead of micro-changes targeting one area, as had been done on the former versions; the pitfalls behind a single modification were obviously numerous.

Chiefly, the main slowdown comes with the handling of Unicode: there is much, much work to be done any time a string operation is executed. As Fraser explained it in his blog post, Unicode is not simply the ability to use characters out of the ISO-8859-1 encoding. It also comes with all the subtlety of introducing characters longer than one character, be it:

– combining chars, to add as many accents as needed on the same letter (this type of character OS X gives you easily, since it uses the NFD form).

surrogate pairs – who uses music? The treble clef, pictured left, is one of these characters stored as a surrogate pair. 

 As one may guess, it becomes slightly more difficult to compare two strings when they can be the same, even having a different number of characters – even worse, finding a substring within a string is an operation starting from the beginning, since no character indexing is valid when combining chars or surrogate pairs are present. And since LiveCode users love to use ‘items’, ‘word’, ‘line’ or any of the new chunks introduced in 7.0, it would be best to bring the script executions back to their pre-Unicode timing, when possible.

The main goal has been to avoid as much as possible using the CPU-costly Unicode functions, which boils down to storing more states for a string – is it native, combined, does it include surrogate pairs? This is makes the string operations work differently according to their content, and discards any slowdown which could be caused by Unicode’s intricate rules. In the end, some operations even became faster than they were before – ‘before’ shows it for instance.

In the same way, the engine is now clever enough to keep in mind when a string has been converted to a number. Since a LiveCode variable only stores strings, that is something which comes quite handy when used in a loop, and probably could be tagged as speedup!

Following on the examples coming as bug reports against the slowdown, here is a comparison between 6.6.1 and (future) DP-3:

That should have you enjoying the DP-3!

In my last article, I described how Unicode text can be broken down into its individual subcomponents: characters are composed of one or more codepoints and these codepoints are encoded into code units which comprise one or more bytes. Expressed in LiveCode:

byte a of codeunit b of codepoint c of character d

This article will explain how you can use these components when processing Unicode text.

Characters to Codepoints: Normalisation

Following the chunk expression above, the first step is breaking a character up into its constituent codepoints. As discussed yesterday, these could be in either composed or decomposed form (or even somewhere in between!). Should you prefer one particular form over another, LiveCode includes a conversion function:

put normalizeText("é", "NFD") into tDecomposed

The “NFD” supplied as the second parameter says you want the string in a Normal Form, Decomposed. For composition, you would specify “NFC”. (There are also “NFKC” and “NFKD” forms but these are not often useful. The “K” stands for “compatibility”…).

What do you think will happen when you execute the following line of code?

answer  normalizeText("é", "NFC") is normalizeText("é",  "NFD")

LiveCode will happily tell you that both strings are equal! This shouldn’t really be a surprise; when considered as graphical characters they are the same, even if the underlying representation is different. Just like case sensitivity, you have to explicitly ask LiveCode to treat them differently:

set the  formSensitive to true

With that set, LiveCode will now consider composed and decomposed forms of the same text to be different, just as it treats “a” and “A” as different when in case-sensitive mode. Also like the caseSensitive property, it only applies to the current handler.

Lets use this knowledge to do something useful. Consider a search function – maybe you’d like to match the word “café” when the user enters “cafe” – here’s how you’d remove accents from a bunch of text:

function stripAccents pInput
local tDecomposed
local tStripped

-- Separate the accents from the base letters
put normalizeText(pInput, "NFD") into tDecomposed

repeat for each codepoint c in tDecomposed
-- Copy everything but the accent marks
if codepointProperty(c, "Diacritic") is false then
put c after tStripped
end if
end repeat

return tStripped
end stripAccents

The function also demonstrates another very useful function – codepointProperty – which will be our next port-of-call.

Codepoint Properties

The supporting library that LiveCode uses to assist in some of the Unicode support (libICU) provides an interface for querying various properties of codepoints and this is exposed to LiveCode scripts via the new codepointProperty function. To use this function, simply provide a codepoint as the first parameter and the name of the property you’d like to retrieve as the second parameter.

There are a large number of properties that exist, some of which are more useful than others. For an overview of the properties that the Unicode character database provides, please see here (http://www.unicode.org/reports/tr44/). Some of my personal favourites are:

1. “Name” – returns the official Unicode name of the codepoint
2. “Script” – script the character belongs to, e.g. Latin or Cyrillic
3. “Numeric value” – the value of the character when interpreted as a number
4. “Lowercase Mapping” and “Uppercase Mapping” – lower- or upper-cases the character

Example output from these properties:

answer codepointProperty("©", "Name")              -- "COPYRIGHT SIGN"
answer codepointProperty("Ω", "Script")            -- "Greek"
answer codepointProperty("¾", "Numeric Value")     -- 0.75
answer codepointProperty("ß", "Uppercase Mapping") -- "SS" 

Code Units and Bytes: Encoding

The LiveCode engine does a lot work to hide the complications of Unicode from the user but, unfortunately, not all software is written in LiveCode. This means that when you talk to other software, you have to tell the engine how to talk to it in Unicode. This is where text encodings come in – every time you read from or write to a file, process, network socket or URL, text has to be encoded in some way.

To convert between text and one of these binary encodings, use one of the aptly named textEncode and textDecode functions:

put url("binfile:input.txt") into tInputEncoded
put textDecode(tInputEncoded, "UTF-8") into tInput
…
put textEncode(tOutput, "UTF-8") into tOutputEncoded
put tOutputEncoded into url("binfile:output.txt")

If you are using the open file/socket/process syntax, you can have the conversion done for you:

open tFile for utf-8 text read 

Unfortunately, the URL syntax does not offer the same convenience. It can, however, auto-detect the correct encoding to use in some circumstances: when reading from a file URL, the beginning of the file is examined for a “byte order mark” that specifies the encoding of the text. It also uses the encoding returned by the web server when HTTP URLs are used. If the encoding is not recognised, it assumes the platform’s native text encoding is used. As the native encodings do not support Unicode, it is usually better to be explicit when writing to files, etc.

An an aside, we are hoping to improve the URL syntax in order to allow for the same auto-conversion but have not yet settled on what it will be.

As I mentioned in my previous blog post, Unicode text is hard (which is one of the reasons it has taken such a monumental effort to get LiveCode 7.0 ready for release – it is now in public testing if you’d like to try it out). In order to make everything work transparently for the writers and users of LiveCode stacks, a lot has to go on behind the scenes. In this post and its follow-up, I hope to explain how some of these innards work. This first post is a bit technical but will lay the groundwork for some new Unicode text processing techniques.

The most important thing with Unicode is to understand what is meant by a character – different people have different definitions, some quite technical. Older computer software will often refer to 8-bit bytes as a character, a standard which LiveCode and its predecessors followed. Sometimes, "character" is used for the symbols defined by the Unicode standard (these are more properly termed "codepoints"). Neither of these is necessarily what a human reader would think of as a character, however.

Consider the letter "é" – that’s obviously a single character, right? Well, it depends on who you ask… Considered as 8-bit bytes, it could be anywhere between 1 and 8 "characters". Looking at it with Unicode-coloured glasses, it could be either 1 or 2 codepoints. However, in LiveCode 7, it is always a single character. If you were a Unicode geek like me, you’d call this LiveCode definition a "grapheme cluster".

Why do these different interpretations arise? If you’ll bear with me, I’ll take it apart piece-by-piece.

First comes the codepoints. The Unicode standard defines two types of representation for accented characters known as "composed" and  "decomposed". Continuing with "é" as our example, Unicode would call this U+00E9 "LATIN SMALL LETTER E WITH ACCUTE" in its composed form. In its decomposed form, it would be a U+0065 "LATIN SMALL LETTER E" followed by U+0301 "COMBINING ACCUTE ACCENT". Basically, composed versus decomposed is the choice between accented characters being characters in their own right or instead being an un-accented character with an accent atop it. Conversion between these forms is called "normalisation" and will be discussed in my next post.

Next comes the variable number of bytes that are used to store these codepoints – this comes down to how these codepoints are encoded. Sometimes, old 8-bit encodings have a single byte to represent a particular composed character. Unfortunately, these encodings can only represent 256 different characters so Unicode encodings are used instead. The particular encoding used within LiveCode is UTF-16 (but this is internal to the engine and isn’t visible to LiveCode scripts).

The UTF-16 encoding uses 16-bit values to store codepoints, termed "code units". This extra term is needed because although many languages have all of their symbols representable using a single code unit, a number (including Chinese) need two code units per codepoint for certain characters, due to the large number of symbols within the language. Because of this, a codepoint can be either 2 or 4 bytes in length when encoded with UTF-16.

Other common text encodings are:

1. UTF-8. Uses between 1 and 4 bytes to encode codepoints. Common on Linux and MacOS X systems.
2. UTF-32. Always uses 4 bytes per codepoint. Trades space efficiency for simplicity.
3. MacRoman. Always 1 byte, non-Unicode. Legacy encoding on most MacOS systems.
4. ISO-8859-1. Always 1 byte, non-Unicode. Legacy encoding on many Linux systems
5. CP1252. Always 1 byte, non-Unicode. Legacy encoding on many Windows systems.

As you can see, there is a fair bit of complexity behind the transparent Unicode support in LiveCode 7. In my next post, I’ll show you how you can take advantage of knowing how it all fits together.

This longed-for day of the 7.0 engine, stable release is coming soon; some of my colleagues have already described before me the most appealing feature linked to this new version is the handling of Unicode characters. But that’s not all: allowing a flow so different from plain text to run through the engine resulted in touching so many parts of it that a deep refactoring has been executed.

In the 7.0 developers team, we’ve been fighting for almost a year the massive beast that renewing the whole engine was, and it’s now more than ever tamed and keen to act as expected.
All this process involved the use of artful alchemy; starting from a closed-mind golem, we had to move and refashion almost every single stone he was made of, and to rearrange all the cogs linking those moving parts to keep the communication between them going in the same way. This included giving a new shape to his brain; and he now has the ability to learn more easily a new knowledge or update independently what he already has, thus making the addition of a new feature a simpler way to go.

Now that the final shape of the new engine is finished, the only thing separating the community from the first stable release is the cogs correctness. All these parts relocations and updates let a little play come in few mechanisms interaction, and the huge amount of changes makes the tracking of those tiny errors way easier with your help: given the symptoms, it’s always possible for us to find the issue and bring the engine closer to the perfection aimed.

The best point in all of this is not that our creature can now handle Unicode: it’s more about all the advantages the community’s applications will be able to gain from it!

It has been a number of months since Ali reported our progress on the engine refactoring project and the integration of Unicode into LiveCode (Slaying the Unicode Monster) and in that time, much has changed. The project is nearly complete and, as Kevin said yesterday, we are approaching a DP release.

Supporting Unicode and international text has required extensive changes throughout the engine – too extensive to cover in a single blog entry – so today I’ll explain the changes to one of the most visible parts of LiveCode: fields.

In the current releases of LiveCode, it is possible to use Unicode text in fields. Unfortunately, it requires special syntax and can be a bit cumbersome to manipulate properly. In addition, the support is fairly rudimentary and doesn’t work properly for languages requiring complex text layout (for example, Arabic).

7.0 will change all that – Unicode text in fields (and throughout the engine) is manipulated the same way as any other text. In fact, the engine doesn’t distinguish between Unicode text and “plain” text anymore – they are both just text. But that’s a story for another time.

Most of the changes in the field to support Unicode are “below-the-hood” and won’t be immediately apparent. They have, however, allowed for a much greater deal of flexibility in how text in fields is processed and I’ll summarise what this has allowed us to do:

East Asian languages such as Chinese and Japanese. Previously, these could be entered but the field had difficulty with certain characters that required a certain type of Unicode encoding called “surrogate pairs” – the components of these pairs were treated as separate characters, causing problems when one of them was deleted or had its style changed.

Complex scripts where multiple character fragments combine to form one graphical character (called a “grapheme”). For text manipulation, these are now treated as single characters (and new chunk types “codepoint” and “codeunit” have been added for those who need to access the individual components).

Cursor navigation working appropriately for non-English text. Navigating left and right through a field happens on grapheme boundaries, ensuring that the cursor never ends up between a character and its accent. The keyboard commands for moving forwards and backwards by whole words also works for text that doesn’t use spaces as word separators (e.g. Chinese).

Right-to-left and bidirectional text. Mixing left-to-right and right-to-left languages (e.g. Hebrew and Arabic) text in a field now lays text out in the correct order, including the situation when LTR is embedded within RTL or vice-versa.

All of this is available without any extra work on the part of a developer creating a LiveCode app – our goal with our Unicode support is to make it just as easy to create an app with Unicode support as without. We hope you’ll be pleased with the result!