It’s November, Thanksgiving is here and Christmas is on its way. It’s a good time to look back on what has been in equal measure a fantastic and frustrating year, and to consider the way ahead.
This year we have invested more money than any other year in our history into LiveCode and its future. Every penny of your subscriptions, every cent earned from our Services work, and a sizeable chunk of other funding has gone into our platform developments this year. We’ve had a lot to do!
Progress on LiveCode 10
Where is it? This is a question that many of you have been asking, both publicly and privately. Let me start by saying – I feel your pain. We’d love to have shipped it by now. There are major pieces of work in this version which are needed to underpin everything we plan for the future. We all need that faster, smaller Web engine, the speed improvements and better syntax of the Script Compiler, the Web graphics and fonts to name just a few.
There are a number of reasons for the delay in shipping this important piece of technology. Not least is that there is little point in shipping it without the Script Compiler. This is taking longer – a lot longer – than projected. Let’s talk about that.
Let’s talk about the Script Compiler
To understand why this is taking so long, we need to take a little trip through history. The underlying engine that LiveCode runs on was built approximately 35 years ago prior to our stewardship of the engine. Since then, a great many things have been bolted on, chopped off, reworked and entirely refactored from the ground up. The code base is big, it’s old, and it’s very very complex. The last time we had a major refactor was for LiveCode 7 and we learned a great many lessons from that experience. If you were along for the ride on that one you may recall it took some time and when it did ship it had a number of – ahem – challenges as a version, which were not satisfactorily resolved until LiveCode 8. We are working hard to avoid such challenges arising again. As part of the Script Compiler refactor, every single change is having an appropriate accompanying test built, to make sure that when it flaps its wings the butterfly effect doesn’t do strange things to your projects. This particular refactor is going to form a foundation for great things to come, for many years. We’d rather have it right than have it now. (Actually we’d like to have it right AND now, but that has proved beyond the realm of the possible).
While we did assess the project extensively prior to starting it, there are something things that really only become apparent when you get in there. To give you just a flavor of the kind of things that are involved in this particular refactor, here are a couple of reports from the coalface, from our regular meetings on this topic.
Syntax Implementation
Syntax in the engine is implemented in two pieces:
- a ‘syntax class’ which has methods for parsing (parse) and dispatch (exec/eval_ctxt)
- exec/eval methods which actually perform the operation
Syntax classes are generally overloaded with one class potentially able to dispatch to many methods.
For the purposes of the compiler it is important that the number of methods is kept to a minimum, and essential that there is a strict distinction between the code in dispatch and the code in the exec methods.
Specifically, dispatch must only evaluate arguments and then call a method – it must not process arguments in anyway, nor do any runtime checks beyond those related to types.
Our team has been working through all functions, operators and commands making sure that both the number of methods are reduced to the minimum, and that there is a strict split between dispatch and execution.
Bootstrap Engine
The script compiler implementation uses tooling written in lcs to generate both C++ and lcs source files and the bytecode compiler is written in lcs itself:
- token generator creates compressed token tables (used for parsing) [C++]
- nodegen creates methods for creating syntax trees engine-side [C++]
- nodegen creates a library for traversing syntax trees compiler-side [lcs]
- compiler turns lcs into bytecode [lcs]
This presents a ‘chicken-and-egg’ problem in terms of building the engine…
You need a built server engine to be able to run the tools to be able to build the engine (well the part which parses and compiles scripts!)
While the engine build process does currently include some tooling which is run before a running engine is available, they have to be written in one of:
- bash/batch
- python
- perl
This is (obviously!) not practical for the script compiler related tools.
Our original thought had been to run the tools with an engine which doesn’t need the script compiler tooling and commit the resulting files to the repo.
This, as we discovered, is a terrible idea as small changes to the input source files result in large changes in the output source files – the git commits are huge*.
The solution to this problem has been to add an architecture for using and updating a ‘bootstrap’ engine – which is guaranteed to be present before the engine builds.
Bootstrap engine archives are created mechanically using update-bootstrap.sh.
Given a engine commit hash, this does the following:
- Downloads the built engine server zips from staging
- Extracts the server executables and builds tarballs with unique names
- Uploads the tarballs to the static prebuilt area
- Create a branch and commit marking the update with enough information to be able to track which bootstrap engine is being used (via commit hash)
Then the appropriate archive is downloaded and unpacked during the fetch phase of the engine build.
These are just a tiny sample of the many many tasks to be undertaken to produce a Script Compiler that is stable, fast, and provides us with the beautiful syntax we are eagerly looking forward to. We are getting there. A great deal has been done. Quite a lot of careful, complex and concentrated work remains to be done.
What have we achieved within the core technology this year?
Here is a list of all the things we have been aiming at for LiveCode 10, and their current status:
- Web Engine Size Reduction – ICU – DONE
- Web Engine Size Reduction – Web Fonts – DONE
- Array and sequence literals – DONE
- Constant initialiser expressions – DONE
- Improvements to OAuth2 library – DONE
- Sign-in with Google – DONE
- Migrate macOS Web Browser to WKWebView – DONE
- Allow browser widget to stream webcam video – DONE
- Sign-in with Apple Integration – MOSTLY DONE
- In-Engine JSON – MOSTLY DONE
- Script Widgets – IN PROGRESS
- Cross platform audio recording – IN PROGRESS
- MediaFoundation for Player on Windows – IN PROGRESS
- Migrate Windows Web Browser to Edge – STARTED
- Theming based on CSS – STARTED
- Graphics – Driver abstraction – STARTED
- Web remote debugger – STARTED
- Player control on web – STARTED
- Web Engine Size Reduction – Script Parsing/Execution – PART OF SCRIPT COMPILER
- Constant folding – PART OF SCRIPT COMPILER OPTIMIZER
- Constant switch optimisation – PART OF SCRIPT COMPILER OPTIMIZER
- Web Engine Size Reduction – Native Canvas APIs (Skia Removal) – NOT DONE
- Graphics – Use HTML5 Canvas Driver (size reduction / performance) – NOT DONE
- Graphics – Use HTML5 Image APIs (size reduction / performance) – NOT DONE
- Graphics – Move printing infrastructure into libgraphics – NOT DONE
- GoogleAds integration on Android and iOS – NOT DONE
In addition to this list of features, we have been doing all the usual updates to stay abreast of new OS releases, so that LiveCode 10 can support Sonoma, Windows 11 and the latest iOS and Android platforms. Of the list above, almost 70% are either done, in progress or at least under way. Some of the remaining items depend on the completion of the Script Compiler.
There is an argument for shipping 10 as soon as all of the items already in progress are completed, instead of waiting for the full feature list. It is something we keep under constant review. The current Developer Preview release of 10, dp-6, is quite stable and has many useful features in it. We could put an RC label on it and release it, but it would not be the 10 you (and we) are all waiting for.
Why we need 10
For any platform to thrive, it must be relevant to the audience, providing the features that new users expect, and an experience that will delight. It’s not enough for LiveCode to have the power to create pretty much anything you can imagine (something that it’s long time users rightly love it for). To bring in a new, young, vibrant audience, it must also be easy to use, beautiful to look at and stand up against the increasingly sophisticated competition. AI has become in the blink of an eye not a “nice to have” but a part of the essential toolkit for any programming environment that wants to succeed. A fast and easy way to build interfaces is now expected from any serious tool. Low and no-code (however you define these terms) are the future, and LiveCode is going to be a part of that future. To do that, we need LiveCode 10 and all the advances we’ve detailed above, to underpin everything we do with LiveCode Create. We cannot build LiveCode Create without it.
LiveCode Create
LiveCode Create is built 100% on LiveCode. While it is still waiting for some essential engine features from 10, Create continues to make good progress. The interface looks modern, there is some AI already operating in it, and many widgets have been built. Here are just some of the things we’ve done on Create this year:
- Implemented basic AI
- Shipped a working Web editing interface
- New Actions Editor which now supports creating, updating and deleting records from a collection
- New Data Binding – You can now bind objects to data in a collection
- Created these brand new widgets
- Youtube widget
- Button widget
- Field widget (Includes single line, password, email, text area, multiline field options)
- Date Field with Picker widget
- Checkbox widget
- Dropdown Field widget
- Dropdown Button widget
- Text widget
- Simple List widget
- Nav Rail widget
- Table widget
- Slider widget
- Vimeo widget
- Web Video Player
- Avatar widget
- Imagelist widget
- Content Card widget
- Tab widget
- Ratings widget
- Choice Chips widget
- Divider widget
- Stripe integration has been started
- Zapier integration has been started.
In latest news, in our internal builds it is now possible to interact with the cloud data store using actions. For example you can build a custom form and add an action to the button asking it to create a new record using the input from the form’s objects, all without writing a single line of script yourself.
All of this is still in preview stage and is a work in progress.
We are currently making big improvements to the user experience of both the Property Inspector and Actions Editor. These are major parts of the LiveCode Create flow and have to be easy to use. We have some basic AI features integrated into the Script Editor with more CoPilot like features soon to be added. We almost have a first version of the AI UI builder. We’re focussed on providing a great user experience for you. This doesn’t mean just making sure we have all the features but packaging them in an experience that delights or more importantly doesn’t frustrate. There is a long way to go, but so far so good.
The role of Services
Some of you may have noticed that we now offer Services as part of our line-up. There are many benefits to doing this. Doing LiveCode projects for our community helps to keep us in touch with what you need. Quite a number of LiveCode features have started life as part of a Services project. We are supporting some of our longest standing customers with their projects, adding new features, helping with facelifts or creating new apps for a changing world. We view this as part of our partnership with the community – after all no other company is better placed to provide consulting services for LiveCode based apps! Every penny earned from these projects goes back into supporting and developing the LiveCode platform.
Crossing the Rubicon
So in summary, some mixed emotions as we approach the end of the year. Our progress has been so exciting. We’ve done more this year than in any year – in all our decades of history. So much of it is looking amazing. Many very large and very essential new pieces are all getting built in parallel. A modern and fresh new approach to building apps is taking shape, built 100% using LiveCode. So much progress has taken place on that underlying LiveCode technology, version 10 and the Script Compiler.
Yet the end of this year also feels difficult, both for us and our community. We’ve done lots of small point releases and preview versions throughout the year. But none of the big pieces are finished and shipping yet. I know its the biggest pivot in our history, yet like many of you I feel impatient!
Next year will be different. Like buses, these step changes will arrive in quick succession. I can’t wait to delight all of you and help lots of brand new customers realize their app building dreams.
How you can help
We know you’ve been waiting for a long time. We appreciate your patience and your support. Please wait a little longer, continue to maintain your subscriptions, and if you have any service work that would benefit from our team’s expertise we’d be delighted to hear about it.
The darkest hour is always before dawn, but the dawn is coming.
2 comments
Join the conversationGlenn Toole - November 23, 2023
Wow! A very comprehensive, transparent, and honest description of where you are at and where you are going. I appreciate the update and please pass along encouragement to continue the good work!
Craig McArthur - January 13, 2024
Couldn’t agree more Glenn. A fantastic product built and supported by brilliant people. I’m looking forward to version 10, but appreciate the wisdom of ‘have it right rather than have it now’.