Eating Our Own Haggis

by Mark Waddingham on May 29, 2014 15 comments

My turn to blog has come around again and I thought today that I’d spend some time talking about a project I’ve been working on (quietly, in the background) for a while now.

As it stands LiveCode is a slightly different beast from most other programming languages and environments in that the LiveCode Language is tightly coupled to what you might call the LiveCode Framework. This LiveCode Framework is the functionality and object model you have access to when coding in the LiveCode Language – things such as stacks, controls, the message path and such. This integration is very tight (monolithic, one might say) and is quite different from more traditional environments such as Java and C.

The Java (and C) model is more along the lines of – you have a compiler, something capable of running the output of the compiler (virtual machine or the processor itself) and a very simple / low-level collection of functionality (control structures and basic types) which constitutes the language. In this model everything else is implemented in the same language (for the most part anyway) and provided to application programmers through packages (or libraries). The neat thing about this model is that you don’t have to leave the language to ‘extend’ the language, or indeed implement the majority of functionality that you expect from the environments. Perhaps crucially, if you can code in that language, then you can (essentially) extend the language (albeit through the generic function call / object style syntax we all know but perhaps are not great fans of – there must be a reason you like LiveCode, right?).

Indeed the state of affairs with LiveCode (as it stands) very much creates a rift – there is the engine implemented in C++ where the majority of the functionality we depend on is, and there is then the LiveCode Language which sits on top. If you can program in C++ then great, you can write externals or even submit patches to the engine itself; but if you do not program in C++ (or, indeed, do not have the time nor inclination to learn to program in C++) you are, to a certain extent, limited to what the engine gives you.

Wouldn’t it be great if you could for all intents and purposes extend the engine without ever leaving the LiveCode language?

I certainly think so and thus we come to the project I want to talk about. The ultimate goal of this project is nothing less than to reduce the C++ footprint of the engine to a core script compiler, virtual machine and basic type system with everything else being implemented in LiveCode itself (and who knows, perhaps it might even be reduced further over time – LiveCode implemented completely in LiveCode itself?). Of course, it is going to take quite a while to reach this goal, but the seeds are there and have been growing for a while – thus far principally through the refactoring work our team has been undertaking as part of LiveCode 7.

So, what does this actually mean for LiveCode developers as a whole?

Well, there will be a new dialect of LiveCode in which you will be able to write things called extensions. An extension will (in the first instance) either be a library or a widget.

Libraries are collections of commands and functions that integrate into the engine in exactly the same way as engine commands and functions do (the only caveat being initially is that you will be restricted to generic function / command call syntax to access them – full and proper syntax bindings will have to wait until Open Language is born).

Widgets are collections of commands, functions and event handlers – these will be used by a new control type widget and allow you to create you own controls which look, feel and act as if they were in fact part of the engine itself. Indeed, widgets will be very familiar to anyone who has ever written a control in Visual Basic, Delphi, or derived a new control from the base control class in any of the multiplicity of C++ GUI frameworks. This idea of controls is different from the aggregate style of custom control we currently see in LiveCode – rather than using a container in which you put other controls you instead get a ‘paint’ event and all the basic interaction events you need to craft your control entirely the way you want. To help you do this, you’ll also have access to an array of functionality crafted for the task – most importantly, you’ll be able to use a collection of syntax that provides 2d vector drawing capabilities along the lines of the HTML5 Canvas, or CoreGraphics library to draw your widgets.

I mentioned above that this will be based upon a ‘new dialect’ of LiveCode (I’m currently calling it LiveCodeish) but this shouldn’t cause alarm bells (hopefully, at least). Essentially the dialect will be a distilled form of the current LiveCode core syntax and semantics – this syntax will be cleaned up and its functionality will be much better defined (for example, getting rid of auto-conversion of arrays to empty strings and ‘fixing’ the non-standard ‘for x to y’ loop we have). The main aim of this clean up is to ensure we have a solid, predictable, reliable and completely defined base to work from without having to worry about any hangovers from the (very much organically evolved) past.

[ I think it is important for me to mention is that we do not intend there to be two LiveCode languages – the language you use for extensions (what I’m calling LiveCodeish) and the language you use in object scripts. It is our intention that there will be one, and only one, LiveCode – the object script language will also move to LiveCodeish, once we’re happy we’ve got LiveCodeish right (and don’t worry, when LiveCodeish does arrive at the LiveCode script level your existing scripts will all continue to run as they do now and you’ll be able to translate them automatically over time as needed with the script translation tool we are planning as part of the syntax cleanup project that will be enabled by Open Language). ]

There are perhaps three aspects of this proposed new LiveCodeish language which deserve mention.

The first is that this new dialect will provide the ability to directly hook into native code through binding to functions in native code (whether that be Java, Objective-C or C/C++) thus, hopefully, eliminating the need to write ‘glue-code’ externals and, again, making extending LiveCode much more accessible to anyone that can code in LiveCode rather than those few who can, or have the time to delve into these lower-level environments.

The next is that it will (eventually) be typed in a very natural way. You will be able to declare variables and handler parameters as having a given type (by default their type will be any). Whenever you try to assign a value to the variable the engine will attempt to convert it to the appropriate type and only throw an error if this is not possible. For example a variable with an integer type can have an integer put into it directly, a real number can be put into it after its been automatically rounded, and even a string can – as long as the string can be converted to an integer. This simple ability will mean that it will be easier to write ‘correct’ code – you’ll be able to much more easily debug things where it could be type-mismatches that are causing errors. It also means that (in the future) suitably typed code will run more efficiently – for example if you have an integer variable then that can be much more efficiently represented internally than a variable which can hold any type.

The third thing about LiveCodeish which is quite different from normal LiveCode scripts is that there is no message path – well, the message path is there, it’s just that command and functions in extensions do not sit in it. This is exactly the same situation as current engine commands and functions – they are bound at compile time. This not sitting in the message path does not mean that extensions have no access to it, quite the reverse as they will have access in the same way as engine syntax does – you will be able to send messages to objects through the message path just as the engine can. This slight change in perspective brings LiveCodeish much closer to more ‘traditional’ languages – and means that over time a much larger range of optimization potential becomes available (as things are more static in this world – dynamicity is great, except when you want to optimize at compile time!). Indeed, I can see no reason that LiveCodeish should not be eventually compilable to native code with performance at a level where the decision to code something in C or LiveCodeish due to that metric is a non-issue – LiveCodeish will be amply able.

So, there’s been a lot of words here so far about what will be but not many examples. We’re not quite at a point yet where we have something to show along these lines (although it is my principal project right now), but what I can do is give you a theoretical example of what a piece of (useful!) (untyped) LiveCodeish might look like. Here follows a potential future example LiveCodeish script for adding support for the ‘aspell’ library (note that this code might never actually run, it is still somewhat thought-experiment syntax!):

 library com.runrev.aspell
external function new_aspell_config() is pointer from “aspell”
external command aspell_config_replace(pointer, cstring, cstring) from “aspell”
external function new_aspell_speller(pointer) is pointer from “aspell”
external command delete_aspell_config(pointer) from “aspell”
external function aspell_error(pointer) is integer from “aspell”
external function aspell_error_message(pointer) is cstring from “aspell”
external function aspell_speller_error_message(pointer) is cstring from “aspell”
external command delete_aspell_can_have_error(pointer) from “aspell”
external comamnd delete_aspell_speller(pointer) from “aspell”
external function to_aspell_speller(pointer) is pointer from “aspell”
external function aspell_config_retrieve(pointer, cstring) is cstring from “aspell”
external function aspell_speller_check(pointer, cstring, integer) is integer from “aspell”
external function aspell_speller_suggests(pointer, cstring, integer) is integer from “aspell”
external function aspell_word_list_elements(pointer) is pointer from “aspell”
external function aspell_string_enumeration_next(pointer) is pointer from “aspell”
local sSpeller command spell_set_dictionary_language pLanguage local tConfig put new_spell_config() into tConfig aspell_config_replace tConfig, “lang”, pLanguage aspell_config_replace tConfig, “encoding”, “utf-8” local tNewSpellerOrErr put new_aspell_speller(tConfig) into tNewSpellerOrErr delete_aspell_config tConfig if aspell_error(tNewSpellerOrErr) is not 0 then local tErrorMsg put aspell_error_message(tNewSpellerOrErr) into tErrorMsg delete_aspell_can_have_error tNewSpellerOrErr throw tErrorMsg end if if sSpeller is not 0 then delete_aspell_speller sSpeller end if put to_aspell_speller(tNewSpellerOrErr) into sSpeller end spell_set_dictionary_language function spell_get_dictionary_language return aspell_config_retrieve(aspell_speller_config(sSpeller), “lang”) end spell_get_dictionary_language function spell_check_word pWord local tResult put aspell_speller_check(sSpeller, pWord, the number of characters in pWord) into tResult if tResult is -1 then throw aspell_speller_error_message(sSpeller) end if return tResult is 1 end spell_check_word function spell_check_not_word pWord return not spell_check_not_word(pWord) end spell_check_not_word function spell_suggest_spellings pWord local tResult local tSuggestions put aspell_speller_suggests(sSpeller, pWord, the number of characters in pWord) into tSuggestions if tSuggestions is 0 then throw aspell_speller_error_message(sSpeller) end if local tElements put aspell_word_list_elements(tSuggestions) into tElements try repeat forever local tSuggestedWord put aspell_string_enumeration_next(tElements) into tSuggestedWord if tSuggestedWord is empty then exit repeat end if if tResult is empty then put return after tResult end if put tSuggestedWord after tResult end repeat finally delete_aspell_string_enumeration tElements end try return tResult end spell_suggest_spellings

Here you see a collection of functions and commands which in the current world would have to be implemented in C and loaded in an external but in the new world you won’t need to touch C at all.

Now, the above aspell library is just commands and functions so from LiveCode (object) scripts you’d call them with expressions such as spell_suggest_spellings() – this isn’t very LiveCode-like and isn’t really what we want. However, this is where the Open Language project comes in and I’ll talk about how that project impacts LiveCodeish in another post.

There’s one final thought I’d like to finish with. If you look at the above example script it is clear that it is pretty much the same as the current object scripts we write everyday except for the ‘external’ handler declarations at the top; thus a valid question would be to ask ‘Why not just develop LiveCodeish as an updated form of object scripts and we can just put this stuff in the library stack or backscript?’. This is certainly a fair question if you are thinking within the current LiveCode Framework but that’s missing the point of LiveCodeish. LiveCodeish is aiming to be something much more general than a language which can be used in a LiveCode object script (which requires the notion of a LiveCode object to exist to begin with!), it is a distilled form of current LiveCode syntax and semantics which have no dependence on any framework nor any structures imposed upon it apart from those necessary to be a (minimal) programming language. If all goes well then it will be something that can be used to write the LiveCode Framework itself from the ground-up – we will get to eat our own haggis in totality.

Mark WaddinghamEating Our Own Haggis

15 comments

Join the conversation
  • Dar - May 29, 2014 reply

    This is exciting.

    In your example, pointer is used essentially as an opaque type created an used by “aspell”. However, in general, we will need a way to create and unpack C structures and other low-level objects such as dictionaries.

    In writing externals, I often pair a library with the external. The external does a minimum and most of the work is done by the library. The aspell helpers are thus much like the functions in the external. This LiveCode-ish code is much like that in my library that owns the external. So, so far, I still see a need for glue. The glue is there for cstrings, I see. As I mentioned above, some functions to help with conversions to primitive things would be good.

    It would be really cool if this can refer to, say, DLLs in Windows.

    Though LiveCode-ish code would normally be bound at compile time, some things should be overridable and in the message path such as message handlers. In general, one would want a choice of implementing a command either to be compiled or to be overridable and in the message path.

    I’m hoping that this would allow LiveCode programmers to feel comfortable. If one has to program in C++ in his mind and then translate to LiveCode-ish in his mind, that might not help. Though it might be a subset of LiveCode, the fundamental LiveCode things should be there. I think.

    (I didn’t understand the comment on for-loops, but it is probably a distraction. I like ‘for each’ a lot. I have gotten used to ‘with’.)

    I think this is a good direction, but I don’t want my head to explode. Everything should work toward a rich, clean, simple, and awesomely powerful LiveCode.

    This is oh so cool!

    Dan Shafer - June 2, 2014 reply

    Dar,

    >>”It would be really cool if this can refer to, say, DLLs in Windows.”

    OMG! You mean Windows is still using 20th-Century technology!!??

    :-Dan

  • Mark Waddingham - May 30, 2014 reply

    @Dar: You are right in saying that richer bindings would be needed than just a ‘pointer’ type. Indeed, this is planned – I just erred on the side of simplicity when constructing that example. You’ll be able to define types which use specified external methods to initialize, finalize, clone and such so that some of the things you’d have to do manually to manage resource lifetime will be done automatically. Also I’d like to see high-level bindings to objective-C and java (on Android at least) objects – I want to ensure its as easy as possible to call things that are defined in ‘native’ code.’

    You are correct in thinking that ‘glue’ will still be required, but rather than the low-level glue to just call a native function from LiveCode you can concentrate on the glue that raises the level of abstraction of use of the external functions to something more LiveCode-like (and eventually with proper syntax).

    You make an interesting point about being able to specify a command as being ‘overridable’ in the message path, or bound at compile-time. I wonder whether the solution here is actually to do something like that in the way libUrl does. You have an operation activated by a command ‘foo’ that you want the user to potentially override, so ‘foo’ is bound at compile time in a library extension, and then ‘foo’ does a ‘dispatch “doFoo”‘ sending a message through the message path that script can override. It can then determine what to do based on whether its passed or handled – would that approach cover your use-case?

    In terms of LiveCodeish being a subset, it won’t be – notionally all functionality we currently have will be broken up into modules which LiveCodeish can import into a given extension for use via a ‘use’ clause. So there will be modules for the current interface framework, for encryption commands, for internet commands, for string / array commands etc. It is likely it will import ‘sensible defaults’ unless asked not to – it could be a bit of a pain if you have to ‘use’ the various modules for strings, numbers and such in every library.

  • Devin - May 30, 2014 reply

    I think we need a naming contest! LiveCodeish is short and quirky-fun, but it doesn’t really describe what it is. How about Free-Range LiveCode or LiveCode Unbound. Or we could go retro and call it MetaLiveCode.
    🙂

  • Dan Shafer - May 30, 2014 reply

    Mark, you are obviously a deep and clear thinker about computer languages. LiveCode is fortunate to have you on its team.

    I am far too unschooled in computer science to comment meaningfully on the specifics of LiveCodeish. But I do want to continue my years-long tradition of being the Voice of the Inventive User in the midst of this change and strongly urge that as you move LC itself into more of a LiveCodeish syntax model (as you say is the intent), you don’t turn LC into a language only a CS grad can learn and use.

    I don’t know what percentage of current LC users are not professional programmers. I know that in the halcyon days of HyperCard a huge majority of users fit that description. People were sort of seduced into becoming programmers and found, much to their joy and amazement, that they enjoyed the experience once they arrived at its shores.

    In your laudable efforts to, as you say, “be something much more general than a language which can be used in a LiveCode object script (which requires the notion of a LiveCode object to exist to begin with!),” describing it as “a distilled form of current LiveCode syntax and semantics which have no dependence on any framework nor any structures imposed upon it apart from those necessary to be a (minimal) programming language,” you don’t turn it into YACD (yet another C dialect) like JavaScript and so many other scripting languages.

    I, for one, don’t *care* that LC has a non-standard “for x to y” construct. (Though I will admit to that being a sort of funky example!) I don’t give a whit about standard programming languages. That’s why I love LC. It is unapologetically NOT a standard language. While I bemoan its lack of object orientation (and have from the beginning), I am willing to give up that character of the language in return for an approachable, “squishy” syntax. As I read your piece, I found myself sensing fear that you are trying to fit LC into strictures of whatever the Elite Priesthood of Programming decrees to be “standard language constructs” these days.

    For my part, I’d much rather see LiveCodeish become and stay a variant of LC that can be used to do more traditional and complex programming-like stuff without touching a hair on the head of my favorite language. But that’s probably just me.

    Mark Waddingham - May 31, 2014 reply

    Hi Dan – hopefully I can allay your fears, I can see how some of what I wrote might cause concern but those that you’ve voiced are really not my intention at all.

    The idea behind LiveCodeish is that you take the LiveCode language as it is and strip away anything that is specific to the Framework and other functionality that isn’t intrinsic to the language in order for it to be sufficient to be able to start producing useful code. Then, using LIveCodeish we can start to implement more and more of what is currently considered ‘engine-functionality’ in it. This means everyone has more power as not only will they be able to introspect on how areas of LiveCode work more easily, but also contribute and extend themselves.

    The point here is that LiveCodeish is used to make LiveCodeish what LiveCode is right now – remember that ‘Open Language’ is a project we are also working on which brings about the ability to carve up the syntax LiveCode currently has into discrete extensions and also enable extensions to be written with proper syntax. This is the bit that really makes LiveCodeish able to be used to build LiveCode – in the same way that C++ is used to build LiveCode at the moment.

    Now, whilst I am proposing cleaning up some of the more quirky semantics of the language I think there is ample reason to do so. Learning programming isn’t necessarily an easy task, and where things do deviate from what has become ‘standard’ (our repeat with loop), or where things have actions which don’t entirely make sense or make it easier to write code containing errors (conversion from array to empty string) it just makes the job of learning the language harder. I’m really not proposing anything drastic here, just taking an opportunity to improve a few things that many people have bemoaned over the years.

    In terms of the aim of their being only one ‘LiveCode language’ then I was more meaning that the added abilities and improvements that will be part of LiveCodeish will be folded into what is LiveCode at the moment – and yes, I do include the few small changes in semantics here but certainly not taking away any of the syntax we currently have.

    Another way to look at this whole endeavour is perhaps at a tower – at the bottom you have a C++ virtual machine and compiler; atop this you build a simple, small, powerful language (LiveCodeish) which is nothing more than control structures, type management and simple module structure. Then atop this you build all the modules that provide the functionality we are used to in LiveCode (it is Open Language which enables us to do this). At this point the line between LiveCodeish and what is currently LiveCode vanishes.

    LiveCodeish is more about bringing the power and simplicity of what LiveCode currently is to the area of building LiveCode itself, rather than trying to make LiveCode more complex. I often find myself saying when writing engine code, if only we could write this in LiveCode itself it would be so much easier – and that is from where the basis of the LiveCodeish project has sprung.

    Dan Shafer - May 31, 2014 reply

    Mark,

    Thanks for taking time to write such a detailed and thought-out reply. I think I see more clearly the intent of the overall LiveCodeish project. You have allayed many of my concerns with your answer.

    I guess for me the real issue will come in the packaging and description of the newly emerging version of LiveCode. As a non-techie programmer — the folks I call inventive users — I may want to be able to be blissfully unaware of the very existence of the LiveCodeish layer you describe. Just as I am able today to use LC without any awareness or knowledge of C++, I should be able to use LC in its current (I’ll call it “pure” without driving any theological stakes in the ground here) form. IOW, I should have to do something active — load a different version, flip a non-trivial switch, add a library, switch the IDE, *something* explicit — even to see or be aware of LiveCodeish.

    To me, it *almost* feels like what I really want is a sort of LiveCodeish toolkit that sits outside LC for the average user but which integrates smoothly for the advanced coder who really wants to extend the language.

    I do think this whole undertaking will be fruitful and the outcome powerful. I guess I just want jealously to protect what we have from further complexity.

    BTW, I am not familiar with the Open Language effort you reference. Is this an LC project of some sort? I’ve been out of touch on the roadmap in recent years, so perhaps it’s common knowledge in the community. Maybe you can just point me to a discussion of it?

    Mark Waddingham - June 2, 2014 reply

    Hi Dan,

    The Open Language project is a technology we’re working on that allows the LiveCode syntax to be extensible – i.e. extensions will be able to wrap their commands and functions in proper English-like syntax. Indeed, the power extensions will have in this regard will be equivalent to the engine as the engine will move to using the same system to specify all the syntax it currently has.

    I must confess I considered having a clear ‘multi-level’ approach as you suggest – i.e. having a LiveCodeish middleware layer which the higher level was implemented upon; however, I’m not sure this big distinction is necessary as Open Language means we can retain self-similarity with all levels, the only difference being the syntax you choose to have available.

    From the LiveCode users point of view the system will appear like an onion. At the outer-most level you see what you currently see, but you’ll be able to peel back the layers and see the implementation of the commands and functions you are using (in a familiar syntax, although perhaps with a some slightly more difficult concepts being used), which in turn might rest upon lower-level libraries and abstractions. The point here is that you won’t have to learn any more than necessary to do what you want to do – you just pick extensions and syntax which meets your purpose and level of scripting ability. For example, there might be a low-level database syntax extension which deals direct with SQL; then on top of this a higher-level syntax extension which makes things more natural (perhaps presenting things as tables, records and fields).

    Mark.

  • Brahmanathaswami - June 1, 2014 reply

    Since I don’t know how far the implications of these changes reach, the question may not pertain. But, have to ask anyway: would this enable the engine to be multi-threaded? I thinking of a more powerful web server interpreter, if that were possible.

    Mark Waddingham - June 2, 2014 reply

    Hi Brahmanathaswami,

    I’m not sure how threading might help to make a more efficient web-server interpreter exactly; however, the refactoring work and general cleanup of the engine source-tree which is underway should certainly make it easier to create an Apache module or similar. The latter are much more efficient than pure CGIs.

    Mark.

Join the conversation

*