Reprinted from revUp, read the rest of our twice monthly newsletter here.

When I work in other languages I’ve come to depend on version control to help me keep track of everything. I would like to see the same productivity increases and extra documentation trail that version control provides in my LiveCode projects. Unfortunately LiveCode files are a binary format which is great if you want to pack a lot of data into a small space but not so great if you want to have a version control system work out the difference between two versions of a file. Most version control systems will just decide if a file is text or binary and if it’s binary just treat any difference as a merge conflict.

It is possible to install additional drivers for these systems to deal with binary files or structured data like xml in a better way than line by line processing. Unfortunately hosted repository sites like GitHub and BitBucket would still see the files as binary so we wouldn’t gain benefits like being able to comment on specific changed lines in a script. Apps like SourceTree (Atlassian’s Git/Mercurial client) also wouldn’t be able to show us the differences. So we need a text based file format.

The lcVCS repository in SourceTree (click to zoom)

Towards the end of last year (I’ve been working on this for a while) I decided I would try and crack this nut. Along the way the project (lcVCS) has been a driver for me to do some work on the LiveCode engine including `_internal resolve image`, `the childControlIDs of group|card`, `the cardIDs of group`, `the properties of object` and `is an ascii string`. Additionally I’ve filed half a dozen bug reports on obscure issues that may not otherwise have been identified such as the minuscule rounding error on gradient ramps I found recently. As a result it requires version 6.1.1 of LiveCode.

At first I took a look at what had been done before. Mark Wieder has done a considerable amount of work on this issue with exported stackFiles saved as XML with a directory structure mirroring the stackFile hierarchy. I made contact and told him I was looking at the problem and he has offered a wealth of advice and been my goto guy on this ever since.

I was inspired by Vincent Driessen’s article ‘A successful Git branching model’ so my focus since the beginning has been to ensure we can have multiple development branches of a LiveCode project open at the same time. That means we need to be able to merge these branches so that changes on both are kept. Mark’s work had proved you could reliably export and import a stackFile. When I came to look at the merge issue, however, I hit some rather large roadblocks.

The first and arguably the easiest to overcome is the fact that stackFiles save everything including things that will almost certainly cause merge conflicts like the rects of resizable stacks, text in fields from your last test etc. So if you have two branches of a project and in both you have resized a resizable stack you would have a merge conflict on that… heaven forbid you have a resizeStack handler! My solution was to dispatch a message to each object just before it is exported so it can reset itself to default settings. In practice I usually handle the message in the card script. Anything that is pure session data should be cleared or reset to its default state. Messages are unlocked while the handler is run so your resizeStack handlers or anything else can execute.

The second major issue is one I’m still struggling with. LiveCode objects each have an ID. The stack ID is actually a place holder for the next object ID to be used on a stack. So IDs in stacks are akin to an auto-incrementing primary key in a database. The problems arise when we have multiple branches of the project all creating objects. In each branch the stack is assigning the next available ID. ID’s are a very common way to refer to objects, for example, an image ID is used as the icon reference in a button. So when merging branches of a project we need to ensure that the merge is robust to ID conflicts.

In lcVCS I’ve handled this by assigning every object a UUID (Universally Unique ID) which is basically a random number so large you would need to generate a billion UUIDs a second for 36 years to have a 10% chance of a conflict. Object references are translated into UUIDs to export the stack and when imported they are translated back. This process allows lcVCS to gracefully handle different objects on different branches having the same ID by allowing a new one to be assigned to one of the objects and updating any references to the object.

Unfortunately this isn’t a simple problem. Not only are IDs used as icons, patterns and behaviors they are also often stored in custom properties. For example, the DataGrid stores it’s row template as an ID. To work around this issue I implemented a plugin scheme so that anyone that has a control or library that stores IDs could add support for their custom property set in lcVCS. One day I hope there will be an engine level solution to this problem but I doubt we will see that for a while.

A related issue is when you have a project with multiple stackFiles you need to ensure that they are imported and exported in the correct order so that everything is found. To deal with this I’ve implemented a project file that maintains an ordered list of the stackFiles in your project.

Lately I’ve been working with Trevor DeVore on Clarify 2. Trevor has been very supportive of lcVCS and Clarify 2 is pioneering the use of it on a more complex project than lcVCS itself (lcVCS exports and imports itself which is fun). One of the things I found is that deeply nested groups can create very long paths in a directory structure that mirrors the stackFile hierarchy. On Unix based systems this is fine but Windows can’t handle paths longer than 260 characters so I had to redesign the directory structure to be much shallower. My hope is that the change was the last major re-design lcVCS will need (it’s had many!).

It’s interesting that when you decide to solve one problem you often end up solving others too. Both lcVCS and the mApp framework are LiveCode plugins. I needed to both use the plugins and work on them and the idea of moving the stackFiles from the plugins folder into their Git repositories every time I wanted to export and commit didn’t appeal so I looked for another solution. When I implemented the project file system I found a solution there by allowing the build path to be outside the repository. lcVCS can now import projects directly into the plugins directory or any other directory for that matter. It then dawned on me that if I provided an easy way for users to search for lcVCS projects, install them locally and to switch between branches or tags in the repository that I would have a great way to deliver open source plugins and libraries. A bit of Googling and I found GitHub has a search API, I could search for the lcVCSProject.json file and it would return all the public lcVCS based projects on GitHub. A few shell calls to Git and I had a list of tags (often version numbers) and branches to put in a switch to button.

lcVCS GitHub search (click to zoom)

So now lcVCS is both a way to help you version control your app, plugin or library and a way to deliver your open source plugin or library into the hands of other LiveCoders just busting to help you out with a contribution or two.

The lcVCS project is open source and licensed under the GPL 3 and is available on GitHub. Because lcVCS imports itself you will also need to get the current versions of the stackFiles which are available from the downloads page under a free or paid account at mergExt. The simplest thing to do is get the stackFiles then add lcVCS as a project so that you can update it as new versions become available. You will also need to install mergJSON (also available under a free or paid account) which is the super fast dual-licensed JSON parser external lcVCS is built on. While you’re there why not check out what mergExt can offer for your apps?

If you’re interested to start using lcVCS I am available for hands on consulting to help you and your team get up and running. There’s also some rules your project will need to follow in order to successfully transition:

If you use IDs in scripts anywhere (such as setting the icon of a button) then replace it with a name based reference.

Implement lcVCSExport handlers to ensure everything is set back to defaults during the export.

If you have any custom objects or libraries that maintain custom property sets that store IDs then implement a plugin to support it. The plugin api is quite simple and there’s a number of examples demonstrating their use. If possible contribute the plugin back to the project so we can maintain them centrally.

Limit object references between stackFiles where possible. Where it’s not possible ensure that there are no circular references. For example, stackFile A has an object reference to an object in stackFile B while stackFile B has an object reference to an object in stackFile A.

Order your stackFiles so that the stackFiles that have object references to objects in other stackFiles are imported and exported after the stackFiles they refer to. The stackFiles list in the lcVCSProjects stack can be re-ordered by drag and drop.

If you have object references to objects that no longer exist (such as icons referencing deleted images) clear the property.

By Monte Goulding, published in revUp this week.

It probably comes as no surprise to most LiveCoders that I’ve been a little desperate for an Android Externals SDK for a while now. One of the most common pre-sale questions I field from mergExt customers is – do you have this for Android or that for Android. Up until this week I’ve had to say no, we can’t do that yet because RunRev haven’t released an SDK. Usually I cheekily ask them to harrass RunRev’s support team to help make it a higher priority (sorry Heather). It’s not just self interest though. I strongly believe that externals are a critical component of the platform enabling us to get the job done when the engine doesn’t cover a feature we need.

One of the great things for me about LiveCode going open source is I’ve been able to gain a much broader insight into how externals work by studying the api in the engine, RunRev’s externals and the lcidl compiler. For those that don’t know what the lcidl compiler is it’s an executable that reads in an interface definition file for the external and generates a C++ file that acts as a glue between the external implementation and the engine. One of the first things I noticed was it is quite easy to use the old externals SDK on iOS and Android for C or C++ externals. That’s how all of RunRev’s externals work cross platform and how mergJSON and mergMarkdown went cross platform. The problems start though when you want to access Android’s Java API.

We had some great discussions with Mark Waddingham, Mark Wieder and Jan Schenkel at RunRevLive 13 and on the Engine Contributors forum about how to implement the Java based externals, but things didn’t really start moving until Mark Waddingham implemented the core design of a support class, an implementation class and the JNI (Java Native Interface) glue binding it all to the engine. It was an exciting moment for me to test Mark’s demonstration external which consisted of the creation of a large button.

Say Hello to Android Externals… the first button

I immediately implemented a command lovingly named mergPopToast. A Toast on Android is a short message that is presented on screen for a few seconds before fading away.

Toasting the World

Having implemented my first command I wanted to get to work only to find that one of the most useful features in the iOS SDK, LCObjectPost, wasn’t fully implemented yet. You could post a message to an object but no parameters. So I went to work exploring JNI (something I hadn’t done before) and worked out how to do it. I submitted it to Mark and he promptly re-wrote it but I think my ideas helped a little bit… To test it all out I created mergStorageStartMonitoring which sends a callback (mergStorageStateChanged) whenever access to the sd card changes. I also added a function mergStoragePath to get the standard paths to the user’s public storage folders for good measure.

The next thing we needed was a way to start an Activity and get results back when it’s finished. Android applications are broken down to activities which can be thought of as cards with a specific feature. You start a new Activity with an Intent (to give it parameters for what it should do), when finished the user is returned to the Activity that started it with any results. The problem though was the engine’s Activity would handle all returns and not know what to do with the result from an Activity the external started. So we needed a way for the Java side of the engine and the Java side of the external to communicate directly. Starting the Activity and handing its result back to the external. When Mark commited an implementaton of an EngineAPI class I decided the perfect thing to test it on was barcode scanning by Intent (mergZXingGetBarcode). The function uses the Barcode Scanner or Barcode Scanner + app by Intent to scan. If the user doesn’t have the app it directs them to the Google Play store to get it.

Once I got into the swing of things I implemented a few more features. Android’s sharing Intent is functionally equivalent to the UIActivityController on iOS (with the exception of an annoying bug in the Facebook app) so I implemented mergPopActivity for Android using that. I also implemented mergAVPick to record and pick video from the media library. A client also required the ability to view a PDF so I implemented mergDocShowPreview to open a file in one of the user’s apps that supports it.

This collection of features is now known as mergAndroid and is currently available as a bonus to mergExt Complete users. The list of features will grow over time as people need things and eventually as the Android Externals SDK matures the goal is to merge these features into the iOS externals available in the suite and attempt to provide as near feature parity as we can.

It’s worth re-iterating that we definitely wouldn’t have mergAndroid yet if LiveCode wasn’t open source which has enabled us to have code level discussions and contributions in order to get something functional. A couple of people have asked how they might get started building Android externals. There isn’t any documentation just yet and eventually I hope to create some templates for setting up an IntelliJ IDEA project with an ant based build but in brief:

• checkout from this link, which is the most up to date until Mark merges my most recent pull request
• build the android engine (there’s a shell script in the tools folder)
• base your external on the sample revtestexternal in the repo
• build using the shell script that comes with revtestexternal

Read the rest of the newsletter here.