Async Data Transfer in LiveCode: Problem and Solution

Media Monitoring App

Digital Pomegranate was recently contacted to create a mobile solution for a back end media monitoring system. For those unfamiliar with media monitoring,  you will be amazed how large the industry actually is. They read almost EVERY web page,  tweet, public Facebook post and all other types social media data – ALL THE TIME. They also watch and listen to EVERY TV & Cable show, radio station and even online videos like YouTube.  This is a massive industry with hundreds of thousands of people monitoring consumer’s media consumption and sentiment.

In today’s world, Marketing and Public Relations Professionals need to be aware of trends and potential crises immediately. In our 24 hour news and media cycle, they are expected to be aware of what is being said and how to craft and distribute their client’s message effectively. Whether representing a global corporation, a movie or music star, governments or nonprofit agencies; there is an expectation that someone professional is aware of the situation and be able to reply to whatever the situation demands.

LiveCode to the Rescue

The client is a consortium of smaller local media monitoring agencies who pool their resources together to compete effectively with the large multinational giants. They all update a central system that is then shared to the clients. But the current web based system was not up to the task of getting reports in a timely manner or providing an offline solution which was needed. They required multi platform solution on a small budget and in a short time span. LiveCode was the perfect tool for what we needed. We were able to design the user interface in 72 hours, get it approved and started coding to pull the data from their REST API servers. Everything was fine until we started beta testing with live data.

BIG Bump in the Road

The reason I started explaining the media monitoring industry in this blog post first, was to emphasize the amount of data. Because we did not do our proper homework when we were designing the App to understand the HUGE volumes and volumes of data we were dealing with. In our testing we would download a hundred or so news items at a time and even if it was a little slow, it was not so bad. This was not because of the server, it was because LiveCode’s HTTP calls were being processed in a queue. Doing the processing one at a time, LiveCode took about a second for each call. Well, the first time I logged in as a real customer in our new App and saw 2000 news items, it did not dawn on me what I was in for. That translated to  (2000 / 60) = 30 minutes. How were we going to expect users to tie up their phone and wait half an hour to get their media reports? The client was not going to change their system, so we had to find a solution.

Async Data Transmission

So we had two choices, trash the LiveCode App or get a solution to work in LiveCode. With LiveCode’s new focus on enhancing the core with externals and widgets, we decided to focus on keeping the language we loved, and find an external that could solve our problem. The solution had to be an external that could communicate with the server in an asynchronous manner with callbacks. This is very common in C applications and even in Javascript. The best way to explain the process is to think of the “send” command in LiveCode with messages. You tell the system to do something and it will call you back (it sends you a message) when it is done. You can write code that will keep 50 requests going all at the same time.The solution was tsNet, now acquired by LiveCode Ltd. and as of LiveCode 8.1 available to Indy and Business users. This cut our REST API calls from 30 mins to under 5 and even a bit faster on wifi. 

The good news is, the code is quite simple. 

--Make sure we don't have more than 50 requests at a time

repeat while lAsyncCount > 50

wait for 0 milliseconds with messages

end repeat

--Download the JSON and add it to the Database in 
addNewsItem

put tsNetGet(tItemID, tURL, tHeaders, "addNewsItem", "", "", "") into tError
put lAsyncCount + 1 into lAsyncCount

Externals, a True Lifesaver

I can not emphasize enough the situation we were in and how externals helped us. The new internet external is a requirement for any LiveCode App that communicates with a cloud server. It allowed us to focus on the changes requested by the client and finish in weeks and not months. But not only did we rely on the tsNet external, we also utilized mergPop for social sharing on iOS and mergAndroid for social sharing on Android.

Newspaper clippings are cut and scanned into PDFs, but our client was not satisfied with the user experience with the workaround we created to view them in a browser on Android, which does not open them natively. So again we use externals to load PDF – mergReader on iOS and mergAndroid on the Android. In the past, LiveCode has looked to include functionality in the “core” LiveCode system. Those days are over and just like most developing systems out there, new functionality will be delivered as an external or a widget and users will expand the functionality this way.

I am happy to report that marketing and PR pros now truly have everything at their fingertips because of LiveCode and tsNet, I give it five stars!!!!

The version of LiveCode you are using will determine what networking features you can use in your application. You can find out what is included in which version on the link below.

Compare Networking Features