Functional features in LiveCode – is it just a dream?

by Alex Brisan on February 8, 2018 No comments

If you’re like me, and you come to LiveCode from more “mainstream” programming languages, such as Java (8 or 9) or Python (heck, even C++), then you might be missing some of the very nice functional features for operating on Arrays and Lists, such as maps, filters or even list comprehensions. In this blog post I’ll try and answer the question, whether implementing functional features in LiveCode is a dream?

Before we get to the code, we need to establish a few prerequisites : the most important element when dealing with these features is the ability to pass around function references to use for execution (or even lambdas in some languages). Now unfortunately, in LCS this is not available natively, so we need to settle for a workaround. Essentially, we will be using the “dispatch” and “do” methods of the language to emulate function references.

Ok, so let’s start by implementing list comprehensions (with quite a number of assumptions, but fully functional within these assumptions). If we look at python’s way of writing a list comprehension, it would be something like

def some_function(array):
	return [x + 2 for x in array] 

The actual list comprehension in this snippet is

[x + 2 for x in array]

It is our goal to replicate this syntax as closely as possible. To be more precise, we would like to write an interface that accepts and evaluates expressions of the following form:

[LCS_EXPRESSION for VAR_NAME in ARRAY_VAR]

Since we’re starting from scratch in LCS implementation, we need to first parse such an expression into its main components. If we dissect it, we can observe that it contains two major components:

The expression that is applied to each element : x + 2
The generator from which we draw the x’s : for x in array

Now, here comes the first assumption that we’re making about the code : the source array must be available in the scope of the handler that contains the code for the comprehension.

Considering this, we can write the following code to parse this expression into its components:


function parseComprehension pComprehension
   local tExpression, tGenerator
   
   set the itemdelimiter to " for "
   
   put char 2 to -1 of item 1 to -2 of pComprehension into tExpression
   
   put char 1 to -2 of item -1 of pComprehension into tGenerator
   
   local tReturnObject, tSourceObjectName, tSourceObject
   
   set the itemdelimiter to space
   
   put item -1 of tGenerator into tSourceObjectName
   
   do "return " && tSourceObjectName in caller
   
   put the result into tReturnObject["Source"]
   
   put tExpression into tReturnObject["Expression"]
   
   return tReturnObject
end parseComprehension

Now as you may very well notice, in this case we are not doing any parsing of the source expression. This is because we assume that that is valid LCS code and so we can just pass the parsing down to the LCS engine by using the “do” syntax. Essentially, the only fancy thing that goes on here is dumping the contents of our source array into a more generically named object to ensure what in specialty terms is called alpha equivalence.

Now that we parsed the expression, we need a way to get the result. A first implementation might look something like:

function executeComprehension pParsedComprehension
   local tResult, x
   
   -- run through all the elements of the array
   repeat with tIdx = 1 to the number of elements in pParsedComprehension["Source"]
      put pParsedComprehension["Source"][tIdx] into x
      
      -- execute the code, and put into its place in the new array
      put value(pParsedComprehension["Expression"]) into tResult[tIdx]
   end repeat
   
   return tResult
end executeComprehension

Now the more attentive reader will notice here that we assume that the variable name is “x”. This is somewhat of an unfair assumption to make, so we can extend the code with a separate function, that takes care of this by replacing whatever name the user gave to the variable with a predefined one.

This function would look something like


function substituteComprehension pExpression, pGenerator
    set the itemdelimiter to "in"

    local tVarName

    put word 1 of item 1 of pGenerator into tVarName

    return replaceText(pExpression, tVarName, "tVariable65537")
end substituteComprehension 

The choice of the variable name with which we make the substitution is tVariable65537, which is just formed of tVariable and a prime number large enough for it to be improbably used in another context.

Again, this is a function that we need to plug in to the parser, an updated version of which would be

 
function parseComprehension pComprehension
    local tExpression, tGenerator, tSubstitutedExpression

    set the itemdelimiter to " for "

    put char 2 to -1 of item 1 to -2 of pComprehension into tExpression

    put char 1 to -2 of item -1 of pComprehension into tGenerator

    -- this is new, use the substitution mechanism to generate a new expression
    put substituteComprehension(tExpression, tGenerator) into tSubstitutedExpression

    local tReturnObject, tSourceObjectName, tSourceObject
    set the itemdelimiter to space

    put item -1 of tGenerator into tSourceObjectName

    do "return " && tSourceObjectName in caller

    put the result into tReturnObject["Source"]

    -- notice here that we’re returning the new expression
    put tSubstitutedExpression into tReturnObject["Expression"]

    return tReturnObject
end parseComprehension 

Updating the runner is a simpler task :

function executeComprehension pParsedComprehension
   local tResult, tVariable65537
   
   repeat with tIdx = 1 to the number of elements in pParsedComprehension["Source"]
      put pParsedComprehension["Source"][tIdx] into tVariable65537
      
      put value(pParsedComprehension["Expression"]) into tResult[tIdx]
   end repeat
   
   return tResult
end executeComprehension

Ok, now in order to give it a more humane interface, let’s create a wrapper. This is something that’s quite easy, we just do:


function comprehension pExpression
   local tComp
   global gExpressionToParse
   
   put pExpression into gExpressionToPars
   do "global gExpressionToParse; return parseComprehension(gExpressionToParse)" in caller
   
   put the result into tComp
   delete variable gExpressionToParse
   
   return executeComprehension(tComp)
 end comprehension 

The only trick here is that in order to get access to the scope of the caller, we need to do some manipulations with globals in order to properly access the variables that we wish.

All in all, this means we can get the result of executing a list comprehension just by doing:

put comprehension("[y * 2 for y in tA]") into tExecutionResult

Which, I hope you would agree is a very close form to what we have in python.

Let’s now have a look at the map function. Map is a function that takes an array and a unary function, returning a new array whose elements are obtained by applying that unary function elementwise to the initial array.

Using the dispatch mechanism, that is quite easy to implement:

function map pArray, pMethodName
   local tResult
   
   repeat with tIdx = 1 to the number of elements of pArray
      dispatch function pMethodName to me with pArray[tIdx]
      
      put the result into tResult[tIdx]
   end repeat
   
   return tResult
end map

Now someone might rightfully argue that implementing 6 lines of code into a separate function might be a waste of time, but speaking from experience I can say that especially as someone who has worked in python for a long time can find the absence of such features quite annoying, as they are baked in our workflow.

To end on a more filtered note, we can go ahead and implement filter, which is a close cousin of map, with one small difference. Filter returns a new array whose elements are members of the initial array that satisfy a given unary predicate. We would implement this something like:

function funcFilter pArray, pPredicateName
   local tResult, tNewCount
   put 1 into tNewCount
   
   repeat for each element tElem in pArray
      dispatch function pPredicateName to me with tElem
      
      if the result then
         put tElem into tResult[tNewCount]
         
         add 1 to tNewCount
      end if
   end repeat
   
   return tResult
end funcFilter

Notice how we named the function funcFilter, because filter is an existing keyword in LiveCode and so it would have clashed.

The sharp reader (or the one who has a lot of experience with functional features) might notice that it is possible to express maps and filters as list comprehensions, so we let’s try and implement that as a final step.

When it comes to maps, it’s quite easy to implement a map version that uses a list comprehension. All we need to do is:


function mapAsComprehension pArray, pMethodName
   put pArray into tBuffer

   local tComprehensionString

   put "["& pMethodName & "(x) for x in tBuffer]" into tComprehensionString

   return comprehension(tComprehensionString)
end mapAsComprehension 

The only special thing that we are doing here is that we need to store a reference to pArray in tBuffer, which is expected to be accessible within the scope of the script that is using this function.

Indeed, we can now see that list comprehensions are just syntactic sugar for maps. Any list comprehension

[f(x) for x in tA]

can be expressed as

map(tA, f)

To use the python notation.

When it comes to filters, things get slightly more complicated, due to the fact that our current implementation of the list comprehension mechanism is limited to simple generators, that do not allow any conditions to be imposed on the elements that are drawn from the source array.

[LCS_EXPRESSION for VAR_NAME in GLOBAL_VAR ]

We can somewhat easily do this by extending the parsing and execution functions like this:

function parseComprehensionWithIf pComprehension
   local tExpression, tGenerator, tSubstitutedExpression
   
   set the itemdelimiter to " for "
   
   put char 2 to -1 of item 1 to -2 of pComprehension into tExpression
   
   put char 1 to -2 of item -1 of pComprehension into tGenerator
   
   put substituteComprehension(tExpression, tGenerator) into tSubstitutedExpression
   
   local tReturnObject, tSourceObjectName, tSourceObject
   
   set the itemdelimiter to space
   
   local tString
   
   put "" into tString
   
   if item 5 of tGenerator is "if" then
      repeat with tIdx = 6 to the number of items of tGenerator
         put item tIdx of tGenerator & space after tString
      end repeat
      
      put substituteComprehension(tString, tGenerator) into tReturnObject["Condition"]
      put item 4 of tGenerator into tSourceObjectName
   else
      put item -1 of tGenerator into tSourceObjectName
   end if
   
   do "return " && tSourceObjectName in caller
   
   put the result into tReturnObject["Source"]
   put tSubstitutedExpression into tReturnObject["Expression"]
   
   return tReturnObject
end parseComprehensionWithIf

And update the execution mechanism to:



function executeComprehensionWithIf pParsedComprehension
    if pParsedComprehension["Condition"] is empty then
        return executeComprehension(pParsedComprehension)
    end if

    local tResult, tVariable65537, tCounter, tChecker

    put 1 into tCounter

    repeat with tIdx = 1 to the number of elements in pParsedComprehension["Source"]
       put pParsedComprehension["Source"][tIdx] into tVariable65537

       do "put " && pParsedComprehension["Condition"] && " into tChecker"
       if tChecker then
          put value(pParsedComprehension["Expression"]) into tResult[tCounter]

          add 1 to tCounter
        end if
    end repeat
    return tResult
end executeComprehensionWithIf

Now, for those of you who might be interested in playing around with this, I’m providing a sample stack that contains all these functions embedded into the stack code.

As a conclusion, I think we’ve definitely established that those people who move over to LiveCode from other languages such as Python can definitely use LiveCode in a not so complicated manner to get some of their creature comforts back, leading me to saying that functional features in LiveCode are NOT a dream 🙂 Indeed, the only bits that seem to be missing are maps/reduces and filtering arrays on more than just regular expressions (functionality that is already built in)

P.S. If you’re looking into switching to LiveCode, make sure to have a look at our cheat sheets (https://livecode.com/resources/cheat-sheets/) 🙂

Alex BrisanFunctional features in LiveCode – is it just a dream?

Join the conversation

*