Last day until RunRevLive.14 kicks off in San Diego. Get Your Ticket >

Desktop Externals

  1. Desktop Externals
    1. Introduction
    2. Setting up the environment
    3. An aside on projects
    4. An aside on build configurations
    5. Our first external – working with text: a take on Hello,
      World!
      1. Setting up the project
      2. Exploring our new project
      3. Getting started with the IDEs
      4. Adding functionality – our first function
      5. The external declarations block – describing the
        functionality the external provides
      6. The function definition – implementing an external
        function
      7. Testing our function – hooking into our
        external
      8. Adding functionality – our first command
      9. The LiveCode externals API – getting and setting string
        variables
      10. Implementing our new command
      11. Testing our command
    6. Integrating an external into the IDE
    7. VariableEx calls – the Swiss-army knife of the
      externals API
    8. Manipulation of binary data
    9. Delving into arrays
    10. A word of caution
    11. Case-Study: an image-effect framework
    12. Background
    13. Getting Started
    14. Abstracting variable access
    15. Introducing effects
    16. Our first effect
  2. Linux Externals
    1. Introduction
    2. Before we begin
    3. Setting up the environment
    4. A simple external – wrapping iconv
      1. Exploring our new project
      2. All about iconv
      3. Sorting out the declarations
      4. Implementing the function
      5. Testing our external
    5. Moving forward
    6. FAQ

Desktop Externals

Introduction

In an ideal world you would be able to solve any computing problem
without leaving the confines of LiveCode. Unfortunately, though, there will
always be things that very-high level programming is not suited for. For
example, an implementation of an algorithm in LiveCode may be too slow, you
may need a specific OS feature that isn’t wrapped by a command or function
in LiveCode, or you may just want to access a pre-existing library in
LiveCode.

To support such cases, LiveCode has what we call the ‘externals
interface’. An external is simply a shared library (dll on Windows, bundle
on Mac OS X) written in a lower-level language that can be loaded (at
runtime) into the LiveCode environment. Once loaded, your scripting
environment is augmented with the commands and functions exported by the
external.

In this, the first of a series of articles, I will cover the basics of
writing a (cross-platform) external for Windows and Mac OS X in C/C++.

Before we begin, before going any further you will need:

  • LiveCode 2.7.x or later (any edition should do)
  • Visual C++ 2005 – any edition including Express (if you want to build
    externals on Windows)
  • XCode 2.4.x (if want to build externals on Mac OS X)
  • Version 2 of the Revolution externals build environment download
  • A set of pre-constructed environments for each step in the tutorial
    article:download
  • Supporting files for this article download
  • A cup of coffee (not strictly required, but I find it helps when coding
    in lower-level languages…)

Note: The need for LiveCode 2.7.x is just because the
External Creator assumes the new installation structure is 2.7 when setting
up projects – the externals you build will operate in any LiveCode version
and edition.

Setting up the environment

Before you can begin doing anything in a lower-level language, it is
important that you set up your ‘build environment’. Modern IDEs such as
Visual C++ and XCode make this straightforward when targetting a single
platform – but it can be trickier if you want to use the same code base to
compile on multiple platforms (or even multiple variants of the same
platform such as exist with Mac OS X).

To help with this, a skeleton environment similar to the system we use
internally is included. This skeleton has the basic structure allowing you
to write externals using both Visual C++ 2005 and XCode 2.4.x.

After unpacking ‘ExternalsEnvironmentV1.zip’ onto a suitable place on
your hard-drive, it is worth spending a few moments familiarising yourself
with its contents:

image1 

The contents you see here serve the following purposes:

  • External Creator V1.rev – a small LiveCode utility to help you
    setup an external project
  • libexternal – this project folder contains the special glue-code that
    all externals need to be linked with in order to bind to LiveCode. (It is a
    project in its own right that builds as a static library).
  • templates – this folder contains template project files used by the
    External Creator to setup new external projects.
  • configurations – this folder contains two kinds of file vsprops and
    xcconfig. These contain settings for different variants of builds on
    different platforms.

In later sections we will refer to this folder as the environment – in
addition to what it currently contains, it will also hold our external
projects themselves.

Note: Although the environment folder will unpack to
ExternalsEnvironmentV1, you are free to rename it to anything.

An aside on projects

Both Visual Studio and XCode rely on the idea of projects to organize
development. A project defines the type of entity to be built along with all
the build parameters and source-files needed to do the build.

In Visual Studio, projects are organized into solutions – a collection of
projects that can reference each other to help organize dependancies. In our
case, our solution will contain one project for each different external, all
which depend on the project libexternal to ensure that the special glue-code
that makes an external an external is available.

In XCode, on the other hand, there is no need for a solution. Any XCode
project can reference another XCode project and depend on what it produces.
However, in the end this amounts to the same thing as Visual Studio –
although the XCode model is sligthly more flexible.

An aside on build configurations

One concept that will be new to you if you haven’t built projects in
lower-level languages before will be that of build configuration. A build
configuration is a collection of settings that define both the type of build
and also the variant of the current platform to build for.

For example, lower-level languages typically require you to compile in a
different way if you wish to debug your code (this is the Debug
configuration in our case) or release your project (this is one of the
Release configurations in our case).

Furthermore, the existence of single OSes targetting different
architectures means that there are generally a multiplicity of Release
builds needed. For example, for Mac OS X, we need ones for both PowerPC and
Intel, as well as one for both (i.e. Universal).

However, you needn’t worry about all the details here as the skeleton
environment comes with a collection of configurations that are completely
compatible with LiveCode (i.e. they should just work!).

Our first external – working with text: a take on Hello,
World!

Setting up the project

The first thing to do is to create a skeleton project for our first
external. Included with the skeleton environment is a LiveCode stack called
External Creator. This little tool helps to start off creating an external
by setting up the project (potentially for multiple platforms) including
directory structure and build configurations. Furthermore, it creates an
empty test stack to load the new external.

So, with the External Creator loaded into LiveCode do the following:

  • fill in the Name field with rnahello (we use the
    three-letter prefix rna to stand for LiveCode Newsletter Article)
  • check the appropriate platforms
  • choose C++ (no exceptions, no rtti) as the language
  • ensure the Installation paths are filled in correctly.

You should end up with something like:

When you have configured all your settings, just click
Generate.

Exploring our new project

Having generated the project, you should now find an rnahello folder
within the environment. It is worth spending a few moments exploring this
folder to see what has been created:

Here we have the following:

  • rnahello.vsproj – the Visual Studio project file (present if you
    checked Windows)
  • rnahello.xcodeproj – the XCode project folder (present if you checks
    Mac OS X)
  • test_rnahello.rev – an empty stack that will load the external
    (used for testing and debugging)
  • src/rnahello.cpp – the outline of our main C++ file that will contain
    the implementation of our commands and functions

So, now we have the structure in place we can actually add some
functionality…

Getting started with the IDEs

Open up Visual C++ or XCode depending on the platform, and load either
externals.sln (into Visual Studio), or rnahello.xcodeproj into XCode. In
both cases you should be presented with (what will, at least, become) a
familiar interface.

Both IDEs use a hierarchical approach to manage projects – in the case of
XCode, the project itself will be the root node, whereas in Visual Studio
the root node will be the solution:

image5 

You may notice in Visual Studio that one of the projects is highlighted
in bold. This signifies which is the current Startup Project – the project
that is built and run whenever you click the Run button. You can change the
startup project by right-clicking on a project node and choosing Set as
Startup Project – this will becomed particularly important for you if you
add new externals to the environment.

My personal preference is to use XCode in all-in-one mode where you
switch between Edit/Build/Debug by the little tab control in the top-left –
this article reflects this preference. By default, however, XCode is not
configured in this mode – to change this select the General pane in
Preferences and choose All-in-one from the Layout option menu.

You may wish to spend some time looking around the IDEs and the
project/solution structure we have created to familiarise yourself with them
before going any further.

Adding functionality – our first function

At this stage, you should be looking at the rnahello project created with
the External Creator. This project has all the appropriate build
configurations configured as well as the necessary setup for launching the
test stack on run so you can debug and/or see your external running –
basically we have a skeleton external that will work, but not actually
provide any functionality to the development environment.

In this section we will add an external function to our external. An
external function is exactly the same as a function handler (defined by the
function keyword) you will be familiar with in LiveCode,
except that it’s implementation is in native code defined in an external. As
with a function handler, an external function takes a number of parameters
as input and returns a value as a result.

The External Creator will have created an outline main file rnahello.cpp
which you may want to familiarise yourself with before continuing.

We will now add our first external function to our project – called
rnahellouser. This will take one parameter (a name) and return a friendly
message.

The external declarations block – describing the
functionality the external provides

The external glue code provided in libexternal provides a number of
C/C++-macros that help in the construction of the external declarations
block that tells LiveCode what the external exports to it.

A block starts with:

EXTERNAL_BEGIN_DECLARATIONS()

This is followed by a sequence of command or functions declarations:

EXTERNAL_DECLARE_FUNCTION(, )

or

EXTERNAL_DECLARE_COMMAND(, )

Then the block finishes with:

EXTERNAL_END_DECLARATIONS

In our case we wish to declare the existence of an external function
called rnahellouser. So, scroll to where the (empty) external declarations
block is in the rnahello.cpp file and insert the line:

EXTERNAL_DECLARE_FUNCTION("rnahellouser",rnaHelloUser)

Between the comments ‘// BEGIN USER DECLARATIONS’ and ‘// END USER
DECLARATIONS’. This will cause a call to a function
rnahellouser in LiveCode to be mapped to a call to the C++
function rnaHelloUser which we will implement next.

The function definition – implementing an external
function

Now we have declared the existence of our external function, we now need
to implement it. For any external handler (command or function), the C/C++
prototype is the same:

void handler(char *p_arguments[], int p_argument_count,
             char **r_result, Bool *r_pass, Bool *r_err)

Here:

  • p_arguments is an array of C-strings of length p_argument_count – each
    element of the array is an argument passed to the external handler.
  • r_result is a pointer to a C-string variable that should be set to a
    pointer to the result of the external handler. (The C-string returned in
    this way must be allocated using malloc/calloc/realloc and ownership of the
    memory will pass to LiveCode.)
  • r_pass is a pointer to a Bool variable that should be set to True if
    the handler invocation should be passed in the message path.
  • r_err is a pointer to a Bool variable that should be set to True if the
    invocation of this handler should raise a runtime error in LiveCode.

Add the following function to the rnahello.cpp file between the // BEGIN
USER DEFINITIONS and // END USER DEFINITIONS:

// Function:
//   rnaHelloUser(pName)
// Parameters:
//   pName – string
// Result:
//   a string containing a friendly message
//

void rnaHelloUser(char *p_arguments[], int p_argument_count,
                  char **r_result, Bool *r_pass, Bool *r_err)

// First check we have been passed a single argument – if not it’s an
error
//
if (p_argument_count != 1)
{
*r_result = strdup("wrong number of parameters");
*r_err = True;
*r_pass = False;
return;
}
// Next compute the length of our required string
// = length of first argument (p_arguments[0]) + length of "Hello, !" + 1
//
unsigned int t_buffer_length;
char *t_buffer;
t_buffer_length = strlen(p_arguments[0]) + 8 + 1;
t_buffer = (char *)malloc(t_buffer_length);
if (t_buffer == NULL)
{
*r_result = strdup("out of memory");
*r_err = True;
*r_pass = False;
return;
}

// We have allocated our buffer – so now construct our string
//
sprintf(t_buffer, "Hello, %s!", p_arguments[0]);

// t_buffer now contains a pointer to the result so just return...
//
*r_result = t_buffer;
*r_err = False;
*r_pass = False;
}

In this function we have used a number of standard library functions
which we will need to include in our environment. In C/C++ this is done via
#include directives, so add the following to the
rnahello.cpp file before the #include directive.

#include <cstdlib>
#include <cstdio>
#include <cstring> 

This imports memory management (cstdlib), i/o (cstdio) and string
manipulation (cstring) functions into our file.

All being well, you should now be able to compile your external and
launch it within LiveCode…

In Visual Studio:

  • Ensure rnahello is the Startup Project
  • Make sure Debug is chosen in the Solution Configurations drop-down list
    (on the toolbar)
  • Click the green Run button

Note: Due to the relaunch feature added to recent
versions of LiveCode, you need to make sure the version of LiveCode you are
debugging externals in is not already running when you launch a debug
session from Visual Studio

Note: You will likely get a message saying ‘Unable to
find debug symbols for LiveCode.exe’ the first time you run your
application – do not worry about this, just click to continue and choose not
to show the dialog again.

In XCode:

  • Choose Build mode (either by choosing Build Results from the Build
    menu, or by clicking on the Build icon in the Page tab in the in the
    top-left).
  • Choose rnaHello from the Target drop-down
  • Choose Release from the Configuration drop-down
  • Choose Build
  • Now choose the Debug configuration and click Build and when it has
    finished click Debug. (The previous two steps shouldn’t be necessary,
    but there seems to be a glitch in XCode that prevents it picking up the
    correct settings for launching the debug executable if you have only ever
    built a Debug variant of a project – from now on you won’t need to build a
    release variant before a debug one in this project).

Assuming the above steps were followed correctly, after a short while an
empty stack called rnaHelloTest should pop-up inside the development
environment.

Testing our function – hooking into our external

At this point you should be looking at the LiveCode IDE with a stack
opened up. This stack should have already loaded the external into the
LiveCode environment – so now we just need to hook into it.

So do the following:

  • First add a field called Name to the stack
  • Add a button called rnaHelloUser to the stack and place the following
    into its script:
on mouseUp
answer rnaHelloUser(field "Name")
end mouseUp

Now enter any string into the field and click the button – you should get
an answer dialog popping up with a friendly message.

Before continuing, make sure you save your test stack (we will be
building on it in the next section) and then quit LiveCode.

Adding functionality – our first command

In the previous section, we looked at adding an external function.
In this section we will look at adding an external command instead. Just
with external functions, external commands are effectively the same as
command handlers (defined with the on keyword) in LiveCode
except that their implementation resides in a external.

In reality, there is very little difference between external commands and
functions (just like there is little difference between command and function
handlers in LiveCode). Indeed, the difference is mainly syntactic, and the
fact that you have to use the result to get the return
value of a previously executed command. Therefore, we will use this
opportunity to introduce how to set and get local variables in an external
handler.

The command we will implement will be called rnaHelloUserIndirect and
will have syntax:

rnaHelloUserIndirect pUserVariableName,
pOutputVariableName

Here pUserVariableName will be the name of a (handler)
local variable from which to fetch the user’s name, and pOutputVariableName
will be the name of a (handler) local variable in which to
put the result.

The LiveCode externals API – getting and setting string
variables

In addition to providing the necessary ‘glue’ to allow a shared library
to be loaded as a LiveCode external, ‘libexternal’ also exports a number of
functions that allow some access to the internals of the currently running
LiveCode application.

In this section will make use of two of these functions GetVariable and
SetVariable. These functions give you access to the contents of the local
variables present in the context that called the external handler – as long
as the value of those variables are text strings (i.e. not binary data).

The prototypes of these functions are:

char GetVariable(const char p_variable_name, int
r_success)
void SetVariable(const char p_variable, const char p_value, int
r_success)

The GetVariable call takes a name of a variable (as a C-string) and a
pointer to a return variable. If the call succeeds, *r_success will be
EXTERNAL_SUCCESS and a copy of the value of the variable as a C-string will
be returned. If the call fails, *r_success will contain EXTERNAL_FAILURE and
NULL will be returned.

Note: The ownership of the memory returned passes to you
– i.e. you must free it when you are finished with it.

The SetVariable call takes a name of a variable, the value you wish to
set the variable to and a return variable. If the call succeeds *r_success
will contain EXTERNAL_SUCCESS and the specified variable will be set to the
required value. If the call fails *r_success will contain EXTERNAL_FAILURE
and no other change will have occurred.

Note: LiveCode copies the string you pass to it in
p_value so there is no concern about ownership – i.e. if you’ve allocated
memory to store it, you still have to free it.

Implementing our new command

As before, we first have to declare the existence our new command by
adding an appropriate entry in the external declarations block. Therefore,
add the following line in the ‘USER DECLARATIONS’ as before:

EXTERNAL_DECLARE_COMMAND("rnahellouserindirect",
rnaHelloUserIndirect)

Next we have to provide an implementation of this command which we do by
adding the following to the ‘USER HANDLER DEFINITIONS’ section of the
file:

// Command:
//   rnaHelloUserIndirect pUserVariableName, pOutputVariableName
// Parameters:
//   pUserVariableName – name of a variable in local context to get the name
from
//   pOutputVariableName – name of a variable in local context to put the
result into
//
void rnaHelloUserIndirect(char *p_arguments[], int p_argument_count,
                          char **r_result, Bool *r_pass, Bool *r_err)
{
// In this command we have to keep track of several (memory) resources. To
help with this
// we use a variable 't_error' that determines whether an error has occurred.
This is used
// throughout the function to determine whether to continue processing, and
at the end to
// decide whether to return an error or not.
//
const char *t_error;
t_error = NULL;

// Check to see if we have been passed two parameters
//
if (t_error == NULL && p_argument_count != 2)
t_error = "wrong number of parameters";

// Fetch the value of the 'name' variable (argument 1)
// Remember that the return value of 'GetVariable' becomes 'ours' so we have
to
// free it later if the call succeeds.
//
char *t_name;
t_name = NULL;
if (t_error == NULL)
{
int t_success;
t_name = GetVariable(p_arguments[0], &t_success);
if (t_success == EXTERNAL_FAILURE)
t_error = "unable to get value of name variable";
}

// Allocate memory for the return message to be put into the output
variable
// (If this step succeeds, we need to free memory pointed to by t_message
later)
//
char *t_message;
t_message = NULL;
if (t_error == NULL)
{
t_message = (char *)malloc(strlen(t_name) + 8 + 1);
if (t_message == NULL)
t_error = "out of memory";
}

// Format our message appropriately
//
if (t_error == NULL)
sprintf(t_message, "Hello, %s!", t_name);

// Set the value of the output variable
// Remember that the ‘SetVariable’ API call copies the contents of t_message
and so we still
// have to remember to free it ourselves.
//
if (t_error == NULL)
{
int t_success;
SetVariable(p_arguments[1], t_message, &t_success);
if (t_success == EXTERNAL_FAILURE)
t_error = "unable to set output variable";
}

// Free the message memory buffer if it was ever allocated
//
if (t_message != NULL)
free(t_message);

// Free the name memory buffer if it was ever allocated
//
if (t_name != NULL)
free(t_name);

// If we succeeded, return an empty result and no error, otherwise return a
copy
// of the error message and flag as an error.
// (NB: We return a pointer to an empty string here because pre-2.5 versions
of LiveCode
//  would crash if you returned a NULL pointer in r_result)
//
if (t_error == NULL)
{
*r_result = strdup("");
*r_pass = False;
*r_err = False;
}
else
{
*r_result = strdup(t_error);
*r_pass = False;
*r_err = True;
}
}

Again, you should now be able to Run (Visual Studio), or Build and Debug
(XCode) your project and after a few moments, the test stack will appear
again.

Testing our command

You should again find yourself staring at our test stack, loaded into the
LiveCode IDE. Add a new button to the stack called rnaHelloUserIndirect and
place the following in its script:

on mouseUp 
   local tName, tOutput
   put field "Name" into tName
   rnaHelloUserIndirect "tName", "tOutput"
   answer tOutput
end mouseUp

Notice here that we are passing strings to rnaHelloUserIndirect that
contain the names of the variables that we wish to access
in the external, rather than their values.

Clicking on rnaHelloUserIndirect should now give you exactly the same
message as clicking rnaHelloUser – our new command works!

Integrating an external into the IDE

The External Creator helpfully builds us a stack that automatically loads
our external when it is opened through either XCode or Visual C++. However,
when you get to a point when you wish to start testing any externals you
create in the context of actual LiveCode projects, you will need to be able
to integrate them with the IDE.

To do this, you will first want to build Release builds of your
externals. In Visual Studio, simply choose the Release configuration from
the drop-down list on the toolbar, while in XCode you need to switch to the
build pane and choose one of Release, Release x86-32 or Release
PowerPC-32.

When you have done this, all you need to do then is copy them into
appropriate places in your Documents folder for LiveCode to pick up next
time it is launched. This can be done by creating the following hierarchy
inside the standard Documents folder:

My LiveCode /
   Externals/
   Runtime/
           Windows/
                x86-32/
                    Externals/
            Mac OS X/
                Universal/
                    Externals/
                PowerPC-32/
                   Externals/
               x86-32/
                   Externals/

Then inside each Externals folder, put the appropriate build of the
external along with a text file called Externals.txt. The externals.txt file
is return-delimited list of pairs:

For example for rnahello on Windows the file should contain:

Hello External,rnahello.dll

Whereas on Mac OS X it should contain:

Hello External,rnahello.bundle

Note: You only need the Runtime hierarchy for building
Standalones containing your externals. Furthermore, you need only create the
architecture folders within each platform folder for the architectures that
you need.

Note: In general, you should always build a Release
(Universal) external on Mac OS X to place in the non-Runtime Externals
folder – as this is the folder the IDE uses.

VariableEx calls – the Swiss-army knife of the
externals API

In our rnaHello external we created earlier, we saw how to use the API
calls GetVariable and SetVariable. These calls are fine if you wish to
manipulate small amounts of text data, but because they operate using
NUL-terminated strings, they are completely useless if you want to work with
binary data. Furthermore, they give no way of accessing variables that might
be arrays. Therefore, to deal with such cases, it is necessary to become
familiar with a related pair of functions GetVariableEx and
SetVariableEx.

This pair of functions have three advantages over their simpler
cousins:

  • The value you pass in and out is a pair consisting of a data pointer
    and a length – i.e. there is no reliance on a NUL-terminator to determine
    the length of the data.
  • You can specify a key, enabling you to access values of individual
    elements of the array.
  • The data returned by GetVariableEx is not copied, resulting in a
    performance gain when fetching large blocks of data (and you don’t have to
    worry about freeing anything).

The syntax of these functions is a natural extension of GetVariable and
SetVariable:

void GetVariableEx(const char p_name, const char
p_key, ExternalString r_value, int r_success)

void SetVariableEx(const char p_name, const char p_key, const
ExternalString p_value, int r_success)

Notice here that there is an extra p_key parameter and rather than the
value being a char *, it is now an ExternalString *.

Manipulation of binary data

As mentioned above, one of the advantages of this pair of functions is
that they allow you to work with arbitrary binary data. This comes about
because a value is passed and returned as a pointer to an ExternalString
structure. This structure is defined with two members:

struct ExternalString
{
const char *buffer;
int length;
}

Where buffer is a pointer to the data, and length is its length in bytes.
For example to get the value of a variable called tBinaryData one would do
something like:

ExternalString t_value;
GetVariableEx("tBinaryData", "", &t_value, &t_success);

And to set the value of a variable called tBinaryData to some
pre-existing value, one would do something like:

ExternalString t_value;
t_value . buffer = (const char *)t_my_buffer;
t_value . length = t_my_buffer_length;
SetVariableEx("tBinaryData", "", &t_value, &t_success);

Here, you will notice we pass the empty string (“”) as the second
argument – the p_key parameter. This tells the engine to return the value of
the given variable as if it were a normal (non-array) variable.

One thing to be wary of when using these forms is that the data you get
back by using GetVariableEx will not (in general) be NUL terminated, even if
it is ‘text’ in some sense. Importantly this means you should take great
care when using the standard C string manipulation routines.

Delving into arrays

The other main use of VariableEx function pair is to access elements of a
LiveCode array variable. To do this, simply pass the name of the key you
wish to access in the p_key parameter. For example, if you want to get
element foo of variable tBar one would do something like:

ExternalString t_value;
GetVariableEx("tBar", "foo", &t_value, &t_success);

Or if you want to get element n of variable tBaz, you would need to do
something akin to:

char t_key[16];
sprintf(t_key, "%d", n);
GetVariableEx("tBaz", t_key, &t_value, &t_success);

Here you are seeing a side-effect of LiveCode arrays really being
hash-tables with string-based keys – even numeric based arrays require you
to explicitly convert the index to a string before fetching.

One question you might be caused to ask is – what happens if I try to get
or set an element that doesn’t exist? The answer is exactly the same as what
happens when you try to do that in LiveCode: in the case of setting, the
element gets created; in the case of getting you get empty back (in this
case empty is a buffer of length 0). In particular, r_success only reports
EXTERNAL_FAILURE if the variable itself does not exist.

A word of caution

As the third benefit of the VariableEx calls, we mentioned the increased
efficiency gained by the data not being copied when the external fetches it
from LiveCode. This is all well and good, but does come with a small word of
warning: any changes to the variable, or the referenced variable’s element
will cause the pointer that is returned to become invalid.

For example, the following code is likely to cause a crash:

ExternalString t_original_value;
GetVariableEx("tBar", "", &t_original_value, &t_success);
SetVariable("tBar", "Hello World!", &t_success);
if (t_original_value . length == 5 && memcmp(t_original_value . buffer,
"hello", 5) == 0
printf("It starts with hello");

This is because the engine will free the memory referenced in
t_original_value during the course of the SetVariable call, and so when a
pointer to it is used in the memcmp call there is no guarantee it will still
be valid.

However, don’t get too concerned about this. If you keep to the general
principal of fetching input from input variables, doing the necessary
processing and then writing back the results to the output variables it is
unlikely that you will end up with any dangling pointers.

Case-Study: an image-effect framework

So far this article has been somewhat dry and it is about time we got our
hands dirty. Rather than produce a small and essentially useless external
just to exhibit the introduced APIs, we will instead explore a slightly more
meaty example of an external.

Background

What we will be producing in the remainder of this article is a basic
image effect framework – an external with a simple API allowing you to apply
effects to the image and alpha data of images. While simple, the framework
has been designed to be easily extensible, requiring little more than
implementing the algorithm that actually implements the effect you want.

The framework exports just two external handlers:

  • rnaEffectList – an external function returning a list of all known
    effects
  • rnaEffectApply – an external command taking details of an effect and an
    input image, and outputting a processed image

The external makes extensive use of both the array and binary
manipulation features of the externals API. Indeed, from the point of view
of LiveCode, effects and images are defined by arrays having certain sets of
keys. The external then uses these arrays to dispatch to the appropriate
effect and provide it with the correct data.

The framework is written using C++ and makes use of both abstract base
classes for extensibility, and exceptions for implicit error-handling.

Getting Started

Before continuing we need to setup a new external. So, make sure you have
an unpacked Externals Environment (the one from the last article will do)
and load up the External Creator. Then, generate a new external with the
following settings:

  • Name – rnaEffect
  • Platform – as appropriate
  • Language – C++

When this has been generated, unpack the NewsletterArticle2.zip archive
and copy the source files from within the rnaeffect_skeleton/src folder into
the rnaEffect/src folder inside your externals environment. The files you
have just copied is the framework itself albeit without any implemented
effects. (Don’t worry about overwriting the ‘rnaeffect.cpp’ file, you want
the one from the archive – it actually does something!).

Now load the new project into XCode or Visual C++, depending on your
current platform.

The next thing to do is to add the auxillary files to the project. This
can be done in one of several ways – the easiest is probably by using
drag-drop. Select all the source files (both CPP and H) in the src folder
and drag them onto the project.

In Visual C++, you should drag them onto the rnaEffect project node.

In XCode, drag them into the Source node under the rnaEffect node and just
choose Add when you get prompted with a dialog asking you which targets to
add them to. Be careful you don’t get a duplicate rnaeffect.cpp node – if
you do, just choose one, select Edit -> Delete, and then Delete
References.
These new files serve the following purposes:

  • rnaeffect.cpp – this contains the main implementation of the external,
    containing two external handler definitions
  • effect.cpp/effect.h – this pair of files define an abstract class
    Effect which you can inherit from in order to implement an effect
  • variable.cpp/variable.h – this pair of files define an abstraction of a
    LiveCode variable making them easier to work with (see the next section)
  • utility.cpp/utility.h – this pair of files declare a number of utility
    functions and classes making development easier

It is probably now worth taking a brief look through the files in the
project. Most source files have been heavily commented to help explain what
they are for. In particular have a look through the implementation of the
external handlers in rnaeffect.cpp.

Abstracting variable access

If you have taken a look through rnaeffect.cpp then you will notice that
there is not a single mention of SetVariableEx or GetVariableEx. Indeed,
having introduced our friends GetVariableEx and SetVariableEx, they have now
been rendered obsolete by wrapping them in an abstraction – the Variable
class. At this point take a moment to browse through the Variable.h header
file.

The idea of this class is simple – you declare a variable of type
Variable and pass a variable name to its constructor. This object is then
(notionally) linked to the LiveCode variable and you can use a rich
collection of methods to manipulate its value.

For example, to fetch the value of a LiveCode variable call tBar as an
integer you would do:

Variable t_bar("tBar");
int t_value;
t_value = t_bar . GetInteger();

Or, to set the value of element foo of variable tBaz to the string “Hello
World!” you would do:

Variable t_baz("tBaz");
t_baz . SetCStringElement("foo", "Hello World!");

As you can see, this significantly simplifies fetching and storing
variable values particularly as any required type conversions are done for
you by the wrapper class.

One point worth mentioning is that this version of the Variable class
relies on exceptions to handle errors. i.e. If a variable doesn’t exist, or
doesn’t contain a string that can be coerced to the requested type an
exception will be thrown.

Being just a ‘wrapper’ class, Variable’s implementation is
straightforward and those of you interested in the details may want to take
a good look through the Variable.cpp class to see how it works. (It really
is nothing more than a thin wrapper around the SetVariableEx and
GetVariableEx calls).

Introducing effects

An effect is a class derived from the abstract class Effect. The Effect
class has two pure virtual methods – Calculate and Apply – that must be
implemented. In addition, a means by which to create an instance of the
effect from a LiveCode array must be provided.

The Calculate method is used by the framework to work out how big an
output image will be given an input image of a given size. It has
signature:

void Calculate(unsigned in p_in_width, unsigned int p_in_height,
unsigned int& r_out_width, unsigned int& r_out_height)

The implementation must use p_in_width and p_in_height together with any
parameters passed to the effect when it was created to compute the output
size which should then be stored in r_out_width and r_out_height.

The Apply method is the workhorse of any effect, actually performing the
effect’s operation on an input image and putting the output into an output
image. It has signature:

void Apply(const Image& p_input, Image& p_output)

From the point of view of the framework, an Image is a quadruple:

struct Image

unsigned int width;
unsigned int height;
unsigned char <em>color;
unsigned char </em>alpha;
};

A structure containing the images width and height in pixels, together
with pointers to memory buffers containing the color (pixel) and alpha
(mask) data. These data buffers are in the same format as the imageData and
alphaData properties of a LiveCode image.

The framework takes care of fetching the input data and allocating the
output buffers, so all the Apply method needs to do is take the input and
write it to the output after it has applied its magic.

Our first effect

Our first effect will be the ‘no-op’ effect. It will take an input image
and map it directly to the output image without changing it in any way.
While not actually useful, it is the simplest possible effect we could
implement and will illustrate how to add an effect perfectly.

The first thing we need to do is add the definition and implementation of
the NopEffect class to our project.

In Visual C++, click the right mouse-button on the rnaeffect node of the
project and choose Add -> New Item…. Choose ‘C++ file (.cpp)’, give it
the name nop_effect.cpp, and make sure it is being created in the src folder
(check the Location field). Repeat this again but choose ‘Header file (.h)’
and give it the name nop_effect.h.

In XCode, right click on the Source node in the project, choose New
File… and select Empty File in Project. Give it the name nop_effect.cpp
and make sure it is created in the src folder. Repeat this again but add a
file called nop_effect.h. (In both cases, you do want it added to the
rnaeffect target – which will be the default setting).

Open up the nop_effect.h file and paste in the following:

#ifndef __NOP_EFFECT__
#define __NOP_EFFECT__

#ifndef __EFFECT__ 
#include "effect.h"
#endif

class NopEffect : public Effect

public:
NopEffect(const char *p_info_variable);

void Calculate(unsigned int p_in_width, unsigned int p_in_height,
unsigned int& p_out_width, unsigned int& p_out_height);

void Apply(const Image& p_input, Image& p_output);
};

#endif

This is nothing more than the declaration of our new class, derived from
Effect.

Now in the nop_effect.cpp we need to define three methods – but first we
need to ensure we include the relevant definitions. At the top of the file
place the following includes:

#include 

#include "effect.h"
#include "nop_effect.h"
Follow this by the method implementations. First the constructor:

NopEffect::NopEffect(const char *p_info_variable)
{
}

Since our effect does nothing to any given image, it has no parameters
and so the constructor need not do anything.

Next, we need the Calculate method:

void NopEffect::Calculate(unsigned int p_in_width,
unsigned int p_in_height,
unsigned int& p_out_width, unsigned int& p_out_height)
{
p_out_width = p_in_width;
p_out_height = p_in_height;
}

Again, our effect has no effect, so we just copy the input width and height
into the output width and height.

Finally, the main part – the Apply method:

void NopEffect::Apply(const Image& p_input, Image&
p_output)
{
if (p_input . color != NULL)
memcpy(p_output . color, p_input . color, p_input . width p_input .
height  4);
if (p_input . alpha != NULL)
memcpy(p_output . alpha, p_input . alpha, p_input . width * p_input .
height);
}

Again, our effect does no processing so we just copy the input data to
the output data. Here it is worth noting two things. First, if either color
or alpha data is not present in the input (indicated by the buffer being
NULL), it cannot be present in the output. Second, the size of the buffers
are one byte per pixel for alpha, and four bytes per pixel for color – just
the same as the corresponding data properties of images in LiveCode.

Having implemented the NopEffect class, all that remains is to hook it
into the framework. To do this, open up the effect.cpp file and locate the
// BEGIN USER EFFECT INCLUDES line. After this line put:

#include "nop_effect.h"

This will cause the declaration of our new effect to be imported. Then
find the

// BEGIN USER EFFECTS line and after it put the entry:

{ "nop", EffectCreate },

This line adds an entry to an array that tells the framework to create an
instance of the NopEffect class whenever an effect with name nop is
requested.

Having done all of this, click Run (Visual C++), or Build followed by
Debug (XCode) and, all being well, an empty test stack should appear.

In Visual C++, make sure that rnaeffect is selected as your ‘Startup
project’.
Remember that in XCode, you need to do a release build before a debug build
the first time a project is compiled – otherwise when you debug, the paths
will be wrong and the external won’t get loaded.

Using the framework
You should now be looking at a blank stack ready to start testing our
effects framework and new effect. First of all, let’s check that our effect
is recognized.

Switch to the Message Box and ensure that the rnaEffectTest stack is the
current target. Then execute put rnaListEffects(). You should see a list
consisting of one item nop (if you don’t, retrace your steps and ensure that
both nop_effect.cpp and nop_effect.h are in the project, and that the
s_effects array in effect.cpp has a ‘nop’ entry).

As mentioned before, the framework works by using arrays to communicate.
The rnaApplyEffect external command takes three parameters:

  • pEffectInfoName – the name of an array variable describing the required
    effect
  • pInputImageName – the name of an array variable describing the input
    image
  •  pOutputImageName – the name of a variable to receive the output image
    description

The keys required in the effect array will depend on the effect, but at
the very least there should be one key call name containing the name of the
effect to invoke.

The array describing the input image requires the following keys:

  • width – the width of the image in pixels
  • height – the height of the image in pixels
  • color – the imageData of the image (optional)
  • alpha – the alphaData of the image (optional)

Similarly, the output image variable will contain an array with similar
keys.

So to try out the nop effect, import a suitable image onto the test
stack – any image will do – and call it Input. Then add an additional empty
image object (by dragging from the tools palette) and call it Output.
Finally, add a button called Nop with the following script:

on mouseUp
   local tEffect
   put "nop" into tEffect["name"]
  
   local tInput
   put the width of image "Input" into tInput["width"]
   put the height of image "Input" into tInput["height"]
   put the alphaData of image "Input" into tInput["alpha"]
   put the imageData of image "Input" into tInput["color"]
 
   local tOutput
   rnaApplyEffect "tEffect", "tInput", "tOutput"
   
   lock screen
   set the width of image "Output" to tOutput["width"]
   set the height of image "Output" to tOutput["height"]
   set the alphaData of image "Output" to tOutput["alpha"]
   set the imageData of image "Output" to tOutput["color"]
   unlock screen
end mouseUp

Clicking the button should result in the output image turning out to be
identical to the input image! At this point save the test stack and quit
LiveCode.

An image adjustment effect

Now having implemented a completely useless effect, let us build
something slightly more interesting – a simple implementation of contrast
and brightness adjustment. This effect will be called adjust and take two
parameters brightness and contrast.

Add a pair of files to the project adjust_effect.cpp and
adjust_effect.h.

Next, copy the following into the adjust_effect.h

#ifndef __ADJUST_EFFECT__
#define __ADJUST_EFFECT__

#ifndef __EFFECT__
#include "effect.h"
#endif

class AdjustEffect: public Effect
{
public:
AdjustEffect(const char *p_info_variable);

void Calculate(unsigned int p_in_width, unsigned int p_in_height,
unsigned int& p_out_width, unsigned int& p_out_height);
void Apply(const Image& p_input, Image& p_output);

private:
int m_brightness;
int m_contrast;
};

#endif

This defines another derivation of Effect. This time, though it has some
state – m_brightness and m_contrast. These two integral values will be
extracted from the effect array when the an AdjustEffect object is
created.

Next open up the adjust_effect.cpp file and start off by adding the
following includes:

#include

#include "variable.h"
#include "effect.h"
#include "adjust_effect.h"

After this we need our three methods. First the constructor:

AdjustEffect::AdjustEffect(const char *p_info_variable)<br
/>
{<br />
Variable t_info(p_info_variable);
m_contrast = t_info . GetIntegerElement("contrast");
m_brightness = t_info . GetIntegerElement("brightness");
}

Here, we first wrap the info variable name in a Variable object, then fetch
two integer values from the keys contrast and brightness. These values are
stored in the private state of the AdjustEffect object so the other two
methods have access to them.

Next, we need the Calculate method:

void AdjustEffect::Calculate(unsigned int p_in_width,
unsigned int p_in_height,
unsigned int& p_out_width, unsigned int& p_out_height)

p_out_width = p_in_width;
p_out_height = p_in_height;
}

As before, this effect does not change the size of the image and so we
just copy input size to output size.

Finally, we need the implementation. As previously mentioned, the image
data is made available to the Apply method as (up to) two pointers to memory
buffers. These buffers contain data in exactly the same format as Revolution
image and alpha data properties – that is one byte-per-pixel for alpha data
and four bytes-per-pixel for image data (the latter being in sequence pad,
red, green, blue for each pixel). Therefore, applying an effect is a simple
matter of looping over the input data in an appropriate way, and then
writing the processed data to the provided output buffer.

In this case, we will be applying the following function to the red,
green and blue components of each pixel:

new_value = min(0, max(255, ((value - 128) * contrast) /
128 + 128 + brightness))

This is nothing more than a simple-minded contrast and brightness
adjustment – common in many image-processing applications.

Therefore, we can use something like the following to achieve our desired
effect:

void AdjustEffect::Apply(const Image& p_input, Image&
p_output)

// Process the color data
//
if (p_input . color != NULL)
{<br />
const unsigned char <em>t_in_ptr;<br />
t_in_ptr = p_input . color;<br />
<br />
unsigned char </em>t_out_ptr;<br />
t_out_ptr = p_output . color;<br />
<br />
// Since this effect is not sensitive to row or column, we just iterate<br
/>
// through each pixel without regard for its x or y location.<br />
//<br />
for(unsigned int p = 0; p &lt; p_input . width <em> p_input . height;
++p)<br />
{<br />
// Skip the pad byte in both input and output<br />
t_in_ptr++;<br />
t_out_ptr++;<br />
<br />
// Since this effect is not sensitive to channel, we just iterate<br />
// through 3 channels for each pixel - red, green and blue.<br />
//<br />
for(unsigned int c = 0; c &lt; 3; ++c)<br />
{<br />
int t_in_value;<br />
t_in_value = </em>t_in_ptr++;<br />
<br />
int t_out_value;<br />
t_out_value = ((t_in_value - 128) <em> m_contrast) / 128 + 128 +
m_brightness;<br />
if (t_out_value &lt; 0)<br />
t_out_value = 0;<br />
else if (t_out_value &gt; 255)<br />
t_out_value = 255;<br />
<br />
</em>t_out_ptr++ = t_out_value;<br />
}<br />
}<br />
}<br />
// We have no effect on the alpha data, so we just copy it straight<br />
// through.<br />
//<br />
if (p_input . alpha != NULL)<br />
memcpy(p_output . alpha, p_input . alpha, p_input . width *p_input .
height);<br />
}

With the adjust_effect.cpp file finished, all that remains is to add:<br
/>

#include "adjust_effect.h"

to the appropriate place in effect.cpp. Followed by an entry in the
effects table in the same file:

{"adjust", EffectCreate },

Now rebuild and run the project.

Using the adjust effect

Let us now hook into our new effect.

First of all, to the test stack add two sliders: one Brightness with
range -127 to 127; and one Contrast with range 0 to 255. Then create a new
button Adjust and give it the following script:

on mouseUp 
   local tEffect
   put "adjust" into tEffect["name"]
   put the thumbPosition of scrollbar "Brightness" into \
   tEffect["brightness"]
   put the thumbPosition of scrollbar "Contrast" into \
   tEffect["contrast"]

   local tInput
   put the width of image "Input" into tInput["width"]
   put the height of image "Input" into tInput["height"]
   put the alphaData of image "Input" into tInput["alpha"]
   put the imageData of image "Input" into tInput["color"]
   
   local tOutput
   rnaApplyEffect "tEffect", "tInput", "tOutput"<br />
   <br />
   lock screen<br />
   set the width of image "Output" to tOutput["width"]<br />
   set the height of image "Output" to tOutput["height"]<br />
   set the alphaData of image "Output" to tOutput["alpha"]<br />
   set the imageData of image "Output" to tOutput["color"]<br />
   unlock screen<br />
end mouseUp

Now try playing around with the sliders and clicking Adjust – you should
see a familiar effect!

Building on the framework

The final code for the framework and effects decribed here can be found
in the rnaeffect_final folder in the NewsletterArticle2.zip archive.

Linux Externals

Introduction

In this chapter, we will introduce an updated version of the External
Creator and show how to use it to produce a simple external that can be used
on Linux.

Before we begin

Before going any further you will need:

  • At least Revolution 2.9-dp-1 (Beta up and running on a recent Linux
    distribution
  • GCC and related tools installed on your Linux system (many
    distributions install these as standard, but if they are not there, install
    them using your distribution’s package management system)
  • Version 3 of the LiveCode externals build environment:<a
    href=”http://developer.runrev.com/externals/ExternalsEnvironmentV3.zip”>down
    load
  • Supporting files for this chapter <a
    href=”http://developer.runrev.com/externals/NewsletterArticle3.zip”>download

Setting up the environment

As with the other platforms, before doing anything in a lower-level
language on Linux you need to set up your build environment appropriately.
On Windows and Mac OS X there are obvious choices for an IDE to build your
external in, however on Linux there is no clear winner in this regard and so
we have opted for a simple Makefile system.

To help with this, the new version of the Externals Environment and
External Creator is able to generate Makefiles in a simple structure for use
on Linux, and with a few simple terminal commands you can be up and running
with Linux external development in no time.

So, before continuing take some time to unpack the
ExternalsEnvironmentV3.zip file into a suitable place on your hard-drive and
take a quick look through its contents. Also feel free to rename the
unpacked folder ExternalsEnvironmentV3 to something more to your liking -
but make a note of its location as this will be needed later when compiling
and running your externals. (In future, we will refer to this folder as the
environment folder).

A simple external – wrapping iconv

Setting up the project
As an introduction to external development for Linux, we are going to wrap a
simple system library call (iconv) that is used for text conversions
allowing access to its features inside LiveCode.

Having unpacked the environment and put it in a suitable place, we now
need to generate the skeleton files for our external. So, without further
ado, load up your copy of LiveCode on Linux and open the ‘External Creator
V3.rev’ stack inside your environment folder.

With this stack loaded, do the following:
fill in the Name field with rnaiconv (as in other external
articles, we using the three-letter prefix rna to stand for LiveCode
Newsletter Article)

  • choose Linux as the target platform
  • choose C++ (no exceptions, no rtti) as the language
  • ensure that the Linux installation path is correct

Once you are happy with your settings, just click
Generate and the creator will chug away for a moment
producing the necessary files.

Exploring our new project

Having generated the project, you should now find an rnaiconv folder
within the environment, as well as two files in the root of the environment
folder – Makefile and run (more on these later).

Taking a look in the rnaiconv folder should reveal a number of files and
folders:

  • rnaiconvtest.rev – an empty stack that will load the external (used for
    testing and debugging)
  • Makefile – the external’s Makefile which will be read by the make
    command to build the external
  • src/rnaiconv.cpp – the outline of the main C++ file that will contain
    the implementation of our functions

Now we have this basic structure we can go onto actually implementing
some functionality…

All about iconv

The iconv command is ubiquitous on Linux systems. This flexible tool
allows you to convert text files between different encodings on the command
line (to find out more about this type man iconv at a terminal prompt).<br
/>

As it turns out, this tool simply wraps a system library call that
provides this ability – and our external will give us direct access to this
ability without need to resort to the shell or open process LiveCode
syntax.

The iconv library consists of three functions:

  • iconv_open – create an iconv_t converter object between the source and
    target encodings
  • iconv_close – destroy a previously created iconv_t converter object
  • iconv – the function that actually does the conversions

To use it is simple, you first open the converter with the appropriate
source and target encodings, do your conversion, then close the converter.
We will wrap this sequence of calls using an external command with the
following prototype:
command rnaIconv pSrcEncoding, pDstEncoding, pInTextVar, pOutTextVar

Something to note here is that we have to pass a variable names to the
function for both input and output – this is because we cannot pass
arbitrary binary data to external commands. To get around this, we pass
variable names and then use the GetVariableEx and SetVariableEx calls to get
and set their values (for more details on these two calls see the previous
external writing article: External Writing for the Unititiated – Part 2)<br
/>

Before going any further, make sure you have your favourite linux text
editor loaded (I tend to use gedit) and open the rnaiconv.cpp file.

Sorting out the declarations

Before actually implementing our function, we first need to sort out a
couple of declarations – we need to import the declarations of the system
functions we wish to use, as well as declare to Revolution that we are
implementing an external function.

To do the former, we just need to include the relevent header files. To
do this we use \#include directives. The one to use to gain access to iconv
is . So, add the following lines just on the line after the BEGIN USER
DEFINITIONS comment:

#include
#include
#include
#include

To actually declare our external function to LiveCode we need to add a
line to the externals export table. In our environment this is easy, there
are a number of macros we use to do this declared in . In this case, we use
the EXTERNAL_DECLARE_COMMAND one. Place the following line on the line after
the BEGIN USER DECLARATIONS comment:

EXTERNAL_DECLARE_COMMAND("rnaiconv", rnaIconv)

Now we have the declarations sorted out, we can actually implement
something!

Implementing the function

Although the method used to build externals differs between platforms,
the actual Externals API is identical – a Linux external function or command
is defined in exactly the same was as on the other two platforms, and such
functions and commands have access to the same set of engine calls. As these
details have been covered in previous articles, we’ll instead go straight
onto the function implementation.

Add the following function to the rnaiconv.cpp file after the \#include
directive we’ve already added:

// Function:
// rnaIconv pSrcEncoding, pDstEncoding, pNameOfInTextVar,
pNameOfOutTextVar
// Parameters:
// pSrcEncoding - the name of the source encoding
// pDstEncoding - the name of the destination encoding
// pNameOfInTextVar - the name of the variable containing the text we
want
// to convert
// pNameOfOutTextVar - the name of the variable containing the text we
want
// to place the converted text into
//
void rnaIconv(char *p_arguments[], int p_argument_count, 
char **r_result, Bool *r_pass, Bool *r_err)
{
int t_success;

// First check we have been passed 4 arguments and throw a (LiveCode)
// exception if not.
if (p_argument_count != 4)
{
*r_result = strdup("wrong number of parameters");
*r_err = True;
*r_pass = False;
return;
}

// Next we attempt to fetch the source text data we wish to use.
// To do this we use the 'GetVariableEx' call from the Externals API
ExternalString t_src_text;
GetVariableEx(p_arguments[2], "", &t_src_text, &t_success);
if (t_success == EXTERNAL_FAILURE)
{
*r_result = strdup("unknown input variable");
*r_err = True;
*r_pass = False;
return;
}

// Next attempt to create an iconv converter object using the given
// source and destination encodings.
// If creation fails, we throw a (LiveCode) exception.
iconv_t t_converter;
t_converter = iconv_open(p_arguments[1], p_arguments[0]);
if (t_converter == NULL)
{
*r_result = strdup("unable to create converter");
*r_err = True;
*r_pass = False;
return;
}

// We now convert the source text we have into an output buffer
// that we extend as appropriate.
//
// iconv works by attempting to convert as much input as possible
// into an output buffer of a given size, therefore we loop,
// extending the buffer on each iteration until no input remains.
//
// At each iteration, we allocate twice the remaining input size
// which should be fine for most encoding conversions.

// If the variable is unknown, throw an error.
// As the iconv call modifies the input buffer and length parameters
// we make copies here.
char *t_in_text_ptr;
size_t t_in_text_left;
t_in_text_ptr = (char *)t_src_text . buffer;
t_in_text_left = t_src_text . length;

// Setup initial values for the out text buffer. These initial values
// are 0/NULL since the first thing we do in the loop is allocate
// memory for that iteration.
char *t_out_text_base;
size_t t_out_text_frontier;
size_t t_out_text_limit;
t_out_text_base = NULL;
t_out_text_frontier = 0;
t_out_text_limit = 0;

// Flags that indicate what error occured (if any)
bool t_memory_error;
t_memory_error = false;

bool t_conversion_error;
t_conversion_error = false;
while(t_in_text_left != 0)

// Extend our output buffer.
char *t_new_out_text_base;
t_new_out_text_base = (char *)realloc(t_out_text_base, t_out_text_limit +
t_in_text_left * 2);
if (t_new_out_text_base == NULL)
{
t_memory_error = true;
break;
}

t_out_text_limit += t_in_text_left * 2;
t_out_text_base = t_new_out_text_base;

// Compute start pointer and bytes remaining for the out text buffer
char *t_out_text_ptr;
t_out_text_ptr = t_out_text_base + t_out_text_frontier;

size_t t_out_text_left;
t_out_text_left = t_out_text_limit - t_out_text_frontier;

// Attempt the conversion
size_t t_iconv_result;
t_iconv_result = iconv(t_converter, &t_in_text_ptr, &t_in_text_left,
&t_out_text_ptr, &t_out_text_left);
if (t_iconv_result == (size_t)(-1) && errno != E2BIG)
{
t_conversion_error = true;
break;
}

// Update the output buffer's frontier (i.e. increase by the number
// of bytes newly converte).
t_out_text_frontier += (t_out_text_limit - t_out_text_frontier) -
t_out_text_left;

}

if (t_memory_error)
{
// If a memory error occured, throw an error.
*r_result = strdup("out of memory");
*r_pass = False;
*r_err = True;
}
else if (t_conversion_error)
{
// If a conversion error occured, throw an error.
*r_result = strdup("conversion error");
*r_pass = False;
*r_err = True;
}
else
{
// No errors occured during conversion so attempt to write out result
// into the output variable
ExternalString t_out_text;
t_out_text . buffer = t_out_text_base;
t_out_text . length = t_out_text_frontier;
SetVariableEx(p_arguments[3], "", &t_out_text, &t_success);

// If the variable is unknown, throw an error.
if (t_success == EXTERNAL_FAILURE)
{
*r_result = strdup("unknown output variable");
*r_pass = False;
*r_err = True;
}
else
{
*r_result = strdup("");
*r_pass = False;
*r_err = False;
}
}

// Deallocate our out text buffer
free(t_out_text_base);

// Close the iconv converter
iconv_close(t_converter);
}

This function is basically an implementation of what we have outlined
above – it opens an iconv converter, processes all the input, then closes
the converter.

Building our external
Now that we’ve coded our function, we need to build and test it. As already
mentioned, the External Creator sets up a simple Makefiles for building on
Linux. A Makefile is simply a text file that describes the commands required
to convert the source files of a project into an actual executable or shared
library.

The ones the External Creator generates are predefined to make it easy
for you to get started with external writing. Two Makefiles will have been
generated, one in the environment and one in the rnaiconv folder. If you
take a look at the latter, you should see the following:

NAME=rnaiconv
TYPE=library

SOURCES=rnaiconv.cpp

# To add your own source files put them into this list.
# Note that source files are searched for in src/
#
CUSTOM_DEFINES=

# Add any custom include directions needed by your
# external to this list
#
CUSTOM_INCLUDES=./src

# Add any custom static libraries needed by your external
# to this list
#
CUSTOM_STATIC_LIBS=external 

# Add any custom dynamic libraries needed by your external
# to this list
#
CUSTOM_DYNAMIC_LIBS=
# Add any specific compiler options you need here # CUSTOM_CCFLAGS=-fno-exceptions -fno-rtti # Add any specific link options you need here # CUSTOM_LDFLAGS= include ../configurations/library.linux.makefile

As you can see from the comments, this file is ready to be added to as
your project grows. The most important line here is the ‘SOURCES’
definition. To add more source files to your external, simply append the
name of the file to this line (making sure they are separated by spaces).
(Note that due to the design of the system, you need to make sure that your
source files are all located in the src sub-folder).

The project Makefile is not designed to be invoked directly, instead it
is called by the Makefile in the environment folder when all the project’s
dependencies are also built – this allows more complicated projects to be
built up by decomposing them into smaller sub-projects present in separate
folders with their own Makefiles. In this case, your external project will
currently depend on libexternal – the standard glue-code required for a
LiveCode external.

Additionally, the environment Makefile offers three targets for each
project – one to build in a debug configuration, one to build in a release
configuration and one to clean the intermediate files thus enabling a full
rebuild. To use these targets you would use the commands:

  • make rnaiconv.debug
  • make rnaiconv.release
  • make rnaiconv.clean

To actually go ahead and build your project is simple:

  • Open up a terminal window
  • Execute cd (where is the full path to the unpacked
    ExternalsEnvironmentV3 folder)
  • Execute make rnaiconv.debug

After a few moments, control should return to you without any error
messages having been reported.

(If you do have problems and things don’t seem to compile correctly, the
NewsletterArticle3.zip archive you will have downloaded contains a
pre-prepared set of files that should work straight out of the box. Simply
unpack this archive, and copy the folder and other files into your
environment folder. Compiling as above should then result in everything
working as expected.)

Testing our external

Now that we’ve built our external, we need to test it. The External
Creator will have already set up an empty test stack for us to use. To make
it even easier, it will also have created a simple run script in the
environment folder.

This run script combines both building and launching into a single
command. To use it just type the comand ./run rnaiconv debug at your
terminal and you should find LiveCode popping up in due course with a blank
stack.

With this on the screen, its easy to see our external in action:

  • Create a button called “Convert”
  • Create one field called “Input”
  • Create another field called “Output”
  • Edit the script of the button and put in this mouseUp handler:
on mouseUp
   local tInput, tOutput
   put field "Input" into tInput
   rnaIconv "ISO_8859-1", "UTF-16LE", "tInput", "tOutput"
   put tOutput into field "Output"
end mouseUp

Once this is done, put some text in the input field (I always find “Hello
World!” a useful phrase) and just click the button. What you should see is
“Hello World!” in the output field, except spaced out – this is because the
output encoding we’ve chosen has two bytes per character, which for Roman
script text will have one NUL byte per character.

To see something more interesting, try entering some input text with
accented characters and use “ISO_8859-1″ for the source encoding and
“MACINTOSH” for the output encoding. In this case you should changes which
comes from the fact that accented characters are in different places for
these encodings.

To get a full list of the encodings that iconv supports you can use the
command iconv –list at the terminal prompt.

Moving forward

This guide has hopefully demonstrated how straightforward it can be to
produce an external for Linux. You can use the External Creator as many
times as you want inside a given environment folder to create additional
externals, all of which can be built and run using the run script in a
similar manner to that described above the rnaiconv project.

When you are happy with your external and want to install it permanently
in your distribution, you just copy the shared library into the appropriate
place inside your My LiveCode folder. Full details of this can be found in a
previous article External Writing for the Uninitiated – Part 1. You can
find the actual shared library file you need for this inside the
_build/release/ or _build/debug/ folders in your environment folder.

FAQ

Where can I get a copy of Visual C++ 2005?
Visual C++ 2005 is available in several editions as part of the Visual
Studio 2005 software suite. Microsoft makes Visual C++ 2005 Express Edition
available free of charge, and is available here

Where can I get a copy of XCode 2.4.x?
The XCode tools are freely available from Apple’s developer site after
registering for Apple’s Developer Connection (also free). The landing page
is here

Do I really have to wait for the IDE to load each time I want to
test my external?

No. If you are on a slow machine, you may want to setup your test stack and
build it as a standalone. You can then change the debugger parameters in
either XCode or Visual Studio to run this executable rather than the IDE.

For Mac OS X:

  • Navigate to the Executables branch of the project tree and Get Info on
    the Test node.
  • Change Executable Path to reference your standalone application.
  •  Go to the Arguments tab and remove all the arguments.

For Windows:

  • Right-click on the appropriate project in the Solution tree-view and
    select Properties.
  • Choose All Configurations from the drop-down list
  • Expand Configuration Properties and choose Debugging
  • Select Command, choose Browse and select your test standalone
    executable
  • Clear the command arguments and working directory fields

Can I reuse the source-code provided with this article in my own
projects?

Yes. You are free to use all source-code and project files included with
this article (or generated by External Creator V1) for the purposes of
producing LiveCode compatible externals.

libexternal seems to be different from the old XCmdGlue
implementation I have…

Yes – this is correct. libexternal is an updated version of the glue-code
which is cleaner. It uses exactly the same LiveCode interface as the
previous implementation and so will continue to produce externals that
will be usable in any LiveCode version.

What’s the difference between C++ (no exceptions, no rtti) and
C++?

Many people use C++ as ‘just a better C’ – and don’t take advantage of
exception handling or runtime-type information. Used in this form, there is
no need for the external glue-code to do anything special. However, if you
wish to use C++ exceptions, it is important the glue code automatically
handles any before control is returned to the engine and this option does
this automatically for you.

Can I use the standard C++ library?
Yes – but make sure you choose C++ when creating the skeleton for your
external. Most recent implementations of the standard C++ library require
exception handling (Visual Studio for example) and are not guaranteed to
work correctly if it is disabled. Creating your external with initial
support for C++ will ensure that unhandled exceptions in your external code
are caught before returning to LiveCode.

Can I write externals in other languages?
In theory any language that can build shared libraries and allows you to
link with the provided glue code could be used. However, in practice, some
languages will require extra glue-code in order to provide an environment in
which their code can run. For example, to access Objective-C you will need
to ensure you set up its memory/exception environment in C wrapper functions
before calling any Objective-C code.

Can I build externals that run on Windows 98SE/ME?
Yes. Any external that does not depend on any OS APIs should run fine on the
Windows 98/ME family of OSes. If you do you use any OS APIs you just need to
make sure they are available on that platform by checking in the Platform
SDK documentation (since you can’t explicitly target a given Windows version,
you won’t find out at compile-time if you’ve accidentally used an API that
isn’t available).

Can I build externals with other versions of Visual
C++?

Yes – but the External Creator only generates project and solution files
compatible with the Visual Studio 2005 family.

Can I build externals with other Windows C/C++ development
environments?

Yes. Externals are just DLLs which export a specific function and so any
compiler should be persuadable to produce externals compatible with
LiveCode. Of course, you will have to set up your own environment in this
case.

Note: the only requirement that LiveCode puts on a DLL
it tries to load as an external is that it exports one symbol whose resulting
name is _getXtable. Some compilers don’t automatically prepend an ‘_’ but
with appropriate options can be made to do so. (getXtable is defined in
libexternal).

Can I build externals with other versions of XCode?

In theory, yes. In practice, no. With the advent of Universal Binary you
really need to be producing externals that compile to both PowerPC and Intel
– this requires a minimum of XCode 2.2 (if I recall correctly). Also, if you
wish to support the same range of OS versions that LiveCode supports you
need to build them in a specific way:

  • PowerPC with gcc-3.3 against the 10.2.x SDK
  • Intel with gcc-4.0.1 against the 10.4u SDK

We use XCode 2.4.1 internally and know these configurations work without
problems and so we recommend that you do the same.

Of course, if you are targetting a specific OS version or architecture,
then you can choose your version of XCode appropriately, but you may have to
setup your environment from scratch.

Can I build externals with other Mac OS X C/C++ development
environments?

This depends on the OS versions and architectures that you are targeting.
Any environment that is based on the Apple GNU toolchains should be able to
produce compatible shared libraries. However, if they are not based on these
toolchains, your mileage will vary.

In all these cases, though, you will have to configure your build
environment yourself from scratch.

Can I use the same environment folder to build on both Windows
and Mac OS X?

Yes. As long as you checked both platform boxes in the ‘External Creator’
when you initially setup your external a project will have been created for
both platforms. To save constantly moving the folder between the two
platforms you could (for example) put the environment on a network share.
However, I recommend that you look into setting up a version control system
such as Subversion to manage your external projects – this isn’t at all hard
to do and numerous tutorials and GUIs exist on the web to help.

I’ve setup a version control system, what files should I
include?

As a general rule, you should only include ‘source’ files in the version
control system – these are the files that cannot be derived from other
files. So, in our case you will need:

  • The top-level folder
  • Each project folder
  • The configurations folder and its contents
  • The externals.sln file (if you are targetting Windows)
  • The src folders and their contents within each project folder
  • The vcproj file within each project folder
  • The vcproj.user file within each project folder
  • The xcodeproj folder and the project.pbxproj file within it (Mac OS X
    build only)
  • The plist file within each project folder (Mac OS X build only)
  • The LiveCode test stack within each project folder
  • Any other files that you have explicitly added

You should not add the _build or _cache folders, nor any ‘hidden’ files
or .ncb files on Windows. Furthermore, all the files within the xcodeproj
folder apart from project.pbxproj should be ignored.

The general principle is this – if a file or folder was not created by
you, or the External Creator it shouldn’t go under version control!

What about UNIX, Linux and Classic?
To begin with, we decided to focus on the two main desktop platforms.
However, we hope to look at the other platforms in future articles.