A LiveCode Shell

by David Williams on December 10, 2014 5 comments

For some time, I have been wishing that I could have some kind of command-line oriented functionality to LiveCode, beyond writing scripts with LiveCode Server. What if I could pipe things directly to it, or run it as an interactive shell? Being able to very quickly make use of the text handling in LiveCode would be extremely handy for certain command line oriented types of tasks. Alas, this is not possible with LiveCode or LiveCode Server as it currently stands. However, after some thought, I concluded that it should be possible to put together a script to provide this functionality.

2014-12-10 11_55_44-MTPuTTY (Multi-Tabbed PuTTY)

Click to Zoom

After some time, I had produced a script which operates as a simple shell type interface. But how does it work? It’s actually fairly simple. The script is a LiveCode Server script, designed to be invoked via command line. When it starts up, it reads in the options passed to it on the command line to determine how it should behave. The options passed to it are in the $x variables, starting at $0. $# tells us how many were passed.

#!/usr/local/bin/livecode-server
## substitute this with the path to your livecode server executable

## array for storing the options passed to this program
local sOpts

## load options
repeat with x = 0 to ($# – 1)
       do “put $” & x && “into tReadOpt”
       if char 1 of tReadOpt is “-” then
              repeat with y = 2 to the number of chars in tReadOpt
                       put char y of tReadOpt into sOpts[(the number of lines in the keys of sOpts + 1)]
               end repeat
       else
            put tReadOpt into sOpts[(the number of lines in the keys of sOpts + 1)]
      end if
end repeat

 There are currently two modes: interactive and execute. We can also pass the -p option to tell it to expect data to be piped to it, and read it into a variable called ‘input’. If we pass it the -e option it starts up in execute mode, and expects you to have specified a command for it to run. It then exits after running the command. Usage for this would be like “lcsh -e ‘put hello world’”. Otherwise, it starts up in interactive mode.

## read from pipe
if optEnabled(“p”) then
       read from stdin until EOF
       put it into input
end if

## get command from last command line option and execute it, then quit
if optEnabled(“e”) then
       do sOpts[(the number of lines in the keys of sOpts)]
       put return
       quit
end if

## interactive mode
repeat
       put “lcsh[” & the folder & “]# “
       read from stdin for 1 line
       put it into tCommand
      if isCommand(tCommand) then
               put shell(tCommand)
      else
              try
                       do tCommand
               catch pError
                      put “Error:” & return
                      put pError
               end try
       end if
     put return
end repeat 

Execute mode simply takes the last command line option and uses do on it, then quits. Interactive mode reads 1 line from stdin at a time (i.e. it reads 1 typed line by the user at a time) and then determines what should be done with it. It uses the isCommand function to check the $PATH environment variable (which contains all of the directories which should be looked in for executable programs) to see if what was entered was a utility that is already installed, such as the standard Linux utilities. If it was, it uses shell() to execute the entered command. Otherwise, it uses the do command to execute it as normal LiveCode code, catching any generated errors so that they can be displayed without causing the program to exit:

## checks to see if the first word is a command in $PATH
function isCommand pCommand
       set the itemDel to “:”
       put the folder into tFolder
       put false into tIsCommand
       repeat for each item tDir in $PATH
               set the folder to tDir
               if word 1 of pCommand is among the lines of the files then put true into tIsCommand
       end repeat
       set the folder to tFolder
       return tIsCommand
end isCommand

There is also the optEnabled function that was referenced above. This checks to see if the program was loaded with a specific option enabled:

function optEnabled pOpt
       repeat for each key tKey in sOpts
               if sOpts[tKey] is pOpt then return true
       end repeat
       return false
end optEnabled

 This is quite clunky and has many, many problems – for example, there’s no ability to pipe the output of some LiveCode to another program in interactive mode, and you can’t open another program that requires interaction. However, I think that with a lot more work, this could become quite a robust and useful tool. It already provides some very handy functionality in it’s current state, namely that I can pipe to it for text parsing:

2014-12-10 12_44_57-MTPuTTY (Multi-Tabbed PuTTY)

Click to Zoom

I will probably keep chipping away at this until I can use it as my primary command line interface. You can get the full code here: http://jasminedavid.on-rev.com/lcsh.txt

 

read more
David WilliamsA LiveCode Shell

Keeping an eye on your server

by David Williams on May 13, 2014 3 comments

Knowing how to interrogate your machines and get the *precise* information you need from them is a pretty wide-ranging topic. There are any number of ways of approaching this depending on what you need the data for. I’m going to go through some of the ways we keep an eye on our hardware, and where necessary, get detailed data from the server for analysis. Please note that the tools and servers we’re looking at here are Linux and may not be applicable to other server environments.

One of the main things we need to be aware of to get a gauge of how well the server is running is the server load average. This is essentially CPU usage calculated over time intervals (usually 1 minute, 5 minute, and 15 minute). We can see the load averages at a glance using the ‘uptime’ command:

root@jasmine [~]# uptime
15:41:41 up 126 days, 1:40, 2 users, load average: 0.10, 0.07, 0.05

This displays (among a few other things), the 3 load averages. A load average of 1 for a computer with 1 CPU core means that the CPU is fully used – anything over 1 means it’s overloaded. Our machines have 8-core CPUs, so we can see that this machine is very calm at the moment – a load of >8 would indicate overloading.

However, this command doesn’t give us much, and is entirely static. A better command for keeping an eye on the load and the rest of the system utilization is the ‘top’ command, perhaps the single most useful all-rounder for seeing system status at a glance:

topscreenshot

This command gives us an interactive and continuously updated display with the uptime, load averages, number (and status) of running tasks (processes), the CPU usage, the memory usage, and then a list of running processes which we can order in various ways. It’s essentially a command-line analogue of the task manager (windows) or activity monitor (osx), albeit a lot more flexible in some ways.

This shows us a sampling of the current running processes, but what if we need to see a comprehensive list of every single running process on the machine? We can use the ps command. The ps command is used for getting information about processes in various ways, in this instance we want to use the a u and x arguments to give us a detailed list of every single running process. The output of ‘ps -aux’ looks something like this:

psaux

We can look through this to get more of an idea of everything that’s currently running, and if we know what we’re looking for we can search through the output (via the ‘grep’ command for example) to determine whether the expected processes are there or not. This is great, but it’s very hard to read if we just want a summary of the current processlist. There are various utilities that do this, but why not write a quick one that does exactly what we want in LiveCode Server?

#!/usr/local/cpanel/cgi-sys/livecode-server

put $0 into tUserFilter
put shell("ps aux") into tProcessList

repeat for each line tLine in tProcessList
if tUserFilter <> empty and word 1 of tLine <> tUserFilter 
then next repeat
add 1 to tOutputArray[word 1 of tLine][word 11 to -1 of tLine]
end repeat

repeat for each key tUser in tOutputArray
put tUser & ":" & return
repeat for each key tProcName in tOutputArray[tUser]
put tab & tProcName & ":" && 
tOutputArray[tUser][tProcName] & return
end repeat
put return
end repeat

This little script gives us a nice summary of what the running processes are for each user and how many of those processes are running. We can also give it a user to filter by if we only want to see processes from that user using the $x variables – command line arguments passed to a LiveCode app are accessible by using $ and their position in the argument list (in this instance we use $0 to access the first argument passed, which we expect to be a username).

psdigest

The above tells us all about load and processes, but what if we want to know about what those processes are doing with the disk? There is some standard functionality in Linux for this, but we often use a great utility called ‘iotop’ (non-standard) to get this info, which functions in a similar way to top:

iotop

We can see, again, that this server isn’t very busy. One other thing we may want to know about is the current open connections and their status, which we can check using netstat. We almost always use netstat with the -n argument, as otherwise it will try and lookup the hostnames of the addresses involved, which is time-consuming.

netstat

This information contains a list of all the open TCP and UDP connections on the server, with the local address:port and the remote address:port pairs for each. This is very useful as we can quickly look through or parse this data for various purposes, for example we can see all the current mysql connections by using grep (the Linux search command) with the mysql port number:

netstat -n | grep 3306

This is useful as we can see where all these connections are coming from, which can aid in diagnosing certain types of issue. We could expand on this slightly to simply get the number of current mysql connections using the wc (word count) command with the -l (count lines) option:

netstat -n | grep 3306 | wc -l

These are some of the basic command line utilities we use to gather information about the system. There are many many more tools of various complexity that we use which would most likely take a small novel’s worth of blog posts to cover. At the heart of it, being able to query the system for specific information at the command line is often more fast an effective than using a complex GUI-based program that may not tell you precisely what you’re looking for.

read more
David WilliamsKeeping an eye on your server

Livecode.org email addresses and talking to PHP from LiveCode Server

by David Williams on April 3, 2014 2 comments

One of the things we’ve introduced recently was the LiveCode Membership product, one of the benefits of which is a livecode.org email address of your choosing. This meant that we needed to set up the livecode.org domain to use with email accounts on one of our servers, and then set up some way for the livecode.com store to communicate with it in order to set up the email accounts. This is made simpler by the fact that we make use of cPanel/WHM, meaning we can use the cPanel API for this kind of operation.

The cPanel API is written in PHP and allows you to perform a wide range of operations. The problem here, of course, is that it’s written in PHP – we want to talk to the API using LiveCode Server! So this raises the question – how do we interact with PHP scripts from LiveCode Server? This is a question that comes up every now and again for the reason that many utilities and systems are written in PHP.

In this instance, the API performs and action and returns a result (as either JSON or XML). For very simple instances where we just want to, for example, be able to get a list of all email accounts, we can set up the PHP script that performs the relevant API call, and then execute it directly using shell() from our LiveCode Server script:

...
put shell("php check_email_accounts.php") into tAccountsXML
...

This method is appropriate when we don’t need to pass information to the script, and where we also don’t need to use the CGI environment, which isn’t initiated if we execute it in command-line mode. Another way is to set up the script to handle data posted to the script:

...
post "user=" & tUser & "&pass=" & tPass & "&auth=r34l1ygr34TaU7h5tr1nG" to url("http://mail.somelivecodedomain.com/create_mail_script.php")
...
    

The script that the data is posted to should then be set up to take the values from $_POST and use them as parameters in the API call.
This way we also have access to the CGI environment in the PHP script should we need it. Security can be accomplished quite easily as we know where the post is coming from and can use .htaccess directives to restrict how the file can be accessed, as well as the standard types of script-based authentication we might use with this.

Ideally, in this instance, we would set up a single PHP script to which we would post the name of the desired API function and a list of parameters, which would be processed into the correct format and then passed directly into the api query function into that script. In this way we could implement a single callCPanelAPI() function in our LiveCode Server script.

read more
David WilliamsLivecode.org email addresses and talking to PHP from LiveCode Server

LiveCode Server

by David Williams on March 6, 2014 5 comments

Here at On-Rev, we make extensive use of LiveCode Server internally for scripting – you’ll see a lot more filenames ending in .lc than you might in .sh, purely for the reason that LiveCode Server is great for making scripts that are a lot more readable than, say, a bash script, and we can drop in shell commands with ease where necessary.

One recent feature that was introduced in LiveCode 6.6 was the ability to use hashbangs (#! /path/to/livecode) in LiveCode server scripts instead of script open and close tags (<?lc ?>). This means that we can have scripts which look like this:

#!/path/to/livecode-server
put “hello world at” && the millisecs

Instead of this:

<?lc
put “hello world at” && the millisecs
?>

The difference is that the latter has to be passed as a file to the LC Server exectuable, which then parses the code for output. This makes sense where LC Server is integrated into a web server software for serving webpages, but if we were doing this in a command line, it would look like this:

root@mybox [~]# /path/to/livecode-server ./script.lc

Whereas by using hashbangs, we can now execute the script directly, as the script contains the information for how it should be executed:

root@mybox [~]# ./script.lc

This seems like a minor change, but it brings the LC Server engine closer to how scripting for system utilities should function in a Unix environment, and allows me to tidy up some of the internal systems we use a little bit.

Additionally, work on the On-Rev client has been ongoing and we hope to release an update as soon as some of the last technical hurdles are overcome – we hope to send out a notification about this to all our On-Rev customers in the near future.

read more
David WilliamsLiveCode Server