Alchemy Data Manipulation Faster Than Vectors

While working with Alchemy to port a roguelike library to handle field of view calculations for a 2d tile game (The Doryen Library, port is called libtcod-alchemy) I noticed some drastic speed differences between using a ByteArray in pure AS and manipulating that same ByteArray inside Alchemy. Out of curiosity I wondered how Alchemy manipulations of large data structures would compare to a native AS Array or the new (and faster) Vector.

As it turns out, Alchemy wins hands down when the data is already in a ByteArray. If you need to transfer data from an array/vector into a ByteArray to hand off to Alchemy and then transfer it back, it is slower to use Alchemy. For this test I have a collection of 1,000,000 numbers and for each iteration of the test, I loop over the numbers and add one.

Alchemy ByteArray additions are about 6 times faster than Vector integer additions and 10 times faster than pure AS manipulations of a ByteArray (when using Array notation [] on a ByteArray to manipulate a Byte - using position, readInt() and writeInt() in AS is much slower).

Below is test program that proves it:

Note that the demo above requires Flash Player 10 (required by Alchemy).

NOTE: If you want view the source, you need to go here and right click on the app (the embedded version in blogger doesn't know where to look for the source).

This tells me a few things:

• There is a great deal of room for improvement in ActionScript's compiler since it should perform better IMHO. I understand some slow downs due to type safety, garbage collection, etc but I didn't expect the huge difference.
• If Alchemy had native access to the same Native calls used to seralize data to AMF that is used in AS (described here), Alchemy would be blazingly fast for manipulating data even when stored in Arrays or Vectors. You would just serialize the objects as AMF into Alchemy, manipulate the data with Alchemy, and serialize it back out as AMF back to AS.
• The ByteArray.position is much slower than I would expect.
• If you need to process large amounts of data, seriously consider using Alchemy. Remember the marshalling tax to and from Alchemy - every bit of data passing into and out of Alchemy takes time to marshall, so you want to do it in large chunks like in my speed test.

You may notice that the first Alchemy test takes more time than normal - this is due to initialization of the Alchemy library.

In case you are curious essentially the same test takes:

• GCC C Code - 1 or 2 milliseconds
• GCC Lua Code - 300-400 milliseconds
• Lua Alchemy - 8 seconds (might be including some of the time to get the tick count from ActionScript)
  

Lua Alchemy Getting Sweeter in v02a

Thanks to Alexander Lua Alchemy now has sugar! What this means is you no longer need to make calls like the following in the Lua Alchemy code:

as3.call(myvar, "someCall", param1, param2)

Instead you can make calls that look like native Lua code:

myvar.someCall(param1, param2)

This includes chaining calls like a.b.c().d

Below is a preview of the release:

Note that the demo above requires Flash Player 10 (required by Alchemy and my use of the local file system to allow opening/saving files). It is also hosted here where it can take up a full browser screen and is much easier to read.

You will notice in the demo code a great number of calls to as3.tolua(xxx). These calls convert ActionScript types into Lua types where possible. We removed all automatic conversion from AS3 to Lua types for performance reasons - each time you cross the Lua<->C<->AS3 boundary you have to pay the Alchemy wrapper performance tax.

For more information on the upcoming release of v0.2a (official downloads available in the next few days) see:

• HISTORY for v0.2a


• Lua Alchemy Homepage

Lua Alchemy v0.1a Released

The first release of Lua Alchemy has been released; the library SWC, demo, and code can be downloaded from here:
http://code.google.com/p/lua-alchemy/

Since my preview post (which is now running the same demo code) we have

• Added many Lua scripting samples.
• Fixed some memory leaks.
• Cleaned up function callbacks created in Lua which would typically be used for event listeners.
  

Below is a preview of the release:

Note that the demo above requires Flash Player 10 (required by Alchemy and my use of the local file system to allow opening/saving files). It is also hosted here where it can take up a full browser screen and is much easier to read.

The next version will include syntax sugar so instead of calling as3.call(var, "function", param) you could call var.function(param) and similar calls to make calling ActionScript code look the same as calling Lua code.

LuaAlchemy Project v.1a Preview

Since my last post my experiment to make Lua a runtime scripting language in Flash using Alchemy has become a full open source project located here:
http://code.google.com/p/lua-alchemy/

I even have a partner named Alexander Gladysh whom I met on the Lua boards.

We are approaching a V.1a release of the project. Using LuaAlchemy you can now create any ActionScript class available at runtime, get/set properties, call functions, and set event listeners (actually you can return a Lua function to use as an ActionScript function). The Lua to ActionScript interface is a little raw right now but we plan on adding some Lua sugar to make accessing ActionScript objects, classes, and namespaces in Lua look and feel like using Lua objects.

Below is a preview of the release:

Note that the demo above requires Flash Player 10 (required by Alchemy and my use of the local file system to allow opening/saving files). It is also hosted here where it can take up a full browser screen and is much easier to read.

Along the way to this release we have encountered a few problems which I will discuss. The first problem was speed. Although the Alchemy V.4a compiled code was fast enough for our purposes, it was about 30 times slower running a life simulator written in Lua than natively compiled Lua. The release of Alchemy V.5a has made life run 4 times faster so progress is being made (v.4a had some debug code compiled in that slowed things down).

The second problem has been gluegen. Gluegen is a tool to help you create wrappers around C code that takes a mix of C and ActionScript code to generate C and ActionScript wrappers. Unfortunately this seems to interfear with CLibInit.setSprite() which we wanted to use to provide a sprite that Lua could use for stdin, stdout, and stderr. Glugen created code creates and inits CLibInit as static members and you seemingly need to call setSprite() before init(). We worked around that problem by providing an output TextArea in the demo below, but we plan on writing the wrapper ourselves and not using gluegen at all.

The third problem is related to garbage collection in Flash and Lua. To make Lua functions usable as an ActionScript function we call AS3_Function() to create an ActionScript function that calls a C callback function passing in some contextual data. This data includes some Lua references in a malloced structure and we would like to know when it isn't needed anymore so it can be released. We haven't figured out any way for Alchemy to tell us when the object is no longer needed. A future version will do some cleanup when the lua_state is closed.

Overall our experience has been a positive one and Alchemy is still in Alpha.

Lua ActionScript Alchemy

At Adobe MAX 2008 San Francisco, Alchemy was released. Alchemy was once known as FlaCC or Cassava and is a C/C++ to ActionScript compiler.

Alchemy works by translating C/C++ code to ActionScript through a LLVM compiler which instead of producing machine or bytecode produces ActionScript. The translated C/C++ code uses a big ActionScript ByteArray as the C/C++ heap and stack. The performance of alchemy is slow if you martial data from C/C++ code to ActionScript or back but faster (supposedly up to 10 times faster) than ActionScript code you write yourself for data intensive actions where everything operates in the C/C++ memory space - speed improvements over AS code are also due to LLVM doing code optimization which is missing from the normal AS compiler.

After downloading and setting up Alchemy, I started trying to port over a few C/C++ libraries. The following libraries have given me some trouble mainly with needing to adjust makefiles or the ./configure script: Angband, Nethack, guile, Python, Ruby, ARToolkitPlus, SDL, and Squeak. I haven't given up on them but I wasn't in the mood to do battle with Makefiles; I wanted to learn Alchemy.

My success came with compiling the scripting language Lua 5.1.4 and embedding it into a Flex application. The following proof of concept can run a Lua script which returns one or more values and are shown in the Lua Stack Panel. Lua can make the following calls into Flex:

• getTextAreaProperty(property_name) - Returns the given property of the text area in the lower left hand corner.
  
• setTextAreaProperty(property_name, value) - Sets the given property on the text area in the lower left hand corner (must accept string values).
  
• callTextAreaFunction(function_name[, param1, ..., paramN]) - Calls a function on the text area in the lower left hand corner (must accept string values).
  
• average(value1, ..., valueN) - Returns the average and sum of the given values.
  

Note that the demo above requires Flash Player 10 (required by Alchemy and my use of the local file system to allow opening/saving files). It is also hosted here.

Should I continue working on this library, I'd want to allow Lua to create and call functions on any arbitrary ActionScript class. Alchemy is still in beta and does have its own problems - not everything builds easily and while pure Alchemy code can be faster than ActionScript, marshaling from ActionScript to C/C++ code and back can slow things down - for example life.lua takes 4 seconds with the standard compile of Lua and 13 minutes! with the Alchemy build. Still it works quite well for my purposes of embedding a runtime scripting language into my Flex applications.

For more information check out the following blog posts:

• http://www.automatastudios.com/2008/11/21/understanding-adobe-alchemy
• http://docs.google.com/View?docid=dg7w6sqr_8fbhvz6d6
• http://techblog.floorplanner.com/2008/11/23/alchemy-first-looks
• http://www.theregister.co.uk/2008/12/02/adobe_alchemy/

Edit: Created a google code project to share efforts in using Alchemy to embed Lua in ActionScript. Uploaded the proof of concept:

• http://code.google.com/p/lua-alchemy
• http://code.google.com/p/lua-alchemy/downloads/list
  

FlaCC (C/C++ to Flash Compiler)

More news on the C/C++ to Flash compiler, now called FlaCC. With this tool developed by Scott Petersen at Adobe would allow almost any arbitrary C/C++ code to be compiled down to ActionScript bytecode to run on the Flash player. I first heard of this at MAX 2007 and there will be a session at MAX 2008 on this (plus hints it will be in keynotes).

While we wait here are a few links to some more recent news:

• http://www.peterelst.com/blog/2008/10/05/more-information-on-flacc/
• From LLVM Dev Meeting
  
  • http://llvm.org/devmtg/2008-08/Petersen_FlashCCompiler.pdf
  • http://llvm.org/devmtg/2008-08/Petersen_FlashCCompiler_Hi.m4v
    

The MAX 2008 Session is "Using C++ Libraries in Flash Player and Adobe AIR" Monday, November 17, 2:00 pm - 3:00 pm. I'm looking forward to that session.

360|Flex

This coming August, I'll be heading out to San Jose to attend the 360 Flex conference:
http://www.360conferences.com/360flex/

I'm new to the Flex scene (almost a year now) and came from a background of 14 years of C++ cross-platform application development so it is exciting and different to immerse myself in the new culture. This is the first 360 I will have attended and only the second conference related to Flex (the first being MAX 2007 in Chicago). I'm now fortunate enough to be working for Universal Mind (as a contractor) and when I saw all the people taking vacation to attend 360, I knew I had to attend.

Of particular interest to me are sessions teaching frameworks, component building, design patterns, process (I'm a big process geek - how you do something is very important for maintainability), and low level programming. There are so many sessions that appeal to me at the same time that I'm going to have to make some rough decisions. Here is a list of the appealing sessions (fortunately I work with some of these people so I can pick their brain later):

• Pre-Conference stuff I cannot attend because my plane doesn't land until 2pm
  
  • James Huang - The R1-RPC/J Toolkit and the D.eval API
    
  • Ben Stucki/Ryan Campbell - OpenFlux
    
• Adam Flater - How to make your AIR app blow up with Merapi
• Alex Harui - Versioning Support for Large Applications and Portals
• Andrew Powell - Implementing BlazeDS
• Axel Jensen - Project Workflow (trac,svn,ANT,XML...)
• Ben Clinkinbeard - Concepts and Strategies for Creating Reusable Components
• Ben Stucki - How to Build a Framework
• Brad Umbaugh/RJ Owen - Diving Deep with the Flex Component Lifecycle
• Brian Weisenthal - Architecting real-time and collaborative applications in Flex
• Chet Haase - Filthy Rich [Flex] Clients
• Chris Scott - Introduction to the Swiz Framework for Flex
• Eric Ko - Reflective Programming
• Jacob Wright - Advanced ActionScript APIs
  
• Jeff Houser - How to Build Flex Components
  
• Jeff Tapper - Unraveling the hassle with Drag an Drop in AIR
• Jonathan Branam - Improve your programming by reading the Flex source code
• Josh Tynjala - Polishing Components for the Unwashed Masses
• Laura Arguello - Decoupling Flex Applications with Mate Framework
• H. Paul Robertson - Advanced AIR and SQLite: An optimization conversation
• Ryan Campbell - PlexiGlass (OpenFlux 3D Components)
• Samuel Asher Rivello - Creating an MMO from Scratch with Adobe Flex 3… in 59 minutes.
• Tom Gonzalez - Data Visualization and Dashboards with Flex
• Troy Gardner - Advanced State management, the Heirarchical State Design Pattern

PS: The 360|Flex team sent me a 1GB flash drive with content from 360|Flex Atlanta and Europe. I could pass it on to a lucky commenter.