Erwan Loison put up this image for me. I guess that it explains very well what an average day in my life feels like.
These days we’ve been adding scripting support to RCSDK. We constrained the answer to the question ‘what scripting language shall we support?‘ to Python and Lua.
Lua won by a long shot. Here’s why:
Both APIs (Python and Lua) are multi-platform, and are more than powerful enough to meet our demands. Both are also very popular and hence they can be considered highly reliable.
Python seems to be much more complete (higher level) and provides a comprehensive set of libraries. However, Lua is much more light-weight and there is some strong evidence that it is also much more efficient (see the related links below).
Last, but not least (and this is what I will talk about) the cost of embedding each of them into your own project happens to be very different.
Put in short:
– Lua is low-level, light-weight, clean and simple. Embedding it into our own SDK was trivial and took half an hour or so.
– Python is high-level, but fat, and its sources seem to be way more hellish/dirty. Embedding it took several hours of work.
– Lua increased the size of our .lib in 1.1 MB. Python did in 13.0 MB.
After taking a look at some benchmarks comparing both runtimes (speed/memory) we decided to choose Lua over Python.
The goal (in our case):
– Add scripting support to RCSDK.
– Avoid any external DLLs. RCSDK is designed to be just one file (a statically-linked library).
– Avoid any external LIBs. RCSDK is deployed in different operating systems, in 32-bit and 64-bit flavors, so it would be a terrible idea to keep many different .LIB files. So Python or Lua sources must be embedded in our own source code project. We always do this for the few exceptions where we use external libraries (libjpg, libpng, libtiff, libexr, libzlib, libfreetype).
– The same source codes must compile seamlessly across multiple platforms (Windows, Mac OS X, Linux) and in multiple compilers (MSVC, Intel C++ and GCC) without any warnings in /W4 at least.
– Short compilation times are a plus, so the fewer/simpler the files, the better.
– The scripting system should have no external dependencies whatsoever. It should not need the end user to install any additional packages or force us to add any files in our setups. Everything should be self-contained in the source codes.
– Last, but not least, efficiency (speed/memory) is a plus as well.
How to embed Lua-5.1.4 into your own C/C++ source project:
– Pick the sources package from the Lua website.
– Inflate the package wherever you like and get rid of everything but the /src folder. In my case I copied those files into a folder in my source project. Make sure to copy all the .c and .h files, but exclude lua.c and luac.c as those define a main()function themselves.
– (Optional step) Rename all the files, adding some prefix to them. It may be important to do this so you avoid future collisions with other .obj files, particularly if your project is large enough. In my case I used the prefix sys_lua_*. Then I ran a Search & Replace in the Lua files to replace #include “ by #include “sys_lua_.
– Lua is plain C, so if your own project is C++ you need to extern “C” the Lua #includes.
Also (bravo, Lua) the newly added source codes compile without any warnings at /W4. In 32-bit I only needed to use a single #pragma (C4996) to shut up those annoying _CRT_SECURE_NO_WARNINGS deprecations in MSVC (not Lua’s fault anyway). I needed a couple more #pragmas in 64-bit.
lua_State* L = lua_open() ; luaL_openlibs( L ) ; luaL_dofile ( L , "test.lua" ) ; lua_close ( L ) ;
You’re set to go. This code will load and run a test Lua script.
How to embed Python-3.2 into your own C/C++ source project:
From the very first moment that you download and inflate Python you notice the following:
While Lua is made of a small bunch of well-organized files, Python’s package has uncountable folders and subfolders with several trillions of files. Ok, Python is bigger and it may be able to do more stuff, but… come on. To begin with, the sources that you need are spread in 8 or 10 different folders where source/non-source files co-exist. Also, you need to hand-pick them selectively, as adding everything to your project will just fail to compile.
– Pick the sources package from the Python website.
– Inflate the package wherever you like and go to the /PCBuild folder.
– You will find a big mess in that folder. Open up the .sln file in Visual Studio (2008) and then remove all the .vcproj entries but pythoncore. This is the one producing the Python .DLL.
– Big hell ahead. Now open up your own source project (VS2010 in my case) and add to it each and every file in the pythoncore project. Be warned that you need to add many, many files. You may try to create a macro and transport the files from the original .vcproj to yours.
– Now spend some time fine-tuning the file pyconfig.h. You need to do this per platform, as Python deals with the OS native sockets, threading, etc… (Not the case in Lua). Python comes with a Windows-ready pyconfig.h which worked for me.
– (Optional step) Rename all the files, adding some prefix to them. It may be important to do this so you avoid future collisions with other .obj files, particularly if your project is large enough. In my case I used the prefix sys_pyt_*. Then I ran a Search & Replace in the Python files to replace the #includes accordingly.
– Make sure to #define Py_NO_SHARE_SHARED so the files are compiled for static (instead of dynamic) linking.
– Now get ready to get hundreds of warnings in VS2010 using /W4. #pragma party. I must say that I took a look at the code locations where those warnings were happening, and… some dirty things were happening down there.
Note that the newly added sources in your project won’t compile in other platforms unless you review the file pyconfig.h. So if you want to use just one config file, you need to #ifdef it per platform.
Also note that Python won’t run (it will throw an error and then crash) unless you copy the /Lib folder you will find in the Python setup into your execution folder. You need to do either that, or set the proper environment variable so Python knows the path of that folder.
The conclusion based on my experience:
– Lua’s impact in your source project is much more gentle (fewer and cleaner files).
– Lua is trivial to embed and highly portable. In our particular context, it took a surprisingly high amount of effort until the compilers swallowed Python, then some extra effort to shut up the warnings, and last, even more effort to get it to work in each of the platforms our SDK is designed to support.
– By default, Lua runs without the need for any external files, environment variables, etc…
– The size of Lua’s runtime wins hands down. Its performance and memory usage are remarkably better. This is the main reason why Lua has become so popular among game developers.
– Python is better equipped, but that extra equipment comes at a steep price.
– Python has more language features. Lua is more barebone which is both good (flexibility) and bad (may need some more effort on your side to implement things).
So, if you need the extra equipment that Python offers, go for it. Otherwise, you should go for Lua. Much more so if you’re paranoid about things such as performance, portability, flexibility, cleanness, executable size, etc…
Last week we were migrating our website to a new web server at RandomControl. While transporting the forum I came across this folder where we store most of the announcement-related images since the early days of fryrender. Some of these images were rendered by our testers and some others were made by me. They are a good example of the kind of stuff I’ve been involved with during the last years.
Some of this stuff reaches back to 2006, when I started coding fryrender. Watching the images in that folder brings a lot of good memories of the different stages the product has been at. These are some of my favorites:
Old Sub-Surface Scattering tests
Classic WinOSi dispersion test
LEGO spaceship (scene by Erwan Loison)
Camera response curves (scene by Jose Manuel Linares)
Instancing tests (scene by Martin Hindricks)
Image-Based Lighting tests
Micro-Polygon Displacement Mapping tests
More MPDM tests
EDIT: The links in this post are gone since we opened the new Arion forum in 2014.
For a long time I’ve been reluctant to enter the world of social networking. It all seemed like a big waste of time to me. However, after some time following regularly the tweets of other software engineers out there I guess that I’ve learnt to appreciate the value of it.
So a few days ago I decided to create a twitter account and started this humble coding blog, where I plan to drop a few brief notes every now and then. I will post about my work progress, and about other random technical stuff of interest for coders.