Creating a Browser Language
During a Hackathon, a coworker and me decided to tackle the problem of creating a new language which runs in the browser. Our end goal: be able to write script elements in our language and have the browser run it for us, in realtime.
None of us is a static compiler/JIT expert so the way we went about it is:
- Create a toy language and its toy lexer/compiler
- Make our toy language compile to LLVM
- Use LLVM’s tools to take advantage of the JIT
- Run our language is Chrome (29+) thanks to PNaCl
Hackathons at Yelp last 2 days. That’s not much time. To prepare ourselves for these 48 hours and to make them as productive as they could be, we set up our environment first.
Setting up LLVM environment
I’m usually developing on a MacBoook, so I tried first to setup llvm and friends on my local machine. Huge failure, for which I don’t have a solution yet.
MacOSX makes you install XCode to install compiler tools like gcc or g++.
Com’on Apple, really?
XCode also comes with some version of llvm installed. As a result executing
locate llvm | wc -l yields
610 on my machine. Basically some sort of tools
for llvm is installed with XCode and each version of the iPhone simulator, and
also macports. No idea why those were here in the first place.
Since I had no idea what version of the tools I should go with I decided to go the hard route and install everything from source in my homedir, following instructions from the LLVM getting started page
cd ~/bin/ mkdir llvm cd llvm/ svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm cd llvm/tools/ svn co http://llvm.org/svn/llvm-project/cfe/trunk clang cd ../projects/ svn co http://llvm.org/svn/llvm-project/compiler-rt/trunk compiler-rt svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite cd ../../ mkdir build cd build/ ../llvm/configure --prefix=/Users/abrousse/bin/llvm make -j10
The listing above generated lots of output, consumed lots of my CPU, and ended with an error:
/Users/abrousse/bin/llvm/llvm/projects/compiler-rt/lib/asan/asan_malloc_mac.cc:19:10: fatal error: 'CoreFoundation/CFBase.h' file not found #include
^ COMPILE: clang_darwin/asan_osx_dynamic/i386: /Users/abrousse/bin/llvm/llvm/projects/compiler-rt/lib/asan/asan_posix.cc COMPILE: clang_darwin/asan_osx_dynamic/i386: /Users/abrousse/bin/llvm/llvm/projects/compiler-rt/lib/asan/asan_preinit.cc COMPILE: clang_darwin/asan_osx_dynamic/i386: /Users/abrousse/bin/llvm/llvm/projects/compiler-rt/lib/asan/asan_report.cc COMPILE: clang_darwin/asan_osx_dynamic/i386: /Users/abrousse/bin/llvm/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc COMPILE: clang_darwin/asan_osx_dynamic/i386: /Users/abrousse/bin/llvm/llvm/projects/compiler-rt/lib/asan/asan_stack.cc COMPILE: clang_darwin/asan_osx_dynamic/i386: /Users/abrousse/bin/llvm/llvm/projects/compiler-rt/lib/asan/asan_stats.cc 1 error generated. make: *** [/Users/abrousse/bin/llvm/build/tools/clang/runtime/compiler-rt/clang_darwin/asan_osx_dynamic/i386/SubDir.lib__asan/asan_malloc_mac.o] Error 1 make: *** Waiting for unfinished jobs.... make: *** [BuildRuntimeLibraries] Error 2 rm /Users/abrousse/bin/llvm/build/Debug+Asserts/lib/clang/3.4/lib/darwin/.dir make: *** [compiler-rt/.makeall] Error 2 make: *** [all] Error 1 make: *** [clang/.makeall] Error 2 make: *** [all] Error 1
So, fuckit. Another dead end. I’m suspecting all revisions from clang, llvm and other projects aren’t tested together that often. I think something to try next would be checking out release-tagged revisions instead of the latest master from each project. WHY THE HELL DON’T THEY HAVE SUBMODULES? That drives me nuts.
Solution for me was to install things from a Ubuntu VM. Package management is saner, and installing llvm is a piece of cake:
sudo apt-get install llvm
All done! Just check that the output of
llvm --version gives you 3.0 and
you’re good to go.
Back to OSX
Back to OSX. I tried what’s on the release
page, where they have precompiled
binaries for OSX. On my version of OSX (10.7.5), LLVM 3.3 didn’t work properly.
Building the 3.0 tools worked properly with what’s in
https://github.com/bukzor/hackathon2013.7 (check out the
End goal: Do hypercube animation by writing a
<script src="woot.bl" type="application/x-bogolisp">...</script>
- (optional) Chrome extension to parse and evaluate what’s in the script tags
- Lexer with tests – converts string to tokens, also known as scanner
- Parser with tests – converts tokens to AST
- Runtime/Interpreter – Uses AST to run the program in the browser
- (optional) Compiler – converts AST to asm.js