As I’ve mentioned in previous posts, I run all my websites on a MiraBox– my nifty, low-power ARM server. As awesome as it is having a low-power server, it does make things more difficult. I’m a Ruby on Rails developer, and I run three websites on this thing, along with a gitolite installation so I can host my own code. However, I recently began a few projects collaborating with other people, and as the projects grow, gitolite is not the collaboration platform of success. I need groups and better permissions. I need issue tracking. I need wiki pages. I need GitLab.

There’s only one problem. If you follow its installation procedure, GitLab does not support the ARM architecture. Rather, gems upon which GitLab depends do not support the ARM architecture. Namely: therubyracer.

Server-side Javascript has long been a nightmare on ARM. therubyracer depends on the libv8 gem, which straight up cannot build on ARM. Which is annoying, since without too much convincing, V8 itself can build on ARM (even though it uses GYP, a build system that I simply cannot figure out).

I tried participating in therubyracer and libv8 gem development, but there’s not a lot of momentum behind ARM development there and later versions of V8 simply don’t compile with the old GCC in Squeeze anymore, so I gave up and punted: Hello Node.js. With some code tweaks, I was able to get THAT installed on my MiraBox. I then removed therubyracer from my Gemfile and Gemfile.lock, and was able to follow the GitLab installation directions. Well… mostly.

As I said before, I already had three different websites on my server using Apache and Passenger– I refuse to also use Nginx and Puma or Unicorn or whatever the kids are using these days. That made things a little harder though… was there something special about Puma/Unicorn? Did I need it? These were questions for Google.

…

No, I take that back– with an utter lack of results other than a few mentions of Apache with no one talking about Puma or Unicorn, they were questions for the mailing list.

…

No… I take that back– no one ever responded. So, I had to figure it out on my own.

It was a while ago that I learned the lesson not to experiment with this type of stuff on a live server. However, I took it even a step further. I had previous experience setting up a GitLab install, but that experience only involved following the GitLab installation instructions to a T. I wanted to do a few things differently in this situation: namely, I wanted to use Apache instead of NginX, I wanted to use Passenger instead of Puma/Unicorn, and I needed to use Node instead of therubyracer. Since I didn’t even know if this setup would work on the best of systems, I decided that testing this setup for the first time on a crawling ARM emulation wasn’t the best way forward. I wanted to see if this worked on a more powerful, normal machine. So I threw a server installation together in VirtualBox, using Ubuntu 12.04 64-bit.

I followed the GitLab installation instructions, but I skipped a few parts this time. GitLab 6.3 uses Unicorn (whereas some previous releases used Puma), and I specifically did not setup Unicorn at all– I left its configuration file at “unicorn.rb.example,” which I assume means it doesn’t run. This required an alteration to the gitlab init script, since it runs both Sidekiq and Unicorn. I simply removed all mention of Unicorn, and rewrote the script to use Sidekiq only.

Also, obviously, instead of setting up an NginX configuration for GitLab, I setup an Apache2 virtual host. Finally, before I ran the bundler to deploy the gems, I removed therubyracer from the Gemfile (and Gemfile.lock) and installed the newest Node.js from source. This setup actually worked perfectly, first try. There was hope! Also, that answered the question of Unicorn/Puma: They are not necessary for GitLab to function.

I then moved to duplicating each of these steps on the QEMU-emulated ARM server. That took ages, I’ll tell you. And after I finished I got nothing. It spun for ages and then said that the server didn’t respond. Okay… so not as good as the VirtualBox version. My investigation started in the GitLab logs. It actually looked like things were happening, but then Firefox always came back with “server didn’t respond.” After a little more digging I realized that everything did indeed look fine– the server just didn’t respond fast enough.

After some searching around, I realized that with Passenger 4.0.20 they introduced the “PassengerStartTimeout” option, which allows one to extend the loading window giving a good-sized application like GitLab time to fire everything up. I set it to 60000 (time out now, I dare you). Success!

It was time to put GitLab on my real server.

I went through the same steps– didn’t configure Unicorn and configured an Apache2 virtual host for Gitlab:

<VirtualHost *:80>
   ServerName www.url.com
   ServerAlias url.com
   DocumentRoot /home/git/gitlab/public
   <Directory /home/git/gitlab/public/>
       AllowOverride all
       Options -MultiViews
   </Directory>

   PassengerStartTimeout 60000 # Required since the MiraBox is so slow
   PassengerRuby /usr/local/rvm/wrappers/ruby-1.9.3-p484@gitlab/ruby

   ErrorLog ${APACHE_LOG_DIR}/gitlab_error.log

   # Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
   LogLevel warn
   CustomLog ${APACHE_LOG_DIR}/gitlab_access.log combined
</VirtualHost>

I removed therubyracer from both the Gemfile and Gemfile.lock, installed Node.js, and deployed all the required gems with Bundler according to the GitLab installation guide. I had everything setup identically to my QEMU-emulated server, and I ran it… and got a 500 error.

What the heck? So I started digging into my logs. I actually tail’d the GitLab production log, and watched it load. I watched it compile all the assets on the first request (as expected). I saw the request get the layout, and the layout partials… and then a 500 error after _head.html.haml. There wasn’t actually an error in the GitLab log, or the Apache log, or being shown in Firefox– just a 500. Normally one actually sees something like “function .empty() doesn’t exist for Nil.NilClass” or something along those lines… I saw nothing. It was totally bizarre. So I edited _head.html.haml and started old-school debugging: adding print (logger.debug) statements. Using this technique I discovered that the 500 happened right after application.js was being included. Interesting… so it involved the asset pipeline. Just for kicks I tried precompiling the assets with:

$ bundle exec rake assets:precompile RAILS_ENV=production

And I got an error! I’m afraid I don’t have the error message saved, but it involved an invalid byte and something about UTF-8. Regardless, it was failing to compile its assets when they were requested, so it barfed a 500 without actually giving any sort of descriptive error (which was annoying).

What do you do when assets need to be compiled but the server has errors when compiling them? Fortunately I already had the solution in place: Compile it in another, identical environment. I chose the VirtualBox server since it was way faster than the QEMU server. I ran the same precompile line on there, then tar’d up the generated assets and placed them in the correct spot on my production server.

The moment of truth: I restarted everything and it worked beautifully :) .

GitLab lessons learned:

  1. Unicorn/Puma is not necessary, but not using them requires an altered boot script.
  2. NginX is not necessary– Apache2 works great.
  3. Node.js is a replacement for therubyracer, but it doesn’t seem 100% compatible with the ARM architecture. Asset compilation is wonky… compile elsewhere and deploy.

The only question I have is this: Why did Node.js fail to compile assets on my production server, but succeed to compile assets on my QEMU-emulated ARM server? I have yet to find an answer.