I think we can all agree that testing one’s Rails project is a good thing. I think we can also agree that those tests should be run on a regular basis (i.e. after every change). I think it’s a logical conclusion then that having a continuous integration environment setup for one’s project would be considered ideal. This is really pretty easy for open-source projects, since a number of companies provide free support for them. The go-to CI for Rails projects seems to be Travis CI, but it has a limitation: browser testing with javascript is hard.

The typical tool for Rails integration testing is Capybara. When I first got started with it, I began with the Selenium web driver for javascript tests since it was easy (Firefox just popped up and started running my tests). Unfortunately, when I wanted to get my project running with Travis CI, I quickly ran into a problem: Travis CI is headless. I couldn’t just launch Firefox and start running my tests.

Headless option: PhantomJS

I did a little research and came across PhantomJS, which would allow me to run my javascript tests completely headless. It was easy to get integrated with Capybara and running in my testing setup, but it didn’t work the same way as Selenium– several of my tests started failing, and I wasn’t sure why. Had I started out using PhantomJS, it probably would have been fine. Unfortunately, this was after my project was nearly feature-complete– I had many, many tests, and I wasn’t interested in rewriting them.

Better option: Sauce Labs

Fortunately, my work ethic is such that I’ll spend ages trying to do the least amount of work possible. That paid off: I came across Sauce Labs. Not only could I continue using Selenium, but I could run my Selenium tests across an enormous range of browsers and devices! I was thrilled, but my enthusiasm was quickly quelled by the lack of documentation.

The Problem

I read the documentation from Travis, but came away asking “How do I run my tests on multiple browsers?” I read the documentation from Sauce Labs only to realize it was basically the same document. I found some more documentation from Sauce Labs that introduced me to a gem for running multiple tests, only to learn after a while that I couldn’t use it with MiniTest. Also, that gem doesn’t work with Ruby 1.9.3, whereas the rest of my project worked with both 1.9.3 and 2.1. Using the gem would mean I couldn’t run 1.9.3 tests on Travis, which was a deal-breaker.

The Solution

I finally figured out the secret: the Travis build matrix. It was actually pretty easy to do this combining the original Travis documentation for Sauce integration (linked above) with the build matrix environment variables.

Let’s quickly discuss Travis environment variables. You can define them in your .travis.yml file like this:

env:
 - FOO=bar
 - BAZ=qux

My initial assumption was that the build in Travis would now have access to two environment variables: FOO and BAZ. That actually isn’t correct– each item in the env list causes a new Travis build to spin up! In other words, the above file would actually cause two Travis builds to begin, one with the FOO variable defined, and one with the BAZ variable defined. If I wanted the behavior I initially expected, I’d have to do this:

env:
 - FOO=bar BAZ=qux

That’s the key. I wanted a new build for each Sauce Labs combination I wanted to test, which meant that I needed a new env item for each. That ended up looking something like this:

env:
 # Linux tests
 - PLATFORM="Linux" BROWSER="firefox"
 - PLATFORM="Linux" BROWSER="chrome"

 # Mac OS X tests
 - PLATFORM="OS X 10.10" BROWSER="firefox"
 - PLATFORM="OS X 10.10" BROWSER="chrome"

 # Windows tests
 - PLATFORM="Windows 7" BROWSER="firefox"
 - PLATFORM="Windows 7" BROWSER="chrome"
 - PLATFORM="Windows 8.1" BROWSER="firefox"
 - PLATFORM="Windows 8.1" BROWSER="chrome"

That gave me 8 different Travis builds with those corresponding environment variables defined. Now all I had to do was use them.

I was able to tweak the information provided in the original Travis CI document about integrating with Sauce Labs, and came up with this to put in my test_helper.rb:

# <snip>
# Only run with Sauce Labs if we're running on Travis CI.
if ENV['TRAVIS']
   capabilities = Selenium::WebDriver::Remote::Capabilities.send ENV["BROWSER"]

   # Set the browser version. If this is nil, Sauce Labs will use the latest version.
   capabilities.version = ENV["VERSION"]

   # Set the operation system.
   capabilities.platform = ENV["PLATFORM"]

   # Set the tunnel ID to connect to the identified Sauce Connect tunnel from Travis.
   capabilities['tunnel-identifier'] = ENV['TRAVIS_JOB_NUMBER']

   # Give the build a name in Sauce Labs so they aren't just all "Unnamed Job."
   capabilities['name'] = "Travis ##{ENV['TRAVIS_JOB_NUMBER']}"

   Capybara.register_driver :selenium do |app|
       Capybara::Selenium::Driver.new(app,
                                      browser: :remote,
                                      url: "http://#{ENV['SAUCE_USERNAME']}:#{ENV['SAUCE_ACCESS_KEY']}@ondemand.saucelabs.com/wd/hub",
                                      desired_capabilities: capabilities)
   end
end

# <snip>

Finally, I had multiple Travis CI builds all talking to different Sauce Lab environments, running my tests. This setup has been working great for me so far, and I encourage you to give it a try.