So you’re developing a complex application. Maybe it has some really specific dependencies, or requires a lot of setup. In many cases (such as my own), it’s a web application. I have a suite of tests, varying all the way from unit tests through integration tests. The latter typically uses Selenium, and I often integrate it with Sauce Labs. I’ve written an article about this before. However, even those integration tests aren’t testing the real application: it’s not running against a production database, it’s not running with a production web server, and so on. Doing this is not easy; oftentimes the solutions I see look like this:

  1. Create custom docker images with the rest of the system (dependencies, database, web server, etc.) setup correctly
  2. Inject the code into it
  3. Run tests against it

Not only this, but one needs to ensure one’s image stays in line with one’s actual dependencies and continues to mirror a production deployment.

There’s a better way: package your application as a snap.

I’ve written about the Nextcloud snap a number of times. It’s one of the more complex snaps out there, bundling Apache, MySQL, PHP-FPM, etc. It’s a production-ready snap, following Nextcloud’s recommended settings. This makes it a perfect testing target! Except… there are no hosted CI options that can actually run snaps. All the ones I tried didn’t have a kernel configured properly for snaps (e.g. Travis). I was starting to investigate integrating my private GitLab instance (with its own amazing CI) with GitHub just to run tests against the snap.

Then I heard about Circle CI. By default they run docker on Ubuntu Trusty, but they have actual VMs available as a beta (the machine executor), which have a Xenial kernel and can actually run snaps! It’s a bit limited, as I’ll discuss, but it solves a real need at least for the Nextcloud snap, and I’d like to walk you through doing the same thing.

Backstory

The Nextcloud snap publishes on a number of different channels:

  • stable: The current stable release of Nextcloud (currently v11.0.3. Yes, I know v12 is out, we’ll talk about that in a sec)
  • candidate: Release candidates, used for testing calls before releasing to stable
  • beta: Snap development branch. Automatically built as new features are added to the snap (e.g. when pull requests are merged for the snap)
  • edge: Daily builds of Nextcloud’s master branch
  • 11/edge: Daily builds of Nextcloud’s v11 maintenance branch
  • 12/edge: Daily builds of Nextcloud’s v12 maintenance branch

Any one of these could immediately be installed with:

$ sudo snap install nextcloud --channel=<channel>

It’s actually pretty neat: want to see what work went into Nextcloud yesterday? snap install --edge nextcloud. It’ll automatically update daily, giving you the opportunity to easily track development.

We have all these things happening automatically, but with zero CI because as I mentioned, we couldn’t find any hosted CI solutions that were shiny enough to actually run snaps. Of course we didn’t release to beta much less candidate or stable without some extensive manual testing, but it wasn’t good enough: no one was testing edge.

So Nextcloud version 12 was just released upstream, and we quickly discovered that it doesn’t work in the snap. If we were actually able to run tests against the snap we could have caught this before v12 was actually released!

The solution

A few weeks ago, I saw a forum post from Alan detailing his use of Circle CI for building/pushing snaps. Our builds are already automated so I wasn’t so interested in that, but what caught my eye was the fact that he was installing Snapcraft from the snap, i.e. Circle CI could run snaps! I took a closer look as soon as I could.

It turns out that, like Travis, Circle CI runs on Ubuntu 14.04 (Trusty). Snaps actually can run on Trusty, but the kernel was too old. However, instead of using Docker as an executor type, one can use the (still-in-beta) machine executor type, which spins up a VM that is still based on Trusty, but uses a newer v4.4 kernel.

Once I enabled the Circle CI integration for the project, it was a simple matter of actually creating acceptance tests to run, and then creating the correct Circle CI configuration to build the snap and run the tests against it.

Creating acceptance tests

Nextcloud actually already has a suite of acceptance tests. However, they make a lot of assumptions about where they are and how they’re run, such that they wouldn’t work for our purposes.

I have a lot of experience writing Rails apps, and one of my favorite tools from that world is Capybara. I’ve never used it in a Ruby-only project, but it was a perfect fit for what I needed to do, so I went for it. I needed five gems– here’s my Gemfile:

source 'https://rubygems.org'

# Test driver
gem 'capybara'

# Webdriver (for javascript)
gem 'capybara-webkit'

# Create xvfb from ruby (needed by webkit)
gem 'headless'

# Use nice acceptance tests DSL
gem 'rspec'

# Run tests with a decent CLI
gem 'rake'

Install those gems and their dependencies:

$ sudo apt install gcc g++ make qt5-default libqt5webkit5-dev ruby-dev zlib1g-dev
$ gem install bundler
$ bundle install

Then setup rspec:

$ rspec --init
 create  .rspec
 create  spec/spec_helper.rb

That spec/spec_helper.rb is where we put some setup code for the tests:

require 'capybara'
require 'capybara/dsl'
require 'capybara/rspec'
require 'capybara-webkit'
require 'headless'

include Capybara::DSL

Capybara.configure do | config |
   # Set javascript driver to webkit (selenium is the default)
   config.default_driver = :webkit

   # Assume the snap is available at localhost. Change here if otherwise.
   config.app_host = 'http://localhost'

   # The snap will already be running, no need to run a server.
   config.run_server = false
end

Capybara::Webkit.configure do |config|
   # Don't raise errors when SSL certificates can't be validated
   config.ignore_ssl_errors

   # Raise JavaScript errors as exceptions
   config.raise_javascript_errors = true

   # Allow pages to make requests to any URL without issuing a warning.
   config.allow_unknown_urls
end

RSpec.configure do |config|
   config.expect_with :rspec do |expectations|
       expectations.include_chain_clauses_in_custom_matcher_descriptions = true
   end

   config.mock_with :rspec do |mocks|
       mocks.verify_partial_doubles = true
   end

   config.shared_context_metadata_behavior = :apply_to_host_groups

   # Webkit needs a framebuffer in order to work, so create one before the tests
   # run, and destroy it afterward.
   config.before(:all) do
       @headless = Headless.new
       @headless.start
   end

   config.after(:all) do
       @headless.destroy
   end
end

With the generic setup out of the way, I decided to write a simple test that would have caught the issue introduced in v12: just login with and without valid credentials (this test assumes the existence of an “admin” user whose password is “admin”).

feature "Logging in" do
   scenario "Logging in with correct credentials" do
       visit "/"
       fill_in "User", with: "admin"
       fill_in "Password", with: "admin"
       click_button "Log in"
       expect(page).to have_content "Documents"
   end

   scenario "Logging in with incorrect credentials" do
       visit "/"
       fill_in "User", with: "wronguser"
       fill_in "Password", with: "wrongpassword"
       click_button "Log in"
       expect(page).to have_content "Wrong password"
   end
end

Short and sweet. In order to run this test, I created a Rakefile in the directory containing the spec/ folder, with the following contents:

require 'rake'
require 'rspec/core/rake_task'

RSpec::Core::RakeTask.new(:test) do |t|
   t.pattern = Dir.glob('spec/**/*_spec.rb')
end

task :default => :test

Now the entire suite of tests (even when more specs are added) can be run with rake test. I verified that these tests pass normally, and throw terrible javascript errors in v12, so this simple suite would indeed catch the issue that bit us. Time to get it into CI!

Circle CI Config

Since I needed to use that new executor type, I needed to use Circle CI 2.0. Here’s the YAML I ended up using:

version: 2
jobs:
 build:
   working_directory: ~/nextcloud-snap
   machine: true
   steps:
     - checkout
     - run:
         command: |
           sudo apt update
           sudo apt install -y snapd
           docker run -v $(pwd):$(pwd) -t ubuntu:xenial sh -c "apt update -qq && apt install snapcraft -y && cd $(pwd) && snapcraft"           
     - run:
         command: |
           sudo snap install *.snap --dangerous
           sudo apt install gcc g++ make qt5-default libqt5webkit5-dev ruby-dev zlib1g-dev -y
           sudo gem install bundle
           cd tests
           bundle install --deployment
           sudo nextcloud.manual-install admin admin
           bundle exec rake test

It’s beyond the scope of this post to teach the entire Circle CI syntax, but I want to go over the important parts in more detail: the two run steps (they’re run in order).

- run:
   command: |
     sudo apt update
     sudo apt install -y snapd
     docker run -v $(pwd):$(pwd) -t ubuntu:xenial sh -c "apt update -qq && apt install snapcraft -y && cd $(pwd) && snapcraft"

As I mentioned, the machine executor type is Trusty. Well, the Nextcloud snap depends on packages in the Xenial archives, so it needs to build on Xenial. So we fire up a Xenial Docker image just to build the snap. We mount the current working directory into place so that the built snap ends up on the host once complete.

- run:
   command: |
     sudo snap install *.snap --dangerous
     sudo apt install gcc g++ make qt5-default libqt5webkit5-dev ruby-dev zlib1g-dev -y
     sudo gem install bundle
     cd tests
     bundle install --deployment
     sudo nextcloud.manual-install admin admin
     bundle exec rake test

This step runs after the one that builds the snap, so the first thing we do is install the built snap. This works because as I said, snaps run on Trusty (given the right kernel). We then install the prerequisites for our test suite, create an “admin” user with its password set to “admin”, and then run the tests.

Conclusion

Building, installing/running, and testing the complete, complex snap is awesome. We don’t have to worry about any differing dependencies between production and testbed (because they’re literally the same binary), and Capybara works wonderfully well standalone in this manner. I’m really thankful for Circle CI actually supporting running snaps, particularly after hitting a wall with e.g. Travis.

Right now the snap is built and tested for every pull request in the snap. We need to start running these tests daily against the daily snaps as well, but unlike Travis (which recently introduced cron jobs), Circle CI does not natively support running jobs periodically. It does support triggering jobs via a web API though, so I may just trigger it from cron running on my own infrastructure. Beyond that, I look forward to fleshing the test suite out further. If you have good ideas, make a pull request! We’d love to see what you have in mind.