So you’re developing a complex application. Maybe it has some really specific dependencies, or requires a lot of setup. In many cases (such as my own), it’s a web application. I have a suite of tests, varying all the way from unit tests through integration tests. The latter typically uses Selenium, and I often integrate it with Sauce Labs. I’ve written an article about this before. However, even those integration tests aren’t testing the real application: it’s not running against a production database, it’s not running with a production web server, and so on. Doing this is not easy; oftentimes the solutions I see look like this:
- Create custom docker images with the rest of the system (dependencies, database, web server, etc.) setup correctly
- Inject the code into it
- Run tests against it
Not only this, but one needs to ensure one’s image stays in line with one’s actual dependencies and continues to mirror a production deployment.
There’s a better way: package your application as a snap.
I’ve written about the Nextcloud snap a number of times. It’s one of the more complex snaps out there, bundling Apache, MySQL, PHP-FPM, etc. It’s a production-ready snap, following Nextcloud’s recommended settings. This makes it a perfect testing target! Except… there are no hosted CI options that can actually run snaps. All the ones I tried didn’t have a kernel configured properly for snaps (e.g. Travis). I was starting to investigate integrating my private GitLab instance (with its own amazing CI) with GitHub just to run tests against the snap.
Then I heard about Circle CI. By default they run docker on Ubuntu Trusty, but they have actual VMs available as a beta (the
machine executor), which have a Xenial kernel and can actually run snaps! It’s a bit limited, as I’ll discuss, but it solves a real need at least for the Nextcloud snap, and I’d like to walk you through doing the same thing.
The Nextcloud snap publishes on a number of different channels:
stable: The current stable release of Nextcloud (currently v11.0.3. Yes, I know v12 is out, we’ll talk about that in a sec)
candidate: Release candidates, used for testing calls before releasing to
beta: Snap development branch. Automatically built as new features are added to the snap (e.g. when pull requests are merged for the snap)
edge: Daily builds of Nextcloud’s master branch
11/edge: Daily builds of Nextcloud’s v11 maintenance branch
12/edge: Daily builds of Nextcloud’s v12 maintenance branch
Any one of these could immediately be installed with:
$ sudo snap install nextcloud --channel=<channel>
It’s actually pretty neat: want to see what work went into Nextcloud yesterday?
snap install --edge nextcloud. It’ll automatically update daily, giving you the opportunity to easily track development.
We have all these things happening automatically, but with zero CI because as I mentioned, we couldn’t find any hosted CI solutions that were shiny enough to actually run snaps. Of course we didn’t release to
beta much less
stable without some extensive manual testing, but it wasn’t good enough: no one was testing
So Nextcloud version 12 was just released upstream, and we quickly discovered that it doesn’t work in the snap. If we were actually able to run tests against the snap we could have caught this before v12 was actually released!
A few weeks ago, I saw a forum post from Alan detailing his use of Circle CI for building/pushing snaps. Our builds are already automated so I wasn’t so interested in that, but what caught my eye was the fact that he was installing Snapcraft from the snap, i.e. Circle CI could run snaps! I took a closer look as soon as I could.
It turns out that, like Travis, Circle CI runs on Ubuntu 14.04 (Trusty). Snaps actually can run on Trusty, but the kernel was too old. However, instead of using Docker as an executor type, one can use the (still-in-beta)
machine executor type, which spins up a VM that is still based on Trusty, but uses a newer v4.4 kernel.
Once I enabled the Circle CI integration for the project, it was a simple matter of actually creating acceptance tests to run, and then creating the correct Circle CI configuration to build the snap and run the tests against it.
Creating acceptance tests
Nextcloud actually already has a suite of acceptance tests. However, they make a lot of assumptions about where they are and how they’re run, such that they wouldn’t work for our purposes.
I have a lot of experience writing Rails apps, and one of my favorite tools from that world is Capybara. I’ve never used it in a Ruby-only project, but it was a perfect fit for what I needed to do, so I went for it. I needed five gems– here’s my Gemfile:
Install those gems and their dependencies:
$ sudo apt install gcc g++ make qt5-default libqt5webkit5-dev ruby-dev zlib1g-dev $ gem install bundler $ bundle install
Then setup rspec:
$ rspec --init create .rspec create spec/spec_helper.rb
spec/spec_helper.rb is where we put some setup code for the tests:
With the generic setup out of the way, I decided to write a simple test that would have caught the issue introduced in v12: just login with and without valid credentials (this test assumes the existence of an “admin” user whose password is “admin”).
feature "Logging in" do scenario "Logging in with correct credentials" do visit "/" fill_in "User", with: "admin" fill_in "Password", with: "admin" click_button "Log in" expect(page).to have_content "Documents" end scenario "Logging in with incorrect credentials" do visit "/" fill_in "User", with: "wronguser" fill_in "Password", with: "wrongpassword" click_button "Log in" expect(page).to have_content "Wrong password" end end
Short and sweet. In order to run this test, I created a
Rakefile in the directory containing the
spec/ folder, with the following contents:
require 'rake' require 'rspec/core/rake_task' RSpec::Core::RakeTask.new(:test) do |t| t.pattern = Dir.glob('spec/**/*_spec.rb') end task :default => :test
Now the entire suite of tests (even when more specs are added) can be run with
Circle CI Config
Since I needed to use that new executor type, I needed to use Circle CI 2.0. Here’s the YAML I ended up using:
version: 2 jobs: build: working_directory: ~/nextcloud-snap machine: true steps: - checkout - run: command: | sudo apt update sudo apt install -y snapd docker run -v $(pwd):$(pwd) -t ubuntu:xenial sh -c "apt update -qq && apt install snapcraft -y && cd $(pwd) && snapcraft" - run: command: | sudo snap install *.snap --dangerous sudo apt install gcc g++ make qt5-default libqt5webkit5-dev ruby-dev zlib1g-dev -y sudo gem install bundle cd tests bundle install --deployment sudo nextcloud.manual-install admin admin bundle exec rake test
It’s beyond the scope of this post to teach the entire Circle CI syntax, but I want to go over the important parts in more detail: the two
run steps (they’re run in order).
- run: command: | sudo apt update sudo apt install -y snapd docker run -v $(pwd):$(pwd) -t ubuntu:xenial sh -c "apt update -qq && apt install snapcraft -y && cd $(pwd) && snapcraft"
As I mentioned, the
machine executor type is Trusty. Well, the Nextcloud snap depends on packages in the Xenial archives, so it needs to build on Xenial. So we fire up a Xenial Docker image just to build the snap. We mount the current working directory into place so that the built snap ends up on the host once complete.
- run: command: | sudo snap install *.snap --dangerous sudo apt install gcc g++ make qt5-default libqt5webkit5-dev ruby-dev zlib1g-dev -y sudo gem install bundle cd tests bundle install --deployment sudo nextcloud.manual-install admin admin bundle exec rake test
This step runs after the one that builds the snap, so the first thing we do is install the built snap. This works because as I said, snaps run on Trusty (given the right kernel). We then install the prerequisites for our test suite, create an “admin” user with its password set to “admin”, and then run the tests.
Building, installing/running, and testing the complete, complex snap is awesome. We don’t have to worry about any differing dependencies between production and testbed (because they’re literally the same binary), and Capybara works wonderfully well standalone in this manner. I’m really thankful for Circle CI actually supporting running snaps, particularly after hitting a wall with e.g. Travis.
Right now the snap is built and tested for every pull request in the snap. We need to start running these tests daily against the daily snaps as well, but unlike Travis (which recently introduced cron jobs), Circle CI does not natively support running jobs periodically. It does support triggering jobs via a web API though, so I may just trigger it from cron running on my own infrastructure. Beyond that, I look forward to fleshing the test suite out further. If you have good ideas, make a pull request! We’d love to see what you have in mind.