Snap updates: automatic rollbacks

When researching snaps, one of the main advantages everyone talks about is the fact that they're transactionally updated. That is, an upgrade either succeeds or fails, it doesn't leave the snap in a broken state. If you have snap A installed and an update for it is released, it'll automatically update. If that update is somehow broken, the snap will roll back to the previously-working revision. However, no one has really talked about how that works. How does a snap know that an update is broken? That's what I want to talk about here.

Snapd won't check the health of your snap for you, as it doesn't know enough about your snap to perform a decent health check. However, it's pretty easy to check the health yourself.

Snapd recently gained support for hooks, which is a way for snapd to notify individual snaps about various events. There's one hook in particular that supports configuration. It's beyond the scope of this post to walk through configuration itself, but there's something important about this hook that's worth knowing: it runs upon initial install, and it runs when the snap is updated, after the snap's services are started. This makes it a decent place to perform a health check.

I'd like to explain with an example, but first we need to make sure we're on the same page. This support is pretty new, and you need to be running the right version of snapd.

Prerequisites

First of all, verify that you're at least on Xenial running version 2.17.1ubuntu1:

$ snap --version
snap 2.17.1ubuntu1
snapd 2.17.1ubuntu1
series 16
ubuntu 16.04

Also make sure your core snap (it could also be called ubuntu-core, that's okay) is up to date (hook support was only recently added):

$ sudo snap refresh

Now get on with it

Now that we're on the same page, let's continue with our example. Hooks are simply executables contained within the meta/hooks directory inside the snap. Snapcraft doesn't yet have explicit support for hooks (update: it does now), but we can bend it to our will for this example.

Let's first create a directory for our new snap, and create the snapcraft.yaml for it:

$ mkdir rollback-test
$ cd rollback-test && snapcraft init
Created snapcraft.yaml.
Edit the file to your liking or run `snapcraft` to get started

Edit that snapcraft.yaml and make it look like this:

name: rollback-test
version: '1.0'
summary: Snap to test rollback functionality.
description: You heard me.

grade: stable
confinement: strict

parts:
hook:
plugin: dump
source: .

Now create a new file: meta/hooks/configure. The tree should look like this when you're done:

$ tree
.
├── meta
│   └── hooks
│   └── configure
└── snapcraft.yaml

Edit the meta/hooks/configure file and make it look like this:

#!/bin/sh
echo "This will succeed."
exit 0

Make that file executable (hooks must be executable):

$ chmod a+x meta/hooks/configure

And finally, create the snap:

$ snapcraft
Preparing to pull hook
Pulling hook
Preparing to build hook
Building hook
Staging hook
Priming hook
Snapping 'rollback-test' |
Snapped rollback-test_1.0_amd64.snap

Now you have a snap containing a configure hook that does exactly nothing other than exiting successfully. Install the snap:

$ sudo snap install --dangerous rollback-test_1.0_amd64.snap
rollback-test 1.0 installed

$ snap list
Name Version Rev Developer Notes
core 16.04.1 641 canonical -
rollback-test 1.0 x1 -

Take a careful look at the revision for the snap we just installd: x1. As you may know, revision numbers are assigned by the store. However, this snap didn't come from the store: we just created and installed it locally. Snapd knows working with revisions can be handy here, so it makes one up for you: The "x" indicates that it was a local snap, and the "1" indicates that it's been installed once. You could actually install it again over the top of it and get the revision to increment:

$ sudo snap install --dangerous rollback-test_1.0_amd64.snap
rollback-test 1.0 installed

$ snap list
Name Version Rev Developer Notes
core 16.04.1 641 canonical -
rollback-test 1.0 x2 -

You can see this works like an update from the store. You'll notice that you now have two revisions on your disk (with only one active, of course):

$ ls -l /snap/rollback-test/
total 0
lrwxrwxrwx 1 root root 2 Dec 12 12:47 current -> x2
drwxrwxr-x 3 root root 27 Dec 12 12:42 x1
drwxrwxr-x 3 root root 27 Dec 12 12:42 x2

Alright, let's get to the cool part. You probably didn't notice it, but your configure hook did actually run as part of the build step. Check this out:

$ snap changes
ID Status Spawn Ready Summary
7 Done <snip> <snip> Install "rollback-test" snap [...]
8 Done <snip> <snip> Install "rollback-test" snap [...]

I've cleaned the output up a little, but you get the idea. You see two changes there corresponding to the two back-to-back installs we did. Let's take a closer look at that most recent change:

$ snap change 8
Status Spawn Ready Summary
Done <snip> <snip> Prepare snap "/tmp/snapd-sideload-pkg-798830678" (unset)
Done <snip> <snip> Mount snap "rollback-test" (unset)
Done <snip> <snip> Stop snap "rollback-test" services
Done <snip> <snip> Make current revision for snap "rollback-test" unavailable
Done <snip> <snip> Copy snap "rollback-test" data
Done <snip> <snip> Setup snap "rollback-test" (unset) security profiles
Done <snip> <snip> Make snap "rollback-test" (unset) available to the system
Done <snip> <snip> Start snap "rollback-test" (unset) services
Done <snip> <snip> Clean up "rollback-test" (unset) install
Done <snip> <snip> Run configure hook of "rollback-test" snap if present

Notice that last step: Run configure hook [...] if present. So you can see that the snap's services start up, and then the configure hook runs. This means that your configure hook can make sure your services are running correctly, etc. But how would it tell snapd if something is wrong, i.e. the snap needs to be rolled back? That's easy: fail.

Alter your meta/hooks/configure hook and make it look like this:

#!/bin/sh
echo "This will fail."
exit 1

Notice we're exiting non-zero here-- the hook will fail. Rebuild your snap so that it includes the new hook:

$ snapcraft clean
Cleaning priming area for hook
Cleaning staging area for hook
Cleaning build for hook
Cleaning pulled source for hook
Cleaning up snapping area

$ snapcraft
Preparing to pull hook
Pulling hook
Preparing to build hook
Building hook
Staging hook
Priming hook
Snapping 'rollback-test' |
Snapped rollback-test_1.0_amd64.snap

Now try to install it again:

$ sudo snap install --dangerous rollback-test_1.0_amd64.snap
error: cannot perform the following tasks:
- Run configure hook of "rollback-test" snap if present (This will fail.)

Neat, huh? The failing hook actually made the installation fail. What does that entail, then? What is the state of the snap? See for yourself:

$ snap list
Name Version Rev Developer Notes
core 16.04.1 641 canonical -
rollback-test 1.0 x2 -

If the installation succeeded, we'd be on revision x3. However, the installation failed, so snapd took action: it went back to the revision it knew was previously working: x2.

Conclusion

I obviously walked through this example locally without touching the store, but the exact same principle applies if the snap is coming from the store and the update is caused automatically. If you published the first snap we made, and then published the second version of it as an update, the update would have reverted just like it did in our example. Give it a try yourself!

Admittedly this example was simplistic. However, hopefully you can see how this would expand to a snap-wide health check. Since the configure hook runs after the snap's services are fired up, you can communicate with them, verify they're running correctly, and exit non-zero if you detect a problem. This would allow snapd to take remedial action and undo the upgrade.

Comments

Snapcraft tells me that the dump plugin requires a source. I was able to get your example to work by adding `source: .`

By Matthew Borger

Reply

Ah right sorry, dump got a default source in snapcraft v2.23, which is what I was using. I've added a `source` key there to be backward compatible. Thanks for the heads up!

By Kyle

Reply