In the previous post of this series, we discussed the process of and lessons learned from building PHP 7 for the Nextcloud snap. Nextcloud supports a number of databases, but they recommend MySQL (the v9 docs that I used when making this decision are lost to the dust of time, but MySQL is still the recommended option in v21), so for this post that’s what we’ll focus on getting into the snap.

MySQL is included in the Ubuntu archives, and honestly something like this should work fine:

# ...
   parts:
       mysql:
           plugin: nil
           stage-packages: [mysql-server]
# ...

But it doesn’t.

Lesson 5: seccomp denials are not recoverable

I find that a lot of the people I talk to think of snaps as containers (they often compare them to docker). Yes, snaps are self-contained. Yes, snaps are isolated. But in my opinion, no, snaps are not containers. Admittedly they utilize some of the same underlying technology, but they aren’t in a chroot of any kind, they don’t get their own network namespace, etc. They’re simply very confined applications, and calling them containers is confusing to those who are familiar with other kinds of containers. It’s not my goal to discuss the entire security story behind snaps (check the whitepaper out if you’re interested), but you need to know a part of it for this lesson.

One of the confinement technologies used in snaps are seccomp filters, which allows one to provide a whitelist of system calls for a given application environment. This is one of the things handled by the concept of “interfaces”. When you specify interfaces for a given application, you expand the whitelist of syscalls it’s allowed to make (among other things). When configuring the whitelist, one specifies what should happen if a disallowed syscall is made. There are several options:

  • Kill the offending application immediately.
  • Notify the offending application via a signal.
  • Return an errno from the syscall.
  • Notify a tracer of the denial.
  • Allow the syscall.

Only two of these options were valid for the snap use-case: either kill the offending application, or return an errno from the syscall. The other options were I believe either racy or impossible to implement in a generic manner. The obvious winner there is to return an errno– let the application handle its own denial, maybe it can recover. However, therein lies a seccomp filters shortcoming: if one uses the errno option, none of the denials get logged! So the security team chose the last remaining option: kill the offending application immediately.

It’s terribly unfriendly to the application, but at least you get a warning about it in the syslog so you know what the issue is (and snappy-debug will make recommendations for you). It’s worth noting here that the security team is working with upstreams to enable logging for errno, so we hope to use that instead once it’s available. (Update: that work has completed, and snapd now uses errno instead of killing offending applications. Yes, it makes this lesson useless, but I’m leaving it intact for historical reasons.)

Anyway, MySQL attempts to control the priority of its page cleaner threads with a setpriority() syscall, which is not contained within any of snapd’s interfaces. MySQL makes this call, and rightly handles its failure, but snapd doesn’t give it a chance to handle the failure: it kills it immediately. The only way around this was to fork MySQL, apply a patch to make that syscall optional, and build the MySQL part from source. I’m trying to move that patch upstream, but the MySQL folks seem reluctant to accept it since they don’t understand why snapd doesn’t use errno instead of kill. I can’t really blame them, so I’ll probably be maintaining this fork until snapd uses errno.

Beyond that issue, MySQL was pretty easy to get working. It runs based on config files which can contain environment variables, so I just ship custom versions in the snap.

The next post in the series will discuss the memcache.