Snapping Nextcloud: MySQL

In the previous post of this series, we discussed the process of and lessons learned from building PHP 7 for the Nextcloud snap. Nextcloud supports a number of databases, but they recommend MySQL, so for this post that's what we'll focus on getting into the snap.

MySQL is included in the Ubuntu archives, and honestly something like this should work fine:

# ...
parts:
mysql:
plugin: nil
stage-packages: [mysql-server]
# ...

But it doesn't.

Lesson 5: seccomp denials are not recoverable

I find that a lot of the people I talk to think of snaps as containers (they often compare them to docker). Yes, snaps are self-contained. Yes, snaps are isolated. No, snaps are not containers. Admittedly they utilize some of the same underlying technology, but they aren't in a chroot of any kind. They're simply very confined applications. It's not my goal to discuss the entire security story behind snaps (check the whitepaper out if you're interested), but you need to know a part of it for this lesson.

One of the confinement technologies used in snaps are seccomp filters, which allows one to provide a whitelist of system calls for a given application environment. This is one of the things handled by the concept of interfaces. When you specify interfaces for a given application, you expand the whitelist of syscalls it's allowed to make (among other things). When configuring the whitelist, one specifies what should happen if a disallowed syscall is made. There are several options:

  • Kill the offending application immediately.
  • Notify the offending application via a signal.
  • Return an errno from the syscall.
  • Notify a tracer of the denial.
  • Allow the syscall.

I'm not a member of the security team (so take my words with a grain of salt), but only two of these options were valid for the snap use-case: either kill the offending application, or return an errno from the syscall. The other options were I believe either racy or impossible to implement in a generic manner. The obvious winner there is to return an errno-- let the application handle its own denial, maybe it can recover. However, therein lies a seccomp filters shortcoming: if one uses the errno option, none of the denials get logged! So the security team chose the last remaining option: kill the offending application immediately.

It's terribly unfriendly to the application, but at least you get a warning about it in the syslog so you know what the issue is (and snappy-debug will make recommendations for you). It's worth noting here that the security team is working with upstreams to enable logging for errno, so we hope to use that instead once it's available.

Anyway, MySQL attempts to control the priority of its page cleaner threads with a setpriority() syscall, which is not contained within any of snapd's interfaces. MySQL makes this call, and rightly handles its failure, but snapd doesn't give it a chance to handle the failure: it kills it immediately. The only way around this was to fork MySQL, apply a patch to make that syscall optional, and build the MySQL part from source. I'm trying to move that patch upstream, but the MySQL folks seem reluctant to accept it since they don't understand why snapd doesn't use errno instead of kill. I can't really blame them, so I'll probably be maintaining this fork until snapd uses errno.

Beyond that issue, MySQL was pretty easy to get working. It runs based on config files which can contain environment variables, so I just ship custom versions in the snap.

The next post in the series will discuss the memcache.

Comments

Hi

I have a question about using MySQL with snaps. How can you manage the strucuture of the MySQL database with application update ? Someone ask me few days ago during a convention, but I didn't know how to answers this question.

I'll be very interested by a feedback on this

Thx in advance

By Winael

Reply

It depends on how the application you snapped manages such things. For example, Nextcloud's occ command has an "upgrade" subcommand that will take care of running necessary database migrations, and if one runs it when no upgrade is necessary it's a noop. So I simply threw it into the Apache service runner script, and it runs every time Apache fires up. A Ruby on Rails app would work similarly, having a command meant for migrating the database as necessary.

As soon as an "upgrade" hook exists (soon) such things would likely go there, but until that's in place I just run things like that upon service startup.

By Kyle

Reply

Hi,

Did you try to build snap with MySQL inside Raspberry PI2 ?

If use mysql definition in 'parts' from here: https://github.com/kyrofa/owncloud-snap/blob/master/snapcraft.yaml, it does not compile and gives the following error:

Scanning dependencies of target sql
[ 80%] Built target mysqlpump
[ 80%] Building CXX object sql/CMakeFiles/sql.dir/sql_yacc.cc.o
[ 81%] Building CXX object sql/CMakeFiles/sql.dir/sql_hints.yy.cc.o
[ 81%] Building CXX object sql/CMakeFiles/sql.dir/abstract_query_plan.cc.o
[ 81%] Building CXX object sql/CMakeFiles/sql.dir/sql_builtin.cc.o
[ 81%] Building CXX object sql/CMakeFiles/sql.dir/auth/sql_authentication.cc.o
[ 81%] Building CXX object sql/CMakeFiles/sql.dir/auth/sql_auth_cache.cc.o
[ 81%] Building CXX object sql/CMakeFiles/sql.dir/auth/sql_authorization.cc.o
[ 81%] Building CXX object sql/CMakeFiles/sql.dir/auth/sql_user_table.cc.o
[ 81%] Building CXX object sql/CMakeFiles/sql.dir/auth/sql_user.cc.o
[ 81%] Building C object sql/CMakeFiles/sql.dir/auth/password.c.o
[ 81%] Building CXX object sql/CMakeFiles/sql.dir/auth/password_policy_service.cc.o
[ 81%] Building CXX object sql/CMakeFiles/sql.dir/auth/sql_security_ctx.cc.o
[ 81%] Building CXX object sql/CMakeFiles/sql.dir/auth/service_security_context.cc.o
[ 81%] Building CXX object sql/CMakeFiles/sql.dir/keyring_service.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/bootstrap.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/conn_handler/connection_handler_manager.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/datadict.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/debug_sync.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/derror.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/discover.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/field.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/field_conv.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/filesort.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/filesort_utils.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/aggregate_check.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/geometry_rtree.cc.o
[ 82%] Building CXX object sql/CMakeFiles/sql.dir/gstream.cc.o
[ 83%] Building CXX object sql/CMakeFiles/sql.dir/handler.cc.o
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
sql/CMakeFiles/sql.dir/build.make:85: recipe for target 'sql/CMakeFiles/sql.dir/sql_yacc.cc.o' failed
make[2]: *** [sql/CMakeFiles/sql.dir/sql_yacc.cc.o] Error 4
make[2]: *** Waiting for unfinished jobs....
CMakeFiles/Makefile2:5929: recipe for target 'sql/CMakeFiles/sql.dir/all' failed
make[1]: *** [sql/CMakeFiles/sql.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2

Which options are possible to fix this ?

Thanks in advance .

By Andrey

Reply

Yeah, you're running out of RAM. Building MySQL uses a lot of it. When I was building natively on the rpi2, I inserted a flash drive and created 16GB of swap on it:

$ cd <usb drive>
$ sudo fallocate -l 16G swapfile
$ sudo chmod 600 swapfile
$ sudo mkswap swapfile
$ sudo swapon swapfile

Now your build should succeed. After a very long time :) . There's a reason I use the Launchpad snap builders now!

By Kyle

Reply