New and improved way to review GitHub pull requests for snaps

Reviewing pull requests for snaps has been pretty terrible ever since snaps were introduced. There are a few very simple reasons:

1. Building snaps locally in order to review is too much work. This is a fine place to start, but it doesn’t scale, especially if it takes any amount of time to build the snap in question (mine takes over an hour).
2. Most CI engines use Docker. You can build snaps in docker with hacks and tweaks that sort of work sometimes, but it’s not a supported approach, and it’ll just randomly break next week.
3. Using private tokens to access the snap store from a fork will leak the token. You don’t want random people releasing snaps on your behalf. This concern is not limited to snaps, of course.

I dedicated an article to this back in 2019, introducing a solution that worked reasonably well for a number of years, but it had some downsides as well:

1. It used Docker. See (2), above.
2. It required your own infrastructure and a GitHub app, which was difficult to configure. No one wants to do that.

Ultimately, the GitHub Snap Builder fell apart for me personally because it used Docker, and Snapcraft broke my ability to build the snap I cared about (Nextcloud) in Docker. I needed a new solution, but I wasn’t in the mood to roll my own again. I bided my time, investigated, and experimented. Two years ago, I finally started down the path that ultimately led to success, and I’d like to share that with you.

GitHub Actions

For a long time, the Nextcloud snap used Circle CI as our CI system (Travis CI before that). It was the only CI system for years that could actually build snaps, since it supported a special runner type that was a virtual machine instead of being a Docker container. However, in 2021, they suddenly capped CI runtimes to one hour, with no way to override. We were forced to revisit the CI landscape, and were pleased to discover that GitHub Actions, introduced in the meantime, had all the capabilities we needed. We moved to that, and have been happy users ever since.

We were still in the lurch in terms of testing pull requests, though. We could build snaps in GitHub Actions, and run our tests against them, but without something like the now-defunct GitHub Snap Builder, we couldn’t upload them to the store. The Nextcloud snap is an open source project, and we receive pull requests from unknown contributors all the time. Using GitHub Actions to upload snaps to the store would leak our private tokens to those unknown contributors (see (3), above). Until 2020.

On December 15th, 2020, Jaroslav Lobacevski wrote a series of posts about GitHub Actions security. To be honest, I don’t actually know when the critical feature was added, but this series is what introduced me to it: the workflow_run trigger.

Documentation has always been the Achilles heel of GitHub Actions. All they have are reference docs that are so light on detail I find myself wince-laughing half the time. I still can’t read the docs on workflow_run and come away with any meaningful understanding, but Lobacevski’s blog posts convinced me it was what I needed. Let me explain by first explaining how the GitHub Snap Builder solved this problem.

GitHub Snap Builder

The GitHub Snap Builder obviously needs a token for it to be able to upload snaps to the snap store. It did this on pull requests from unknown contributors. How did it avoid leaking that token? By making a very clear delineation between trusted and untrusted code. It used two ephemeral containers:

1. A container to build the snap. This was an unprivileged container with no tokens to leak. It was responsible for building the snap from the untrusted code in the pull request.
2. A container to upload the snap. Once the snap was built by container 1, another container was fired up with the snap store token in its environment. It then used snapcraft to upload the snap to the store. Everything running in this container is trusted, coming from a known-good source. It’s uploading an untrusted snap, yes, but it’s not running anything out of that snap to do it. This process was hard-coded, and didn’t rely on (or run) anything in the pull request, so there’s nothing for a malicous pull request to compromise.

How does that relate to GitHub Actions?

The workflow_run trigger allows for the same untrusted/trusted code separation that was possible in the GitHub Snap Builder. It supports a pull request workflow like this:

1. Pull request is opened from fork foo/nextcloud-snap, branch bar
2. Snap is built of the untrusted code in the pull request, and automatically tested
3. That snap is saved as an artifact

Once that workflow finishes on that fork’s branch, it uses the workflow_run trigger to fire up a new workflow on the master branch: trusted code. That branch has access to the snap store token, downloads the snap from the pull request workflow’s artifacts, and uploads it. Again, all without running untrusted code.

Show me how

Ultimately you will need two workflow files, one for each side of the trust boundary. In other words, you need one to run on pull requests and build snaps, and one to run on master and upload snaps. I’ll reduce this to the minimum you need to make this happen, but if you want to see a real world example, see the ‘test.yml’ and ‘release-pr-snap.yml’ workflows in the Nextcloud snap

Generate new snapcraft token

The first thing you need to do is generate a token so you can authenticate with snapcraft without actually needing to provide a password. This also provides an opportunity to scope access to the specific snap and actions we’ll be doing, to minimize damage in case the token DOES leak. Do that by running a command like this:

$ snapcraft export-login token_file\
           --snaps=<snap-name>\
           --channels=<channel>\
           --acls=package_access,package_push,package_release

As you can see, that requires us to decide our release strategy: what channel do we use when releasing a snap built from a pull request? This is exactly what channel branches are for. In Nextcloud, we use latest/beta/pr-<pull request id>. So the command we use to generate a token that has the ability to upload ONLY the Nextcloud snap ONLY to latest/beta/<branch> (but NOT latest/beta or anything else) looks like this:

$ snapcraft export-login token_file\
           --snaps=nextcloud\
           --channels=latest/beta/*\
           --acls=package_access,package_push,package_release

Create a new environment: snapcraft

In order to protect access to the token you just generated, create a new environment in GitHub. I called mine “snapcraft.” Add a branch protection rule to only allow access from the master branch, and add a single secret, SNAPCRAFT_STORE_CREDENTIALS, set to the contents of the token_file you generated in the last step.

Untrusted code: build the snap

Alright, time for the workflow files. First create the workflow that is responsible for building (and potentially testing) the snap. I called mine .github/workflows/test.yml. Make it look something like this:

name: Run pull request tests

on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-20.04

    steps:
      - name: Check out code
        uses: actions/checkout@v3

      - name: Install dependencies
        run: sudo apt-get update -qq && sudo apt-get remove -qq lxd lxd-client && sudo snap install lxd && sudo lxd init --auto && sudo snap install --classic snapcraft

      # Using sudo here because our CI user isn't a member of the lxd group
      - name: Build snap
        run: sudo snapcraft --provider lxd

      # Just an example, of course. Remove this if you have no real tests
      - name: Test the snap
        run: ./run-tests

      # Upload the snap so that we can download it in our other workflow
      - name: Upload snap artifact
        uses: actions/upload-artifact@v3
        with:
          name: snap
          path: '*.snap'

That simply checks the code out, installs snapcraft (and lxd, which we’re using to build here for a nice clean environment), builds and tests the snap, and then saves it as an artifact.

Trusted code: upload the snap

Alright, now we’re ready to create the workflow that is responsible for uploading the snap. I called mine .github/workflows/release-pr-snap.yml. Make it look something like this:

name: Release snap for pull request

on:
  workflow_run:
    # This maps to the name of our workflow above
    # "when one of these workflows finishes, trigger me"
    workflows: ["Run pull request tests"]
    types:
      - completed

jobs:
  upload:
    runs-on: ubuntu-latest
    environment: snapcraft # request access to the environment we created

    # Only run if what triggered us was a workflow from a pull request,
    # and that workflow finished successfully.
    if: >
      ${{ github.event.workflow_run.event == 'pull_request' &&
      github.event.workflow_run.conclusion == 'success' }}      
    steps:
      - name: Determine pull request commit hash
        uses: actions/github-script@v6
        id: determine-sha
        with:
          result-encoding: string
          script: |
            const workflowRun = await github.rest.actions.getWorkflowRun({
              owner: context.repo.owner,
              repo: context.repo.repo,
              run_id: ${{ github.event.workflow_run.id }},
            });

            return workflowRun.data.head_sha;            

      - name: Set initial commit status
        uses: actions/github-script@v6
        with:
          script: |
            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: '${{ steps.determine-sha.outputs.result }}',
              context: 'Snap Publisher',
              state: 'pending',
              description: 'Preparing environment...',
            });            

      - name: Determine release channel
        uses: actions/github-script@v6
        id: determine-channel
        with:
          result-encoding: string
          script: |
            const pullRequests = await github.paginate(
              `GET /repos/${process.env.GITHUB_REPOSITORY}/pulls`,
              {
                owner: context.repo.owner,
                repo: context.repo.repo,
              }
            );

            const matchingPullRequests = pullRequests.filter(pr => pr.head.sha === '${{ steps.determine-sha.outputs.result }}');
            const pullRequestCount = matchingPullRequests.length;
            if (pullRequestCount != 1) {
              throw new Error(`Expected one pull request to contain commit, but got ${pullRequestCount}`);
            }

            return `latest/beta/pr-${matchingPullRequests[0].number}`;            

      - name: Download snap artifact
        uses: actions/github-script@v6
        with:
          script: |
            const artifacts = await github.rest.actions.listWorkflowRunArtifacts({
               owner: context.repo.owner,
               repo: context.repo.repo,
               run_id: ${{ github.event.workflow_run.id }},
            });

            const snapArtifacts = artifacts.data.artifacts.filter((artifact) => {
              return artifact.name == "snap"
            });
            if (snapArtifacts.length != 1) {
              throw new Error(`Expected one snap artifact, but got ${snapArtifacts.length}`);
            }

            var download = await github.rest.actions.downloadArtifact({
               owner: context.repo.owner,
               repo: context.repo.repo,
               artifact_id: snapArtifacts[0].id,
               archive_format: 'zip',
            });
            var fs = require('fs');
            fs.writeFileSync('${{github.workspace}}/snap.zip', Buffer.from(download.data));            

      - name: Extract snap artifact
        run: unzip snap.zip

      - name: Install snapcraft
        run: sudo apt-get update -qq && sudo snap install --classic snapcraft

      - name: Set uploading commit status
        uses: actions/github-script@v6
        with:
          script: |
            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: '${{ steps.determine-sha.outputs.result }}',
              context: 'Snap Publisher',
              state: 'pending',
              description: 'Currently uploading/releasing snap...',
            });            

      - name: Upload snap
        env:
          SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.SNAPCRAFT_STORE_CREDENTIALS }}
        run: snapcraft upload *.snap --release="${{ steps.determine-channel.outputs.result }}"

      - name: Set final commit status
        uses: actions/github-script@v6
        with:
          script: |
            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: '${{ steps.determine-sha.outputs.result }}',
              context: 'Snap Publisher',
              state: 'success',
              description: "Snap built and released to '${{ steps.determine-channel.outputs.result }}'",
            });            

      - name: Set error commit status
        uses: actions/github-script@v6
        if: failure()
        with:
          script: |
            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: '${{ steps.determine-sha.outputs.result }}',
              context: 'Snap Publisher',
              state: 'error',
              description: 'Snap failed to upload/release',
            });            

Because this code is running on another branch than the pull request’s workflow, there don’t seem to be any primitives available to accomplish most of what we need, so we lean pretty heavily on the GitHub API. I’d be willing to bet there are premade actions out there that could simplify some of this, but I was happy enough with this that I didn’t investigate too hard.

The first thing we do is take the triggering workflow run ID, and determine the commit on which that workflow ran. We need that so we can create new commit statuses, which ultimately results in feedback in the pull request’s “checks” section. Indeed, that is the second thing we do: create a new status saying that the Snap Publisher is “Preparing environment.”

Then we use that commit hash again and determine the pull request that contains it. Using that information, we’re able to calculate the release channel for this snap using the pattern we discussed previously: latest/beta/pr-<pull request id>.

After that, we download and extract the artifact saved by the “test.yml” workflow. At that point, we install snapcraft, so we can actually upload the snap. We update the commit status again, saying we’re uploading/releasing the snap, and then do exactly that: upload the snap and release it to the calculated channel. You’ll notice that’s where we actually use the token, defining the SNAPCRAFT_STORE_CREDENTIALS environment variable, which snapcraft will use to authenticate.

When that’s done, we update the commit status once more saying that it’s done, and providing the channel into which it was released, like so:

The very last step only runs if an error occurs elsewhere in the pipeline, and it simply updates the commit status to point out the failure.

Conclusion

After a few years of painful reviews, the Nextcloud snap project is once again able to more easily evaluate its pull requests. This new workflow not only makes it easier for maintainers to review, but also lowers the barrier to entry for other members of the community to help. Now we can ask for participants in a particular issue to test out new features or fixes without teaching them to do anything beyond that with which they are already familiar. This technique has certainly helped our community, and it’s my hope that the sharing of it can help yours.