VPC Atlas

What a one-person CLI needs before you can trust its binary

How I built the CI/CD and supply chain of VPC Atlas, and the commands that let you check every download.

By Karina Pérez · · 9 min read

Why a small tool gets this much pipeline

The tool only makes read-only calls, but it runs with credentials for an AWS account, so whoever installs it should know what they are running.

Only the maintainer merges and releases, and that's me. CODEOWNERS has one rule with one name. With no second reviewer to catch a mistake or a shortcut, I wrote down the checks a reviewer would make and had a machine run them on every change, in public.

I wanted anyone who downloads a file from the releases page to be able to find out which workflow built it, and from which commit, without taking my word for it. What follows is each decision and why I made it. The reasons carry over to other projects. The YAML only fits mine.

The shape of the pipeline

There are five workflows. ci runs on every pull request and every push to main. release runs when I push a tag. CodeQL and OpenSSF Scorecard also run every Monday, and nightly checks run at 05:00 UTC for what can break without a code change: pip-audit on the installed packages, the links in the docs, a layout benchmark and a Trivy scan of the published image. Every job has a timeout. Only a tag of the form vX.Y.Z publishes:

on:
  push:
    tags: ["v[0-9]+.[0-9]+.[0-9]+"]
  workflow_dispatch:

permissions:
  contents: read

What runs on every change

ci has nine jobs and none waits for another, so a change takes as long as its slowest job. A newer push to the same branch cancels the older run. The job names are the required checks of main, and direct commits to main are not accepted.

Lint and types. ruff and mypy on the Python, actionlint and zizmor on the workflow files. Any finding fails the job. The workflows can publish my releases, so they get the same checks as the Python.

Tests. Linux on Python 3.12 and 3.13, macOS 14 and Windows 2022. Only the Linux 3.12 leg measures coverage, with --cov-fail-under=88.

Docs and package. mkdocs build --strict fails on a broken link, and the sdist and wheel are built from a hash-pinned lock. Dependency review blocks pull requests that add a dependency with a high-severity advisory.

Images. The dashboard and Lambda images are built and started. The dashboard must answer /healthz, and the Lambda image must answer an HTTP API event sent through the Runtime Interface Emulator. Trivy fails the job on a fixable critical vulnerability. The timeout is 40 minutes, the highest in ci.

Helm chart. helm lint --strict and helm template run with the defaults and with every option on. Then the job builds the image from the commit, starts a kind cluster, installs the chart, runs helm test, reaches the dashboard through a port-forward, upgrades with every option on and tests again. The timeout is 30 minutes.

Installers. On windows-2022, ubuntu-22.04 and macos-14, the job builds the archive, pipes the install script into the shell the way curl -fsSL https://vpcatlas.com/install.sh | sh does, and runs vpcatlas --version. Then it hands the installer a checksum that does not match and requires a checksum mismatch failure, with nothing left behind. A check that has never been seen to fail proves little.

What a tag does

To release, I set the same version in pyproject.toml, src/vpcatlas/__init__.py and deploy/helm/vpcatlas/Chart.yaml, merge that through a pull request, tag the merge commit and push the tag. The first job refuses to go on unless the tag and all four version fields agree:

chart=$(sed -n 's/^version: *\(.*\)$/\1/p' deploy/helm/vpcatlas/Chart.yaml)
app=$(sed -n 's/^appVersion: *"\(.*\)"$/\1/p' deploy/helm/vpcatlas/Chart.yaml)
if [ "$package" != "$chart" ] || [ "$package" != "$app" ]; then
  echo "pyproject.toml says ${package} but Chart.yaml says version ${chart} and appVersion ${app}" >&2
  exit 1
fi
if [ "${GITHUB_REF_TYPE}" = "tag" ] && [ "${GITHUB_REF_NAME#v}" != "$package" ]; then
  echo "Tag ${GITHUB_REF_NAME} does not match the version ${package}" >&2
  exit 1
fi

Then, in order:

  1. binaries are built with PyInstaller for Linux, macOS on Intel and Apple silicon, and Windows. Each build is run, and the binary from the unpacked archive must print --version in under three seconds. The Linux x86_64 build also writes the SBOM.
  2. release writes SHA256SUMS, checks it with sha256sum -c, attests every archive, the checksum file and the SBOM, and creates the GitHub Release.
  3. docker and lambda build each image (the dashboard one for linux/amd64 and linux/arm64), scan it with Trivy, and stop before the push on a fixable critical vulnerability. Then they push, sign by digest with cosign and attest.
  4. chart pushes the Helm chart to ghcr.io/karina-pp/charts/vpcatlas, signs and attests it, and pulls it back to verify both.
  5. pypi publishes through trusted publishing: PyPI trusts the workflow run, so no token exists. A repository variable switches the job on, because the publisher must be registered on pypi.org first.
  6. verify runs last. Every job before it tests my files. This one tests what users get.

It runs curl -fsSL https://vpcatlas.com/install.sh | sh on Linux and macOS and irm https://vpcatlas.com/install.ps1 | iex on Windows, with the version pinned, from the live site and not from the commit. vpcatlas --version must print exactly vpcatlas X.Y.Z. On Linux it also verifies the attestation and signature of the published image. If the site, the script, the assets or the signatures are wrong, this job fails.

Started by hand from the Actions tab, the workflow tests and builds but publishes nothing, because every publishing job has a condition on the tag, so a manual run is the rehearsal.

Decisions that keep the supply chain small

Everything I pull from other repositories is pinned by content. The workflows use 25 actions from other repositories, each pinned to a full commit SHA with the version in a comment, and the one container action by its digest:

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
  with:
    persist-credentials: false

Base images are pinned by digest and Python requirements by hash, installed with --require-hashes. A tag can be moved after the fact. A commit SHA cannot. Dependabot proposes the bumps every Monday, with a seven-day cooldown, so a new release waits a week before it reaches me.

All five workflows start with permissions: contents: read. A job asks for more only for what it does, such as id-token: write to sign. All 21 checkouts set persist-credentials: false, so the token does not stay in the working copy.

Signatures and attestations use the short-lived identity GitHub gives each workflow run, and PyPI uses trusted publishing, so there is no signing secret to leak. The one long-lived credential left is the Docker Hub token, because Docker Hub does not accept GitHub's identity.

The release workflow restores no build cache, not even Trivy's, because a cache written by one run must never end up inside a published file.

There is one SBOM, a CycloneDX file written by Syft, that lists what is inside the Linux x86_64 build and is attested against that archive. The other archives are built from the same lock of dependencies, so one file stands for all of them.

After a release, CodeQL with the security-extended queries, Scorecard, pip-audit and a nightly Trivy scan of the image on Docker Hub keep watching what is already published. A failing night opens one issue labelled nightly, and later failures comment on it.

How to check a download

SHA256SUMS travels with the archive, so it only proves the file was not damaged on the way. The attestation proves the archive was built by this repository's release workflow, and that is the check that matters. You need the GitHub CLI and, for images, cosign 3 or newer:

gh attestation verify vpcatlas-X.Y.Z-linux-x86_64.tar.gz --owner karina-pp
gh attestation verify SHA256SUMS --owner karina-pp
sha256sum --check --ignore-missing SHA256SUMS       # macOS: shasum -a 256 --check --ignore-missing SHA256SUMS
gh attestation verify oci://index.docker.io/vpcatlas/vpcatlas:X.Y.Z --owner karina-pp
cosign verify index.docker.io/vpcatlas/vpcatlas:X.Y.Z \
  --certificate-identity-regexp '^https://github.com/karina-pp/vpcatlas/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

A successful run names the workflow and the commit that produced the file. If it fails, do not install the file. The install scripts run the archive check for you when gh is installed and signed in, and VPCATLAS_SKIP_ATTESTATION=1 turns it off.

What it costs, and what I would skip

The first cost is size: the five workflow files add up to about 1,650 lines of YAML. Upkeep is the second: a weekly batch of Dependabot pull requests, and runners that change under you. GitHub retired the Intel macOS 13 runner, so the release workflow now builds that binary on macos-15-intel. Tools disagree too: the lines that use the local setup action carry an ignore comment because zizmor and actionlint 1.7.12 cannot both be pleased. The third cost is the wait. In ci, the timeouts run from 10 minutes for the dependency review to 40 for the images.

For a smaller project, with one binary and no chart, Lambda image or installer script, I'd skip the kind cluster, the emulator test, the installer matrix, the nightly layout benchmark and Scorecard. Scorecard grades the repository, and the pins and permissions it grades do the protecting. I'd keep what costs almost nothing: pinned SHAs, contents: read by default, zizmor, Dependabot, SHA256SUMS with a build provenance attestation, and a last job that installs the published release the way a user does.

What is still missing

  • No code-signing certificate. The attestation tells you who built a file, but Windows and macOS do not read it, so SmartScreen and Gatekeeper can still warn about a file downloaded with a browser.
  • Verification depends on gh. Without gh installed and signed in, the installers only check the checksum, which proves the file arrived intact, not who made it.
  • Signatures start late. Signed images and SBOMs exist only for releases made by this pipeline. Earlier ones carry only the attestations of the archives and SHA256SUMS.
  • Five archives, sometimes four. Linux arm64 is built only while a repository variable is set, so a release can carry four.

Where to look

The workflows are in the repository, every run is on its Actions page, and the full list of checks is on the engineering page of the docs. What the tool itself draws is in the launch post. The package is on PyPI and the image on Docker Hub. If a step does not check what its name says, tell me.