ci: build the image with buildah instead of the host docker daemon #2

Merged

aiko merged 1 commit from ci/buildah-image-build into main

2026-08-14 09:06:49 +00:00

aiko commented

2026-08-14 09:05:13 +00:00

Owner

The image build fails with:

permission denied while trying to connect to the docker API at unix:///var/run/docker.sock

Not a Containerfile problem — the build never starts. docker login talks to the registry rather than the daemon, so the first daemon contact is docker build, and that is where it dies.

Permission denied, not "no such file": the socket is mounted, the job just cannot use it. That is a property of the runner host — socket ownership, mode, or which uid the job container runs as — and this repo has no way to control it or to notice when it changes. It worked before and stopped working without anything here changing.

The fix

Build with buildah. It is daemonless: no socket, no group membership, nothing outside the job that has to keep working. The failure mode stops existing rather than getting patched.

Three flags are load-bearing, and each was established by running it rather than by reading documentation:

Flag	Why
`--storage-driver vfs`	An unprivileged job container has no fuse-overlayfs. Slower than overlay; fine for a manual image build.
`crun` installed alongside `buildah`	Ubuntu's `buildah` package pulls no OCI runtime. Without one, every `RUN` fails with `error running container: ... exec: no command`.
`--isolation chroot`	The default isolation needs namespace setup the job container does not have.

The crun one is worth noting: without it the failure is exec: no command, which reads like a broken Containerfile rather than a missing dependency. That cost a debugging cycle locally and would have cost a red CI run otherwise.

Rehearsed, not assumed

Ran buildah inside an unprivileged ubuntu:24.04 container — the closest local analogue to the job environment — building a Containerfile and reading a value back out of the built image:

uid=0(root) gid=0(root) groups=0(root)
Version:         1.33.7
--- bud:
Successfully tagged localhost/local/test:latest
--- verify:
pretend build for 2.8.1

The first attempt, without crun and with default isolation, failed exactly as described above. That is why both are in here.

Also added

A verification step. After pushing, it pulls the first tag back and runs spacetime --version inside it. The Containerfile already asserts the version during build, but that only proves the layer built — not that this is what reached :latest. What consumers pull is the whole point of pinning, so that is what gets checked.

A diagnostics step printing id, buildah version, and the socket's status. Cheap, and it is the evidence anyone needs if this environment shifts again.

Unblocks

mobrule #273 (the SpacetimeDB 2.8.1 crate bump) cannot go green until an image carrying the pinned CLI is published — check-bindings-sync.sh regenerates bindings with whatever CLI the container has and fails on any diff.

The image build fails with: ``` permission denied while trying to connect to the docker API at unix:///var/run/docker.sock ``` Not a Containerfile problem — the build never starts. `docker login` talks to the registry rather than the daemon, so the first daemon contact is `docker build`, and that is where it dies. **Permission denied, not "no such file"**: the socket is mounted, the job just cannot use it. That is a property of the runner host — socket ownership, mode, or which uid the job container runs as — and this repo has no way to control it or to notice when it changes. It worked before and stopped working without anything here changing. ## The fix Build with **buildah**. It is daemonless: no socket, no group membership, nothing outside the job that has to keep working. The failure mode stops existing rather than getting patched. Three flags are load-bearing, and each was established by running it rather than by reading documentation: | Flag | Why | |---|---| | `--storage-driver vfs` | An unprivileged job container has no fuse-overlayfs. Slower than overlay; fine for a manual image build. | | `crun` installed alongside `buildah` | Ubuntu's `buildah` package pulls **no OCI runtime**. Without one, every `RUN` fails with `error running container: ... exec: no command`. | | `--isolation chroot` | The default isolation needs namespace setup the job container does not have. | The `crun` one is worth noting: without it the failure is `exec: no command`, which reads like a broken Containerfile rather than a missing dependency. That cost a debugging cycle locally and would have cost a red CI run otherwise. ## Rehearsed, not assumed Ran buildah inside an unprivileged `ubuntu:24.04` container — the closest local analogue to the job environment — building a Containerfile and reading a value back out of the built image: ``` uid=0(root) gid=0(root) groups=0(root) Version: 1.33.7 --- bud: Successfully tagged localhost/local/test:latest --- verify: pretend build for 2.8.1 ``` The first attempt, without `crun` and with default isolation, failed exactly as described above. That is why both are in here. ## Also added **A verification step.** After pushing, it pulls the first tag back and runs `spacetime --version` inside it. The Containerfile already asserts the version during build, but that only proves the layer built — not that this is what reached `:latest`. What consumers pull is the whole point of pinning, so that is what gets checked. **A diagnostics step** printing `id`, `buildah version`, and the socket's status. Cheap, and it is the evidence anyone needs if this environment shifts again. ## Unblocks `mobrule` #273 (the SpacetimeDB 2.8.1 crate bump) cannot go green until an image carrying the pinned CLI is published — `check-bindings-sync.sh` regenerates bindings with whatever CLI the container has and fails on any diff.

aiko added 1 commit

2026-08-14 09:05:13 +00:00

ci: build the image with buildah instead of the host docker daemon ac81d5143f

The docker path talked to the runner host's daemon over /var/run/docker.sock
and started failing with `permission denied while trying to connect to the
docker API`. The socket was mounted but unusable — a runner-host property this
repo cannot control, and one that can regress again from a reconfiguration
nobody here would see.

buildah is daemonless, so there is no socket, no group membership, and nothing
outside the job that has to keep working.

Three flags are load-bearing and were established by running it, not by reading
docs. `--storage-driver vfs` because an unprivileged job container has no
fuse-overlayfs. `crun` installed alongside buildah because the Ubuntu package
pulls no OCI runtime, and without one every RUN fails with `exec: no command`.
`--isolation chroot` because the default needs namespace setup the job
container does not have.

Also adds a step that pulls the pushed tag back and runs `spacetime --version`
against it. The Containerfile already asserts the version at build time, but
that proves the layer built, not that this is what reached :latest — and what
consumers pull is the entire point of the pin.

Rehearsed locally: buildah in an unprivileged ubuntu:24.04 container builds a
Containerfile and reads the value back out of the built image.

aiko merged commit cf7d14b896 into main

2026-08-14 09:06:49 +00:00

aiko referenced this pull request from a commit

2026-08-14 09:06:51 +00:00

Merge pull request 'ci: build the image with buildah instead of the host docker daemon' (#2) from ci/buildah-image-build into main

aiko referenced this pull request

2026-08-14 09:52:30 +00:00

ci: put buildah login flags before the registry #3