Docker-in-Docker

glci has comprehensive support for CI pipelines that use Docker-in-Docker services (docker:dind).

Automatic configuration#

glci injects a prologue into every job of a pipeline that has the embedded registry, and configures DinD services on top. Between them, glci automatically:

1. Mounts the Docker socket for DinD service containers
2. Injects buildkitd.toml marking the mock registry as insecure (HTTP) so buildx builders trust it
3. Injects daemon.json with insecure-registries so DinD’s dockerd trusts the mock registry
4. Handles TLS setup — waits for DinD to generate TLS client certs, configures DOCKER_CERT_PATH, DOCKER_TLS_VERIFY, and DOCKER_HOST with the correct TLS port. Only jobs that declare a DinD service wait; see Which jobs wait for DinD
5. Runs a daemon readiness gate — polls docker info until DinD is fully accepting connections, eliminating races where cert files exist but the daemon has not finished TLS initialization
6. Fixes architecture mismatches — e.g. CI downloads amd64 buildx binary but container runs arm64
7. Registers QEMU binfmt handlers for cross-platform builds (one-time, idempotent)
8. Configures pull-through mirroring — each pipeline’s DinD acts as a pull-through mirror of the embedded registry, so images pulled in one pipeline are cached for the next
9. Wraps docker buildx create to inject --config (registry trust) and --driver-opt network= (pipeline DNS) so custom builders can reach the embedded registry
10. Clears stale buildx state — removes ~/.docker/buildx/ inside the job container to prevent the docker image’s default builder config from conflicting with DinD TLS

Isolation model#

Each pipeline gets its own ephemeral /var/lib/docker — no shared mount, no mutex, no serialization. Concurrent DinD pipelines run in parallel without interference.

Per-job DinD networks#

Each job in a pipeline runs on its own Docker network, because glci enables gitlab-runner’s FF_NETWORK_PER_BUILD (the networks are created and named by gitlab-runner). This prevents a critical issue: when multiple DinD jobs run concurrently, they would all register a docker service alias. Docker’s embedded DNS would round-robin between the DinD daemons, causing TLS cert verification failures and unpredictable behavior. A network per build ensures each job’s docker hostname resolves to exactly one DinD instance.

The mock server container is attached to each per-job network so the DinD daemon can reach the embedded registry.

Each job’s docker hostname resolves to its own DinD instance — no cross-talk, no DNS round-robin. The single glci-mock container is attached to all per-job networks simultaneously so every DinD daemon can pull from and push to the embedded registry.

TLS and DinD#

glci handles the full DinD TLS lifecycle automatically. Here is what happens when a job with DOCKER_TLS_CERTDIR: "/certs" starts:

1. DinD service container starts generating TLS certificates at /certs
2. glci’s pre-build script polls for ca.pem, cert.pem, and key.pem to appear (up to 30 seconds). Only a job that declares a DinD service polls; any other job checks once and moves on — see Which jobs wait for DinD
3. Once certs are found, the script sets DOCKER_CERT_PATH, DOCKER_TLS_VERIFY=1, and fixes DOCKER_HOST to use port 2376 (TLS)
4. A readiness gate polls docker info until the daemon accepts TLS connections
5. Job scripts can then use docker commands normally

This matches the standard GitLab.com DinD setup where DOCKER_TLS_CERTDIR=/certs.

Which jobs wait for DinD#

The certificate wait costs up to 30 seconds, so it only runs for jobs glci can see a Docker daemon in: those declaring a service whose image name contains dind. Such a job gets a GLCI_DIND variable, and the injected prologue waits only when it is set.

This matters because DOCKER_TLS_CERTDIR is frequently set in a pipeline’s global variables: while only some jobs run a daemon. Gating on DOCKER_TLS_CERTDIR alone made every other job in the pipeline wait half a minute for certificates nothing was going to write, and then warn about their absence.

Jobs without GLCI_DIND still check once, and a job the pipeline has already aimed at a TCP daemon (DOCKER_HOST=tcp://…) is configured from certificates that are already in place. So a DinD service under a name glci does not recognise still works in the common setup — it just does not stall the pipeline to find out. Such a job does forgo the _glci_dind resource group, so it is not serialized against other DinD jobs, and if you want the full wait you can claim it: glci run <job> --env GLCI_DIND=true.

Certificates are only acted on for a job that declares a daemon or names a TCP endpoint, because /certs is a volume gitlab-runner reuses: another job’s certificates can still be sitting there, and a job talking to a mounted Docker socket must not be repointed at a docker host that does not exist.

Buildx and BuildKit#

glci wraps docker buildx create to transparently inject configuration that makes buildx work with the mock registry and pipeline network.

What glci injects#

The wrapper is injected via pre_build_script in the runner config. It intercepts docker buildx create and adds the required flags, leaving all user-specified flags intact. Everything else passes straight through to the real docker binary.

How the wrapper is installed#

glci prefers to move the real docker binary aside to docker.orig and put the wrapper in its place, because that also covers scripts that call /usr/bin/docker by absolute path.

A job image whose user cannot write that directory — image: docker: user:, or an image with a baked-in non-root USER — gets the wrapper in a private directory instead (/tmp/glci-bin.XXXXXX, created with mktemp and readable only by the job’s user), placed first on PATH, with the real binary left untouched and named to the wrapper through an exported GLCI_DOCKER_ORIG. glci cannot detect this in advance — an image’s own USER is invisible to it — so the fallback is decided at runtime, in the job.

Consequences worth knowing, all specific to that fallback:

• sudo docker … resets PATH, so it reaches the real binary directly and gets no flag injection. Custom builders created that way may not reach the embedded registry. (In the in-place case sudo finds the wrapper, since it is /usr/bin/docker.)
• A script invoking /usr/bin/docker by absolute path likewise bypasses the wrapper.
• If neither location can be written, glci warns and continues without the wrapper. A job is never failed over it.

Where the docker CLI config goes#

glci copies the registry credentials it forwards into the docker CLI’s configuration directory. If the job’s user cannot write it — the same non-root case as above, and the docker:* images ship a ~/.docker that a non-root user cannot write — glci creates a private directory instead and points the CLI at it by exporting DOCKER_CONFIG for the rest of the job. That is also where docker login will write, and where the CLI looks for cli-plugins.

Stale buildx state#

The docker:* images ship with ~/.docker/buildx/current pointing to unix:///var/run/docker.sock. When DOCKER_HOST is tcp://docker:2376 (DinD with TLS), the mismatch causes buildx to auto-create a builder with the docker-container driver, which spins up a separate BuildKit container that cannot verify the DinD TLS certificates. glci’s pre-build script removes ~/.docker/buildx/ to let buildx initialize fresh from DOCKER_HOST.

QEMU and cross-platform builds#

glci automatically registers QEMU binfmt handlers the first time a DinD job runs. This enables buildx to build images for architectures other than the host (e.g. amd64 on Apple Silicon, arm64 on amd64):

# What glci runs automatically (idempotent, cached after first success):
docker run --rm --privileged docker.io/tonistiigi/binfmt:latest --install all

If QEMU registration fails, the daemon logs a warning and cross-arch builds may fail. You can run the command manually to fix it.

Build layer cache#

glci does not inject cache flags automatically. To enable BuildKit layer caching across pipelines, use --cache-from / --cache-to with the embedded registry as the cache backend. The registry volume persists across daemon restarts, so cached layers survive between pipelines. No additional trust configuration is needed – the registry is already trusted by buildx via the injected buildkitd.toml.

docker-build:
  image: docker:27
  services: [docker:27-dind]
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - docker buildx create --use
    - docker buildx build
      --cache-from "type=registry,ref=$CI_REGISTRY_IMAGE/cache:latest"
      --cache-to "type=registry,ref=$CI_REGISTRY_IMAGE/cache:latest,mode=max"
      -t "$CI_REGISTRY_IMAGE:latest"
      --push .

Standard DinD pipelines (basic builds, multi-platform builds, no-TLS builds) work the same as on GitLab CI with no glci-specific changes needed. QEMU binfmt handlers are registered automatically for cross-platform builds.

Troubleshooting DinD#

docker: command not found#

The job container needs the Docker CLI installed. Use a docker:* image or install the CLI in your Dockerfile.

Cannot connect to the Docker daemon#

Permission denied moving the docker binary#

Fixed. Older versions installed the docker CLI wrapper by moving the real binary aside, which fails — and fails the job, before its script: runs — when the job image’s user cannot write that directory. glci now falls back to a wrapper on PATH; see How the wrapper is installed. On an older version, run the job as root (image: docker: user: "root").

cannot install the docker CLI wrapper / could not install the docker CLI wrapper#

glci found nowhere it could write the wrapper: neither the directory holding the docker binary nor /tmp. The job continues without it, so docker build, push and pull are unaffected; what is lost is the flag injection for docker buildx create, so a builder created in the job may not trust or reach the embedded registry. Run the job as a user that can write the docker binary’s directory, or make /tmp writable in the image.

A job with no DinD service warns DinD TLS certs not found after 30s#

Fixed: glci now waits only in jobs that declare a DinD service. Older versions waited whenever DOCKER_TLS_CERTDIR was set, which a pipeline commonly sets globally, costing every job 30 seconds. See Which jobs wait for DinD.

x509: certificate signed by unknown authority#

The DinD daemon or buildx does not trust the mock registry.

Restarting the daemon usually fixes this: glci daemon stop --force && glci daemon start.

exec format error#

The image architecture does not match the host. QEMU binfmt handlers may not be registered:

# Register manually
docker run --privileged --rm docker.io/tonistiigi/binfmt:latest --install all

glci runs this itself before cross-architecture builds; the image is [images] binfmt (see The images glci pulls for itself).

On Apple Silicon with Colima, use Rosetta for better amd64 emulation:

colima stop
colima delete
colima start --vm-type=vz --vz-rosetta --cpu 12 --memory 16

Concurrent DinD jobs interfere with each other#

This should not happen with glci’s per-job network isolation. If it does, check that the per-job networks are being created:

docker network ls --filter name=glci-net-

Each job should have its own gitlab-runner-created network, in addition to the pipeline’s own glci-net-{pipelineID}.

How it differs from production GitLab CI#