Troubleshooting

Start with glci doctor — it runs pre-flight checks on the container engine (Docker or Podman), the daemon, your GitLab token, CI config, and git repository in one command:

glci doctor

It always prints the full report, and exits 1 when a check hard-failed — engine not reachable, Podman API socket not reachable, CI config file missing, CI config does not parse, not a git repository, or an error while checking daemon status — so you can gate on it:

glci doctor && glci run

A stopped daemon, a missing GitLab token, and ignored project-config keys are reported as warnings (~) and keep exit 0: the daemon starts automatically on first use, a token is only needed for remote include: and private registries, and dropped keys are already neutralized. Remediation for a failing row is printed underneath it as an unsymboled → ... continuation line, so it never reads as an extra warning. The report is colored only when stderr is a terminal and NO_COLOR is unset. See glci doctor.

Daemon logs#

Check the daemon log first when something goes wrong:

glci daemon logs        # last 50 lines
glci daemon logs -F     # follow in real time
glci daemon logs -n 0   # full log

Or read directly:

cat ~/.glci/daemon.log
tail -f ~/.glci/daemon.log

Daemon won’t start#

If glci run fails with “daemon did not start within 30s” or “daemon exited during startup … (check ~/.glci/daemon.log)”:

Force a clean restart:

glci daemon stop --force
glci daemon start

Daemon crashes or misbehaves#

The daemon automatically recovers from crashes on startup. If issues persist:

glci daemon status
glci daemon stop --force
glci daemon start

# Nuclear option: clean all daemon state
glci daemon stop --force
rm -rf ~/.glci/daemon.pid ~/.glci/daemon.sock ~/.glci/daemon.log
glci daemon start

A resumed job’s log stops at the crash#

resume: recover trace for job 17897290900070001: request failed: ...
[runner #1/build-job] resume: cannot rebuild trace file ...
[runner #1/build-job] resume: cannot append to trace file ...
[runner #1/build-job] resume: mock server offers no complete line for job 17897290900070001, leaving ... as it is

When the daemon resumes a pipeline it rebuilds each affected job’s log file from the mock server’s copy of the trace, which is the only record that kept growing while the daemon was down. Any of these lines means it could not, so the job’s remaining output goes to the event bus alone (the first three leave the log file as the crash left it; the fourth means the rebuilt file itself could not be reopened for appending).

glci log <id> <job> still shows the full trace for as long as the pipeline is running, because that reads the live stream. Once the pipeline finishes the stream is gone and only the file remains, so the trace ends at the moment the daemon died even though the job ran to completion.

The last of the four lines is deliberate rather than a failure, and it is reported only when there is a surviving log to protect: the mock server had nothing complete to offer, and overwriting a log that still holds the pre-crash output with nothing would lose the better of the two records. A job killed before it printed a whole line has no such log, and it resumes normally, carrying the incomplete line forward instead of being refused.

[runner #1/build-job] resume: WARNING: job 17897290900070001 resumes with an unreconciled trace seam; its log may repeat the output printed after the resume

This one is not a truncated log but a duplicated one. Handing the live trace stream over to the rebuilt file leaves a gap about one frame wide, which the daemon closes by reconciling against the mock server’s copy, retrying a few times. When none of those attempts reaches the mock server the gap stays open, and the reconciliation at the end of the job resumes from it rather than from the end, so everything the job printed after the resume can appear in the log twice. The job itself is unaffected, and the duplicate is in the log file alone.

A resumed pipeline warns about the commit it dispatches against#

daemon: resume: warning: no commit recorded for this pipeline, dispatching against "..." from CI_COMMIT_SHA; jobs may fail to check it out
daemon: resume: warning: recorded commit "..." is not a commit id, dispatching against "..." from CI_COMMIT_SHA; jobs may fail to check it out
daemon: resume: warning: recorded ref "..." is not a ref name, dispatching against "..." from CI_COMMIT_SHA; jobs may fail to check it out

glci does not hand the runner your working tree’s HEAD. It serves a rewritten repository, so the commit jobs check out is a synthetic one, and the daemon records which commit and ref that was in order to resume against the same ones.

Any form of the warning means that record is unusable: the pipeline was prepared by a glci old enough not to write it, its bare repository was never built, or one of the recorded values was rejected as malformed. The commit and the ref are restored together or not at all, so a bad ref discards the recorded commit as well. The resumed jobs fall back to CI_COMMIT_SHA, which the served repository may not contain, in which case they fail in Getting source from Git repository with fatal: unable to read tree.

Start the pipeline again with glci run instead of resuming it.

Docker image missing or stale#

If glci run fails with glci Docker image not available:

make docker

After upgrading, always re-run make docker so the local image matches your binary. glci does not verify that the image matches the binary’s commit — it only picks the first image it finds, per the order below — so a stale image is not detected for you.

How the image is resolved#

glci picks its own container image in this order, using the first one it finds:

1. registry.gitlab.com/gitlab-org/ci-cd/runner-tools/glci:<commit> — the commit the running binary was built from, locally then pulled. Skipped entirely when the binary carries no commit (a plain go build) or a -dirty one (see below). Released binaries normally stop here, since CI publishes a tag per commit.
2. glci:local — a locally built image from make docker (local only; never pulled). Preferred over :latest because it matches the tree it was built from, while :latest can lag behind a new glci internal subcommand the in-container daemon depends on.
3. :latest — locally, then pulled.

Which step you land on depends on how the binary was built:

• make build / make install stamp the commit, so resolution starts at step 1. make docker tags the image with that same commit, so a make docker && make install pair matches exactly at step 1 — meaning docker ps shows the registry-qualified tag, not glci:local.
• A modified working tree is stamped <sha>-dirty. The stamp uses git status --porcelain, so untracked files count as dirty too — a single new file is enough. glci treats a -dirty stamp as unpublished and skips step 1 altogether: it never probes or pulls a -dirty tag, it goes straight to your glci:local, which is the image built from that same tree. No warning is logged, because that is the intended outcome. (make docker in a dirty tree does also tag its output <registry>:<sha>-dirty, but resolution never consults that tag.)
• Plain go build / go install leave the commit empty, so resolution starts at step 2.
• Released binaries normally match at step 1. If their commit tag is unreachable — offline, registry down, tag garbage-collected — they fall through to step 2 and will use a glci:local left over from an earlier make docker, which may be much older than the binary. glci logs daemon: warning: ... falling back to glci:local, which may not match this binary to ~/.glci/daemon.log when this happens. Only a binary stamped with a non-dirty commit can trigger it.

To force the published image instead of a local build, note that make docker applies three tags to the same image: glci:local, <registry>:<commit>, and <registry>:latest. docker rmi glci:local only drops one of them, so resolution finds the identical local build again at step 3 (or step 1, if the commit tag is still there). Untag all of them, then restart the daemon (the resolved image is cached for the daemon’s lifetime):

docker rmi $(docker image inspect glci:local --format '{{join .RepoTags " "}}')
glci daemon stop   # the next `glci run` starts a fresh one and pulls a published image

A bare docker pull <registry>:latest moves the :latest tag to the published image but leaves glci:local and the commit tag pointing at the local build, and both of those win over step 3 — so pulling alone does not switch you over. Under Podman, use podman rmi / podman image inspect; glci drives whichever engine [docker] engine selects, and its own error hints are printed with that engine’s binary name.

Pointing the image somewhere else#

The order above assumes registry.gitlab.com is reachable. Where it is not, set [images] in ~/.glci/config.toml rather than retagging images by hand:

[images]
registry = "proxy.corp.internal/mirror"     # rewrites the host, keeps the chain
# glci   = "proxy.corp.internal/tools/glci:latest"   # or name it outright

registry only swaps the registry host, so the commit-exact tag is still preferred. A per-image glci key is used verbatim — no tag guessing, no :latest fallback, and no glci:local.

The resolved sidecar reference is cached for the daemon’s lifetime, so run glci daemon stop after changing [images] glci or [images] registry — the next glci run starts a fresh daemon and re-resolves. ([images] upstreams is the exception: changing it recreates the mock container on the next run by itself.)

A digest-pinned [images] glci is accepted for Linux but refused when a named runner needs the Windows variant, since a digest cannot name it — use a tag there.

The same section covers the other four images glci pulls for itself (the runner, the helper, the utility image, and binfmt). Retagging a mirrored image to the reference glci expects does still work — glci checks locally before pulling — but it is undone by glci system prune, has to be repeated on every Docker host, and a leftover glci:local silently shadows the published image on the next upgrade. [images] has none of those failure modes. Run glci doctor to see what each of the five resolved to and why.

glci refuses to start: [images] ... is unusable#

Some [images] values are refused outright rather than warned about, because resolving them produces a plausible-looking reference pointing somewhere you did not mean — and glci runs these images with the container engine’s socket mounted. The run aborts before anything is pulled:

[images] in the global config is unusable: images.registry: "acmecorp" has no registry host, so it would resolve against Docker Hub — write it as a host and path, e.g. "proxy.corp.internal/mirror"

These are separate from the [images] warnings, which are shown by glci doctor and glci config but do not stop a run:

Podman#

glci can drive Podman instead of Docker. Select the engine in ~/.glci/config.toml ([docker] engine = "podman", or per-runner [runners.<name>] engine) — there is no --engine flag, and the key is ignored in a project .glciconfig.toml. glci doctor reports the active engine (Docker or Podman); for Podman it also checks the Docker-compatible API socket separately and hints how to start it if it isn’t running.

“cannot connect to Podman” / socket not running#

What you see: Pipelines fail to start with connection-refused errors, or glci doctor reports Podman is selected but unreachable.

Cause: glci talks to Podman over its Docker-compatible API socket, which is not running by default. podman info succeeds without it, which is why glci doctor probes the socket as its own check. Start podman system service:

# Rootless (recommended)
systemctl --user enable --now podman.socket

# Rootful
sudo systemctl enable --now podman.socket

# macOS / Windows
podman machine start

The socket lives at $XDG_RUNTIME_DIR/podman/podman.sock (rootless) or /run/podman/podman.sock (rootful); glci mounts it into the runner at /var/run/docker.sock. On Linux, glci doctor dials every candidate path (an explicit CONTAINER_HOST, then the rootless path, then the rootful one) and reports whichever is reachable, so a rootless setup is never pointed at a rootful socket it doesn’t use. The exception is a remote endpoint: when the resolved Podman endpoint is not a local unix:// path — CONTAINER_HOST, or failing that the default podman system connection, pointing at ssh:// or tcp:// — the socket is on the far side, so nothing on this machine can dial it and doctor reports “cannot verify from host” (a warning, not a failure) even on Linux.

Podman on macOS / Windows (podman machine)#

podman machine is supported. glci resolves the default connection from podman system connection list and exports both CONTAINER_HOST (the ssh:// URI) and CONTAINER_SSHKEY (that connection’s SSH identity). The identity matters: setting CONTAINER_HOST overrides the named connection podman would otherwise read the key from, so without CONTAINER_SSHKEY every later podman call against a healthy machine fails as “not reachable” (exit 125).

The socket glci mounts into containers is the path the machine connection reports — the path on the side the containers actually run on. A default podman machine is rootless inside the VM, so that is /run/user/<uid>/podman/podman.sock. The rootful /run/podman/podman.sock is only used as a fallback and only exists with podman machine init --rootful; mounting it on a rootless machine fails the container with statfs /run/podman/podman.sock: permission denied.

glci doctor reports the Podman socket as “cannot verify from host” on macOS and Windows — and on Linux too whenever the resolved endpoint is remote (ssh://, tcp://), since the socket is equally unreachable from this machine. That is expected, not an error: the check is a warning and does not affect doctor’s exit status. Verify the machine itself with podman machine list.

Docker unreachable but Podman is installed#

What you see: glci doctor fails the engine check with Docker not reachable and warns that Podman is installed.

Cause: Docker is the selected engine (explicitly, or because the running daemon resolved it at startup) but its socket is down, while the podman binary is on PATH. With the engine unset, auto-fallback only kicks in when Podman is actually reachable — an installed Podman whose API socket is down still lands here.

Fix: Either start Docker, or switch engines:

# ~/.glci/config.toml
[docker]
engine = "podman"

Then make sure the Podman API socket is running (see above). If Podman is not installed, doctor instead hints to start Docker (Docker Desktop, colima start, or sudo systemctl start docker).

glci doctor warns about the engine config#

The daemon resolves its engine once at startup, from the global config only. If [docker] engine in ~/.glci/config.toml changed after the daemon started, doctor warns that the running daemon still uses the old one — run glci daemon stop and it re-starts with the new engine on the next command.

A command stops with loading glci config: ... toml: line N#

What you see:

loading glci config: /path/to/repo/.glciconfig.toml: toml: line 2 (last key "skip.jobs"): expected a comma (',') or array terminator (']'), but got end of file

Cause: the named config file is not valid TOML. glci refuses to run rather than proceed with the skip rules, pipeline presets, job overrides and [gitlab] settings the file asked for silently missing.

The same message names ~/.glci/config.toml when the global file is the broken one.

Fix: correct the reported line. For the project file, glci config edit opens it even while it is broken; the global ~/.glci/config.toml has to be edited directly, since config edit never opens it. glci config show prints the same error until it parses, and glci doctor reports it as a failing Project config or Global config row and exits 1.

A broken project file no longer takes a valid global one down with it: whatever parsed is kept, so a dropped error can no longer revert your ~/.glci/config.toml settings (mock server port, Docker host) to built-in defaults behind your back. Resuming an interrupted pipeline stops as well, rather than running the remaining jobs without the [token], [skip] and [manual] rules the project file was holding them to. See Malformed config files.

glci says it is “ignoring” keys from .glciconfig.toml#

What you see: a line like

glci: warning: .glciconfig.toml: ignoring [docker] engine, [docker] privileged, [runners.gpu] docker_host — these settings are only honored in ~/.glci/config.toml, because a project file travels with the repository

glci doctor shows the same text as a ~ Project config row instead of a stray line.

Cause: the keys that decide which container endpoint glci talks to, how far it trusts that endpoint, and how much of the host it exposes to containers are global-config-only. A project config travels with the repository, so honoring them would let a cloned repo silently redirect glci to a container daemon of its choosing — along with your working tree, CI variables, and job token — or bind-mount your filesystem into every job. The affected keys are:

privileged applies to the runner container that holds the engine socket, and the three [paths] container dirs are pasted into gitlab-runner’s [runners.docker] volumes list, which accepts host:container binds — so a project container_cache_dir = "/:/hostroot" would mount your whole filesystem into every job.

Fix: move those keys to ~/.glci/config.toml. Everything else in the project config still applies. See Container-endpoint keys.

paths.container_cache_dir: ... is not a container path#

What you see:

paths.container_cache_dir: "/:/hostroot" is not a container path — it must be an absolute path such as "/cache"; a "host:container" value would bind-mount a host directory into every job container

Cause: container_builds_dir, container_cache_dir, and container_certs_dir end up in gitlab-runner’s [runners.docker] volumes list, which also accepts Docker’s host:container bind syntax. A value containing a colon is therefore a host bind mount, not a mount point, and is rejected wherever it came from.

Fix: use a plain absolute container path (/cache, /builds). container_certs_dir also accepts the Docker named-volume form (certs-vol:/certs), and Windows container paths (C:\builds) are accepted as-is.

A named runner still uses the global endpoint or template#

What you see: a project .glciconfig.toml redefines a runner that also exists in ~/.glci/config.toml, but its jobs keep going to the global docker_host/docker_context — and no key was reported as ignored.

Cause: for a name present in both files every field the global entry owns is carried over — the endpoint fields (engine, docker_host, docker_context, container_socket, tls_cert_path, tls_verify) and the config template — so a project entry cannot erase them either. A bare [runners.gpu] in a project file would otherwise move gpu-tagged jobs onto the local engine with no ignored key to warn about. A project config_template is used only where the global entry has none; when both set one, the global wins and the project’s is named in the ignored-keys warning.

Fix: change the endpoint in ~/.glci/config.toml, or use a different runner name for the project-local runner. See Runner defined in both config files.

Token forwarding rules in .glciconfig.toml are ignored#

What you see: [token] forward_host_token, [token] jobs/job_patterns/stages, or [pipelines.<name>.token] appear in the ignoring warning.

Cause: [token] decides whether your real GitLab token is substituted into CI_JOB_TOKEN for scripts the repository wrote, so a project config may only narrow it. A project forward_host_token = false is always honored; a project true is dropped when your global config explicitly set false; project filters are honored only when your global config sets none (two sets of regexes cannot be intersected). A project pipeline preset’s [token] section is dropped entirely if it would turn forwarding back on, since a selected preset’s [token] replaces the top-level one.

Fix: set the forwarding rules you want in ~/.glci/config.toml, or use --no-token for a one-off run. See Token forwarding in a project config.

Engine selection and auto-fallback#

With the engine unset, glci tries Docker first and auto-falls back to Podman when Docker is unreachable but Podman is. The fallback notice is printed to the terminal the first time the daemon starts (and logged in the daemon log); set engine = "podman" explicitly to silence it. An explicit engine = "podman" never falls back to Docker — if the Podman socket is down, the pipeline fails until you start it.

Note that engine is separate from [defaults] executor: the generated gitlab-runner config always uses executor = "docker" even under Podman, because gitlab-runner has no Podman executor. Do not set executor = "podman".

Docker-in-Docker under Podman#

Rootful Podman runs DinD jobs fine (privileged mode defaults on). Rootless Podman DinD is best-effort and unsupported — privileged nested containers and the docker service alias may not work. Use rootful Podman (or Docker) for pipelines that rely on DinD.

Segmentation faults on Apple Silicon (Colima)#

If jobs crash with signal: segmentation fault (core dumped) from Go toolchain binaries, Colima needs Rosetta enabled:

colima stop
colima delete  # needed if changing --vm-type
colima start --vm-type=vz --vz-rosetta --cpu 12 --memory 16

Git repo preparation#

Preparation fails with archive/tar: sockets not supported#

What you see: every glci run in every project stops before any job starts:

creating git repo tarball: creating tar.gz of bare repo: archive/tar: sockets not supported

The daemon log (glci daemon logs) shows the same thing with the project it gave up on:

daemon: pipeline preparation failed for /path/to/project: creating git repo tarball: ...

Cause: git’s builtin filesystem monitor (core.fsmonitor = true, typically set in ~/.gitconfig). glci serves your commit from a bare repo of its own, and while overlaying uncommitted changes it runs git against that repo with your working tree attached. git then starts git fsmonitor--daemon, which binds a Unix socket inside the bare repo — and a socket cannot be put in a tarball. The daemon is also orphaned when glci removes the repo afterwards.

Fix: upgrade glci — this is fixed in releases after v0.8.0. glci now turns the monitor and auto-gc off for the repos it owns and skips anything unarchivable, so your core.fsmonitor setting no longer matters.

On v0.8.0 or older, either skip the overlay (see Dirty mode):

glci run <job> --no-dirty

or turn the monitor off for your user:

git config --global core.fsmonitor false

The scope has to be global. A repo-local core.fsmonitor false has no effect here, because the failing command reads its config from glci’s bare repo, not from your project’s .git. To clean up daemons already orphaned, run git fsmonitor--daemon stop in the affected repo; pkill -f fsmonitor--daemon also works but stops the monitors serving all of your other repos too (git restarts those on demand).

Related: glci runs its own git commands with your GIT_DIR, GIT_WORK_TREE and GIT_CONFIG_* environment overrides stripped, so git settings you export through the environment (rather than a config file) do not reach them.

Pipeline hangs or won’t cancel#

glci stop <pipeline-id>

glci stop <id> handles orphaned pipelines automatically — if the daemon lost track of a pipeline (e.g., after a restart), it force-removes leftover containers and the pipeline network, then marks it as canceled in history. Restarting the daemon is only needed if force-stop itself doesn’t resolve the issue:

glci daemon stop
glci daemon start

A log line appears twice when attaching to a running job#

What you see: glci log <pipeline-id> <job-name> on a job that is still running prints one line (rarely a few) twice, right where the catch-up output ends and live output begins.

Cause: glci prints the trace recorded so far, then switches to the live stream. A line written at that exact moment is in both.

This is normal, and deliberate: the alternative ordering would drop such a line instead of repeating it. Reading the log again once the job has finished prints the stored trace, which has no such seam.

“Waiting for pipeline preparation to finish…”#

What you see: Running glci run shows Waiting for pipeline preparation to finish... and does not start immediately.

Cause: The daemon serializes pipeline preparation per directory. Another glci run in the same project is already being prepared, so your request is queued until it finishes.

This is normal. The pipeline will start automatically once the earlier preparation completes. Press Ctrl+C to cancel if you don’t want to wait.

glci rebuilds its cached copy of your repository#

What you see: preparation logs warning: bare repo cache entry is empty (no HEAD commit) or ... is corrupt (HEAD tree unreadable), and the run takes as long as a first run. The lines beginning warning: not caching bare repo — its objects are incomplete, it is shallow, its objects are borrowed through alternates, it has no readable HEAD tree — and warning: git repack failed, not caching this repo are the opposite end: glci declined to store what it had just built, so the next run will be slow too.

Cause: the cached copy under ~/.glci/bare-repos/ is damaged, or the repository glci built is one that would break later if it were reused. An entry is checked before it is published and is moved into place in a single step, so an interrupted run leaves no entry rather than a broken one — seeing the first pair normally means something outside glci reached the directory: a full disk, a filesystem error, a manual deletion, or an entry left by a glci older than these checks.

This is normal, and nothing damaged is ever cached. glci discards the entry and clones from your project instead. A stored repository missing its objects would otherwise be reused by every later run and fail inside the job with fatal: HEAD: not a valid SHA1 or an unreadable tree, which is what the checks exist to prevent — so glci would rather rebuild than keep one it cannot vouch for.

If the same warning appears on every run, the cache is not being replaced: most often the directory is unwritable, or the repository glci builds is one it refuses to store. glci system prune clears the cache entirely, which is also the way out if a stored copy turns out to be damaged in a way only the job notices.

Preparation pauses, then logs that the bare repo cache was unavailable#

What you see: waited 3.2s for the bare repo cache lock on <project>, or warning: bare repo cache unavailable, cloning from the project instead.

Cause: each project’s cached repository is guarded by a lock file, ~/.glci/bare-repos/<project>.lock, so that two runs — two pipelines in the same project, a cross-project trigger, or a second glci process — cannot rebuild or discard it underneath each other. One of them waited for another, and the first line is only logged when that wait exceeded a second.

This is normal. Holds are milliseconds in the ordinary case: building a new copy happens outside the lock, and a reader only holds it for a local hardlink clone. If glci gives up waiting it clones from your project instead, which costs time and never correctness. The lock is advisory and the operating system releases it when the process holding it exits, so do not delete the file — deleting it while a pipeline is preparing removes the protection rather than restoring anything. (glci system prune does clear the locks along with the cache, so run it when nothing is preparing.)

Config template errors#

Template parse error#

What you see: runner.config_template: invalid Go template: ... or runners.<name>.config_template: invalid Go template: ... in glci config output.

Cause: The Go template syntax is invalid (unclosed braces, unknown functions, etc.).

Fix: Check your template syntax. Common mistakes:

Template render error#

What you see: rendering custom template for runner "<name>": ... in daemon logs.

Cause: The template references a field that doesn’t exist in the template context.

Fix: Use only fields from the template variable reference. Common fields: .URL, .Executor, .DefaultImage, .PullPolicy.

Key not allowed in a project config template#

What you see: the pipeline fails with

daemon: /path/to/repo/.glciconfig.toml [runners.gpu] config_template sets [runners.docker] host, which a project config may not change because it controls where and how jobs run; move this setting to ~/.glci/config.toml

The message names the config file, the section and the key you wrote, so it points at the exact line to edit. For the default runner the section reads [runner] config_template; glci never says runner "default" or [runners.default], which are internal names you cannot edit.

Cause: a .glciconfig.toml travels with the repository, so its config_template is untrusted input. After rendering, it may not set keys that choose the container endpoint, grant host or privilege access, share namespaces, redirect name resolution, or run code outside the job script. The config_template_file form is checked identically — a path in the project file loads project-controlled content just the same.

Fix: move that key to your own ~/.glci/config.toml, where templates are unrestricted, or drop it. The full list is in restricted keys in a project config template.

Named runner name invalid#

What you see: runners.<name>: name must match [a-zA-Z0-9][a-zA-Z0-9_-]* in glci config output.

Fix: Runner names must start with a letter or digit and contain only letters, digits, hyphens, and underscores.

Jobs not routing to named runner#

What you see: A job runs on the default runner instead of the named runner.

Cause: None of the job’s CI tags: match any named runner name. Tag matching is exact and case-sensitive.

Fix: Ensure the job has a tags: entry that exactly matches the runner name defined in [runners.<name>]:

# .gitlab-ci.yml
my-job:
  tags: [gpu]    # must match [runners.gpu] in config
  script: echo "runs on GPU runner"

Docker networking issues#

Each build gets its own Docker network via FF_NETWORK_PER_BUILD. Job containers reach the mock server via extra_hosts (host-gateway). The runner and mock containers share the per-pipeline network for Docker DNS resolution. Problems here usually surface as connection timeouts or host resolution failures inside jobs.

Container can’t reach the mock server#

What you see: Jobs fail with errors like connection refused, no such host, or could not resolve host when trying to reach $CI_SERVER_URL or $CI_REGISTRY.

Debugging steps:

# List pipeline networks
docker network ls --filter name=glci-net-

# Inspect a specific pipeline network — look for Containers section
docker network inspect glci-net-<pipeline-id>

# Check if the mock server container is running and attached
docker ps --filter name=glci-mock

DNS resolution failures inside jobs#

What you see: getaddrinfo or DNS resolution failed errors for external hosts (e.g., registry.gitlab.com, github.com).

Cause: The job container’s DNS resolver can’t reach an upstream DNS server. Common with custom Docker networks and restricted host DNS configs.

Fix: Check the host’s /etc/resolv.conf or Docker’s DNS settings. If using Colima, restart with --dns to override:

colima start --dns 8.8.8.8 --dns 1.1.1.1

Or configure extra hosts in .glciconfig.toml to bypass DNS for known hosts:

[network.extra_hosts]
entries = ["internal-registry.corp:10.0.0.50"]

Port conflicts#

What you see: address already in use errors in the daemon log when starting a pipeline.

Cause: The mock server requires port 39741 (default) on the Docker host. This port may be taken by another process or a previous daemon that didn’t shut down cleanly. Registry listeners can also conflict if their bind addresses overlap.

Fix:

# Find what's using the port
lsof -i :39741

# Force-restart the daemon to clean up stale listeners
glci daemon stop --force
glci daemon start

You can change the mock server port or registry bind addresses in ~/.glci/config.toml:

[network]
mock_server_port = 39741           # default; change requires daemon restart
registry_bind = "127.0.0.1:0"      # HTTPS listener
registry_http_bind = "0.0.0.0:0"   # HTTP listener

Note: Changing mock_server_port requires a full daemon restart to take effect.

Token & authentication failures#

GitLab token not found#

What you see: gitlab token not set (set GITLAB_TOKEN env var or use --token) or no GitLab token or project configured, using offline parser.

Cause: No token is available, so glci falls back to the offline parser. This means include: project:, include: component:, and remote CI/CD variable fetching are all disabled. The same applies to every config glci parses, not just .gitlab-ci.yml – a child pipeline’s config and a cross-project trigger’s target config resolve their own includes too, and lose the same ones (see Child pipeline is missing jobs).

Fix: Configure a token using any of these methods (first match wins):

# Option 1: environment variable
export GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"

# Option 2: glab CLI (token is picked up automatically)
glab auth login

# Option 3: config file
cat >> .glciconfig.toml <<'EOF'
[gitlab]
token = "$GITLAB_TOKEN"   # env var references are expanded
EOF

# Option 4: command-line flag
glci run --token "glpat-xxxxxxxxxxxxxxxxxxxx"

To see which includes actually resolved, run glci merged – it prints the fully-resolved config on stdout and reports every skipped include on stderr.

For self-managed GitLab instances, also set the URL:

export GITLAB_URL="https://gitlab.example.com"
# or in .glciconfig.toml:
# [gitlab]
# url = "https://gitlab.example.com"

A URL set in the project config only gets a token that the same project config supplies (or an explicit --token) — tokens from GITLAB_TOKEN, glab, and ~/.glci/config.toml are withheld from it (next entries). Set both keys together in .glciconfig.toml, or set the URL via GITLAB_URL / ~/.glci/config.toml / --gitlab-url instead.

Child pipeline is missing jobs#

What you see: a child pipeline runs and passes, but jobs a shared template defines never appear. glci jobs -f child.yml lists them; the child pipeline does not. Or the trigger job fails with extends target ".template-job" not found (referenced by "job").

Cause: the child’s own config uses include: project: or include: component: and no token is configured, so the include is skipped. Whatever it defined is gone – silently if nothing referenced it, as a parse failure if a job extends: one of its definitions.

Both cases log a warning, in the trigger job’s output and in the daemon log:

skipping project include my-group/my-templates: no GitLab token configured

Fix: configure a token (above). If a token is configured and the include still skips, the URL may be withheld from it – see the next entry. To confirm what the child should contain, parse it at top level:

glci jobs -f child.yml

glci merged -f child.yml shows what the child’s includes resolved to, and names the skipped ones on stderr.

GitLab token withheld: “url comes from the project config”#

What you see: glci: warning: [gitlab] url "..." comes from the project config, which travels with the repository, so the token from your <source> was withheld, token = <withheld> in glci config --gitlab, and no include: project:/component: resolution. The <source> is whichever one was dropped: environment (GITLAB_TOKEN / GITLAB_PRIVATE_TOKEN), glab credential-helper, glab CLI config, or global config file (~/.glci/config.toml).

Cause: [gitlab] url is set in the project .glciconfig.toml. That file arrives with the repository, so glci sends a token to that URL only when the repository named the token too — [gitlab] token in the same project file — or when you named it yourself with --token. Everything else is withheld, including [gitlab] token from ~/.glci/config.toml: that is your credential for your own instance, so sending it to a repository-chosen host leaks it just as an environment or glab token would. This is deliberate, not a resolution bug.

Fix: opt in explicitly for that instance, or move the URL to your own config:

# Option 1: name the token for this invocation
glci run --token "$GITLAB_TOKEN"

# Option 2: pair url and token in the config file
cat >> .glciconfig.toml <<'EOF'
[gitlab]
url = "https://gitlab.example.com"
token = "$GITLAB_TOKEN"
EOF

# Option 3: set the URL yourself instead of letting the repo set it
export GITLAB_URL="https://gitlab.example.com"   # or ~/.glci/config.toml, or --gitlab-url

Check the URL against the instance you expect before doing any of this — the warning is how you find out that a repository tried to redirect glci. Option 2 pairs the project URL with an env-ref token, which glci reports separately (next entry). See [gitlab] url in a project config.

“the project config sends $VAR to [gitlab] url”#

What you see:

glci: warning: the project config sends $GITLAB_TOKEN to [gitlab] url "https://gitlab.example.com" — both the variable and the URL come from this repository, so check that you trust it with that value.

Cause: the project .glciconfig.toml sets both [gitlab] url and an env-ref token (token = "$GITLAB_TOKEN"), so the repository picked the host variable to read and where its value is sent. Nothing is blocked — this is the documented way to keep secrets out of the file, and a team’s own config legitimately does it — the request is authenticated as normal. The variable name is printed; its value never is.

Fix: nothing, if the config is your team’s own. Otherwise verify the URL before trusting it with that variable, and drop the [gitlab] block from the project config (set the URL and token in ~/.glci/config.toml, or pass --gitlab-url/--token) if you do not. See [gitlab] url in a project config.

CI_JOB_TOKEN permissions differ from production#

What you see: API calls inside jobs succeed locally but fail in real CI (or vice versa), because CI_JOB_TOKEN in glci is actually your personal access token (with different scopes).

Cause: glci forwards your host GitLab token as CI_JOB_TOKEN. In production, CI_JOB_TOKEN is a short-lived token scoped to the job.

Fix:

# Disable token forwarding entirely
glci run --no-token

# Or test with reduced secrets
glci run --secrets none

“API returned 401” or “API returned 403” when fetching variables#

What you see: Warnings in daemon logs like could not fetch project variables: API returned 401.

Instance-level CI/CD variables aren’t applied#

What you see: A variable defined as an instance-level CI/CD variable on a self-managed GitLab instance isn’t set in your jobs, or an include: that references it (e.g. project: $_GITLAB_TEMPLATES_REPO) fails with HTTP 404 because the variable wasn’t expanded.

Cause: glci fetches instance variables from GET /admin/ci/variables, which requires an administrator token. With a non-admin token GitLab returns 403 and glci skips them (could not fetch instance variables in the daemon log). Project and group variables are unaffected. Separately, remote variables are fetched in parallel with config parsing, so even an admin-fetched instance variable is not available for include: path expansion.

Fix: Supply the variable locally so it is available everywhere, including include resolution:

# .glci.env (gitignored), or --env KEY=VALUE, or a pipeline preset
_GITLAB_TEMPLATES_REPO=project/gitlab_templates
_GITLAB_TEMPLATES_REF=1.0.x

Instance variables are fetched under --secrets all (the default for glci run). See Variables & Secrets for the full precedence order and include-expansion behavior.

Secrets cache is stale#

What you see: Variable values don’t match what you see on GitLab, even after updating them.

Cause: Remote variables are cached in daemon memory for 6 hours by default.

Fix:

# Force a fresh fetch for this run
glci run --refresh-secrets

Or change the TTL in .glciconfig.toml:

[gitlab]
secrets_ttl = "0"   # disable caching entirely

Registry & image issues#

Push fails with “unknown blob” or “manifest invalid”#

What you see: docker push $CI_REGISTRY_IMAGE fails with errors about unknown blobs or invalid manifests.

Cause: The embedded registry lost its blob storage (e.g., after a docker volume rm glci-registry or glci system prune --all).

Fix:

# Rebuild and re-push — the registry volume was wiped
glci registry clean
glci run

“image blobs not found in local registry”#

What you see: glci registry pull fails with image blobs not found in local registry (image may have been proxied from upstream without caching blobs — re-push the image to persist it).

Cause: The image was pulled through the registry as a read-through cache hit. The manifest is stored locally but the blobs were streamed directly from upstream. Only images explicitly pushed to $CI_REGISTRY have their blobs stored.

Fix: Re-push the image from your pipeline (use docker push $CI_REGISTRY_IMAGE/...) so blobs are stored locally.

Insecure registry / TLS certificate errors#

What you see: x509: certificate signed by unknown authority or server gave HTTP response to HTTPS client when pulling from or pushing to the embedded registry.

Cause: The embedded registry uses a self-signed CA. glci automatically configures trust, but some scenarios break it:

Cross-platform image issues#

What you see: exec format error when running containers, or build failures with wrong-architecture binaries.

Cause: The image was built for a different CPU architecture (e.g., amd64 image on Apple Silicon).

Fix:

# Ensure QEMU binfmt handlers are registered (glci does this automatically)
docker run --privileged --rm docker.io/tonistiigi/binfmt:latest --install all

# For Colima with Rosetta (preferred for Apple Silicon)
colima stop
colima delete
colima start --vm-type=vz --vz-rosetta --cpu 12 --memory 16

If glci daemon logs shows warning: could not install QEMU binfmt handlers, QEMU registration failed. Run the docker run --privileged command above manually.

Note on glci’s own image: glci images built before this was fixed shipped an amd64 binary on every platform, because the Dockerfile put defaults on the BUILDPLATFORM and TARGETARCH build args and those defaults won over the requested platform. Two variants of the same bug:

• Published images — the linux/arm64 entry of a multi-arch tag contained the amd64 binary despite the manifest advertising arm64, so on arm64 hosts it ran under emulation or failed outright. Re-pull :latest to pick up a corrected build:

  docker rmi registry.gitlab.com/gitlab-org/ci-cd/runner-tools/glci:latest
  docker pull registry.gitlab.com/gitlab-org/ci-cd/runner-tools/glci:latest
  

  Re-pulling only helps if you track a moving tag. A pinned :<sha> or release tag is immutable — the image published for it stays amd64 forever, so move to a newer tag (or build locally with make docker) instead.

• Local make docker images — same cause, no registry involved: make docker runs a plain docker build with no --platform, so on Apple Silicon it also produced an amd64 glci:local. Re-running make docker now produces a native image.

To check any image yourself — 62 is x86-64, 183 is AArch64:

docker run --rm --entrypoint od \
  registry.gitlab.com/gitlab-org/ci-cd/runner-tools/glci:latest \
  -An -tu1 -j18 -N1 /usr/local/bin/glci

That inspects the variant your daemon picks for your own platform. Add --platform linux/amd64 or --platform linux/arm64 to inspect a specific entry of a multi-arch tag — running a non-native one needs the QEMU binfmt registration above.

$CI_REGISTRY images fail with wrong platform#

What you see: Jobs that use $CI_REGISTRY/... as their image fail with no matching manifest for linux/arm64/v8 (or another platform) even though the image exists on the real registry.

Pulling docker image 127.0.0.1:32768/group/project/image:v1.0 ...
ERROR: Job failed: failed to pull image "127.0.0.1:32768/group/project/image:v1.0"
  with specified policies [if-not-present]: Error response from daemon:
  no matching manifest for linux/arm64/v8 in the manifest list entries

Cause: $CI_REGISTRY resolves to the embedded registry (127.0.0.1:<port>), which proxies the image from upstream. When the upstream image is a multi-arch manifest list, the local Docker daemon selects its native platform (e.g., linux/arm64 on Apple Silicon), but the image may only provide linux/amd64 manifests.

This is common with projects that reference $CI_REGISTRY images to avoid external dependencies.

Fix: Two approaches:

1. Push multi-arch images — if you control the upstream images, build and push them as multi-arch manifests (e.g., with docker buildx build --platform linux/amd64,linux/arm64). This fixes the problem at the source for all consumers.

2. Use per-job field overrides to force the correct platform without touching .gitlab-ci.yml:

# .glciconfig.toml

# Nested form
[jobs."renovate_validate".image.docker]
platform = "linux/amd64"

# Or flat dotted key form (equivalent)
[jobs."renovate_validate"]
"image.docker.platform" = "linux/amd64"

# Or override the image entirely to bypass the registry proxy
[jobs."renovate_validate".image]
name = "registry.gitlab.com/group/project/image:v1.0"

# Or apply to every job in the project with a glob pattern
[jobs."*".image.docker]
platform = "linux/amd64"

Push-through mirror failures#

What you see: docker push succeeds locally but the image doesn’t appear on the upstream registry, or pushes fail with setting push-through config: HTTP 4xx/5xx.

# .glciconfig.toml
[registry]
push_through = true

[registry.upstream]
username = "deploy-token"
password = "$REGISTRY_WRITE_TOKEN"

Remote Docker issues#

When DOCKER_HOST points to a remote machine (TCP or SSH), the daemon and Docker daemon are on different hosts. This changes how bind mounts, networking, and port forwarding work.

Relay proxy containers#

When a named runner targets a Docker daemon on a different machine, glci deploys a relay proxy container (glci-proxy-*) on the remote host to bridge mock server communication. The relay runs whichever image resolution settled on — normally the registry-qualified :<commit> or :latest tag, which glci pulls on the remote daemon if it isn’t there. If resolution landed on glci:local, nothing can be pulled: that tag exists only on the machine that built it and no registry serves it.

What you see: Jobs on the remote runner fail with connection refused or response is not application/json when trying to reach the mock server.

Possible causes:

• The glci image is missing on the remote host. A registry-qualified tag is pulled automatically; a glci:local has to be copied over: docker save glci:local | docker --context <remote> load. Alternatively build a commit-stamped binary (make install) at a pushed commit so glci resolves a pullable tag in the first place
• The relay container crashed — check docker ps -a on the remote host for exited glci-proxy-* containers
• Protocol mismatch after upgrading glci — old relay containers may use an incompatible muxproto format. Clean up all glci containers on the remote host: docker rm -f $(docker ps -aq --filter name=glci-)

Cleanup: If relay containers are left behind after a crash, remove them on the remote host:

docker --context <remote> rm -f $(docker --context <remote> ps -aq --filter name=glci-proxy-)

Bind mount failures#

What you see: Job containers start but files are missing, or mounts fail with no such file or directory.

Cause: Bind mounts reference paths on the Docker host, not your local machine. When Docker is remote, /Users/you/project doesn’t exist on the remote host.

Fix: glci works around this by uploading your project as a tarball into the mock server container, so standard job execution works. But if you have custom volume mounts in your CI config, they will reference remote paths.

Localhost port forwarding doesn’t work#

What you see: Jobs try to reach 127.0.0.1:<port> for the mock server or registry, but get connection refused.

Cause: 127.0.0.1 inside a container on the remote Docker host refers to that container’s loopback, not your local machine. The mock server runs as a container on the remote host and is reachable by container name on the pipeline network, not via localhost.

Fix: This should work automatically — glci connects mock and runner containers to the same Docker network. If it doesn’t, check glci daemon logs for network errors.

Wrong Docker endpoint#

glci resolves the Docker host at daemon startup with this priority:

1. [docker] host in ~/.glci/config.toml (highest; ignored in a project .glciconfig.toml — see Container-endpoint keys)
2. DOCKER_HOST environment variable
3. Docker context (docker context use / DOCKER_CONTEXT)
4. Default Docker socket

DOCKER_HOST outranks the active context, matching the docker CLI — so an endpoint left over in your shell wins over docker context use. Under Podman the order is [docker] host, then CONTAINER_CONNECTION, then CONTAINER_HOST, then the default connection (Podman itself ranks the connection above the host variable).

If glci reaches an unexpected daemon:

glci daemon status                      # engine endpoint + where it was resolved from
env | grep -E 'DOCKER_HOST|DOCKER_CONTEXT|CONTAINER_HOST|CONTAINER_CONNECTION'
unset DOCKER_HOST && glci daemon stop   # fall back to the context; stop applies it

The glci daemon stop is required: the running daemon keeps the endpoint it resolved at startup, so unsetting the variable alone changes nothing. glci warns when it notices the mismatch — including the case where you removed a selection the daemon still has — but it never restarts the daemon for you, because that would let two shells with different DOCKER_HOST values fight over one daemon.

Set the host explicitly in config to override all auto-detection:

# ~/.glci/config.toml
[docker]
host = "ssh://my-server"

glci config show --network prints the configured [docker] host value only — it does not reflect DOCKER_HOST, the active context, or the engine default. Use glci daemon status or glci doctor for the endpoint actually in use.

Variable resolution issues#

Variables not resolving#

What you see: Job scripts contain literal $MY_VARIABLE instead of its value, or variables are empty.

Cause: The variable isn’t defined at any level, or it’s defined at a lower-precedence level and overridden to empty.

Variable precedence (lowest to highest):

Debugging steps:

# Override a specific variable for testing
glci run --env MY_VARIABLE=test_value

# Use --secrets none to test without remote variables
glci run --secrets none --env MY_VARIABLE=test_value

Dotenv variables not appearing in downstream jobs#

What you see: A producer job creates a dotenv report artifact but the consumer job does not have the expected variables.

Note: needs: { job: producer, artifacts: false } blocks file artifact downloads but dotenv variables still propagate — matching GitLab CI behavior.

.glci.env not loading#

What you see: Variables defined in .glci.env are not available in jobs.

Example .glci.env:

MY_SECRET=s3cr3t
DEPLOY_TOKEN=glpat-xxxx
DB_PASSWORD=hunter2

--secrets none still shows some variables#

What you see: Variables like CI_REGISTRY, CI_PROJECT_PATH, etc. are present even with --secrets none.

Cause: --secrets none only disables fetching remote variables from the GitLab API (project and group variables). CI-derived variables (git info, registry URLs) and YAML-defined variables are always resolved.

A CI_*_SLUG value changed#

What you see: after upgrading, a cache keyed on $CI_COMMIT_REF_SLUG misses once, an artifact path or a review-app hostname built from a slug is spelled differently, or glci variables --all prints a slug that does not match the one it printed before — with no change to the pipeline.

Cause: glci computed slugs with an algorithm that diverged from Gitlab::Utils.slugify in two places. It collapsed runs of separators, which GitLab does not do, and it stripped leading and trailing separators before truncating to 63 characters rather than after, so a leading run cost nothing against the budget. Both are now correct, and five variables are affected: CI_COMMIT_REF_SLUG, CI_PROJECT_PATH_SLUG, CI_PROJECT_NAMESPACE_SLUG, CI_DEFAULT_BRANCH_SLUG and CI_JOB_NAME_SLUG.

A value with no adjacent non-alphanumerics, 63 characters or fewer, is unchanged, so main, master and release/1.x all keep the slug they had.

Fix: nothing — the values now match what GitLab publishes for the same ref or job name, which is what a pipeline written for GitLab expects. A cache or artifact keyed on a slug is looked up under the corrected name from now on, so the first run after upgrading repopulates it. CI_ENVIRONMENT_SLUG is a different algorithm upstream, and the one thing that changed there is the input (see below).

CI_PROJECT_ROOT_NAMESPACE_SLUG was not published at all before and is now, so a pipeline that referenced it was getting the literal $CI_PROJECT_ROOT_NAMESPACE_SLUG and now gets the slug of the top-level group.

CI_ENVIRONMENT_SLUG changed, or a review app deployed under a new name#

What you see: a job whose environment: name: contains a variable deploys to a different environment than before, or $CI_ENVIRONMENT_SLUG inside such a job spells the hostname differently.

Cause: glci derived CI_ENVIRONMENT_NAME and CI_ENVIRONMENT_SLUG from the unexpanded environment: name: literal. GitLab expands the name first and slugs the expanded value, and the slug’s hashed suffix is taken over whichever string it was built from, so the two never agreed for a name containing a variable.

A name with no variable reference in it is unchanged.

Fix: nothing. The slug now names the environment GitLab would deploy to. A local deployment target created under the old name is orphaned and can be removed. environment: url: is still left unexpanded, as GitLab leaves it, so the runner expands it at job time against the corrected CI_ENVIRONMENT_SLUG.

CI_MERGE_REQUEST_DIFF_BASE_SHA is missing, or changes: matches everything#

What you see: in a clone with no local copy of the target branch, rules: if: $CI_MERGE_REQUEST_DIFF_BASE_SHA == null takes the null branch, rules: changes: matches every job, and stderr carries merge-base "main" HEAD failed.

Cause: the target branch is named bare (main), and git clone --single-branch -b feature, a worktree cut from a bare mirror and a CI checkout all leave origin/main with no local main for that name to resolve to. glci now tries main, then origin/main, then refs/remotes/origin/main, in both the variable and the changes: evaluator.

Fix: nothing in a clone that has the remote-tracking ref. A shallow clone that shares no history with the target has no merge base at all, and there the variable is still omitted (GitLab always publishes it with a value, so an empty one is a shape it cannot produce); fetch enough history, or name a target the clone has with --mr-target. The notice is printed once per repository and target rather than on every command.

Child pipeline & include resolution#

Child pipeline include not found#

What you see: the trigger job fails with resolving child includes for "<job>": child pipeline include "child.yml": stat /<repo>/child.yml: no such file or directory, or child pipeline include "..." escapes project directory (trigger includes resolve from the repository root ...), for a file that is clearly there next to the CI file. A path naming no location at all (/, ///) reports not a path in the repository, and one reached through a symbolic link is refused as described in A local include is refused as outside the repository. The path in the message is the one glci actually tried.

Cause: trigger: include: paths resolve from the repository root, which is where GitLab reads them from — not from the directory holding the CI file or the one you ran glci in. A CI file in a subdirectory that names child.yml is asking for <repo-root>/child.yml.

Fix: write the path as GitLab reads it, from the repository root:

# apps/web/.gitlab-ci.yml
child:
  trigger:
    include: apps/web/ci/child.yml   # not ci/child.yml

Leading slashes are stripped and mean the same thing (/apps/web/ci/child.yml). A path that climbs out of the repository is rejected, because GitLab has no such file to read either — so a pipeline that relied on it was already failing upstream.

An include fails with “resolving inputs”#

What you see: a configuration that used to lint clean now fails, naming the include it came from:

CI configuration is invalid: parsing CI config: resolving includes: include local "/ci/comp.yml": resolving inputs: required input "env" not provided

Cause: the included file declares spec: inputs: and the include entry does not supply a value for one of them that has no default. This used to be a warning: glci merged the file uninterpolated and reported the pipeline valid, so every $[[ inputs.env ]] stayed literal. That is worse than failing — a job named through such a placeholder is never matched by an extends: or a needs:, and a job whose image: was the placeholder is created with no image at all and silently runs on whatever it inherits. GitLab refuses the pipeline outright, so failing is what agreeing with it looks like.

Fix: supply the input on the include entry. An included file’s inputs come only from its own entry — --input and --inputs-file reach the root file’s spec: and nothing else, which is also true on GitLab, so there is no flag that works around this.

include:
  - local: /ci/comp.yml
    inputs:
      env: staging

If the input should have had a default, add one to the component — either directly or through rules:.

The same holds for a malformed spec: inputs: in the root file, including a rules: list that declares an unknown key, omits a rule’s default:, or is combined with a top-level default:/options:. That is a hard error on every route now. It used to be reported only by the offline parser, so the same configuration passed silently under GLCI_PREFER_API=1: the document went to the Lint API with its $[[ inputs.X ]] placeholders intact, came back valid, and the pipeline ran with the literal text in place of the values.

A wildcard include merges fewer files than on GitLab#

What you see: include: local: 'ci/**/*.yml' (or a trigger: include: wildcard) resolves to fewer jobs locally than the same pipeline produces on GitLab, with no error — the pipeline just runs a smaller set.

Cause: one of three, in rough order of likelihood.

• The file is gitignored. Wildcards resolve against the project’s non-ignored files, so a generated fragment written into a .gitignored path is not includable. GitLab resolves includes out of the repository and would not see it either — meaning the pipeline is already smaller upstream than the ignored file suggests.
• The file is inside a submodule. Git reports a submodule as a single entry, not as its contents, so nothing under it can be included. GitLab cannot include across a submodule boundary either.
• The pattern used ?, [ or { expecting shell-glob behaviour. Only * is a wildcard in an include path; the rest are literal filename characters, so ci/[ab].yml names one file called exactly that. See wildcards in include paths.

Fix: check what actually got merged:

glci merged

glci merged prints the fully-resolved configuration, so a file you expected is either in there or it never matched. If it is gitignored, commit it or stop ignoring it — that is also what makes it work on GitLab.

Include path must end in .yml or .yaml#

What you see: CI configuration is invalid: parsing CI config: resolving includes: local include "ci/child.txt": path must end in .yml or .yaml, for a file that exists and holds perfectly good YAML.

Cause: GitLab accepts an include location only when its last path segment ends in .yml or .yaml (YAML_ALLOWLIST_EXTENSION, checked before the file is read). A config naming ci/child.txt cannot run on GitLab, so glci rejects it too rather than reporting a green pipeline you cannot push. The rule is case-insensitive (ci/child.YML is fine) but needs at least one character before the dot, so a file literally named .yml is rejected.

Fix: rename the file to .yml/.yaml and update the include. Three cases catch people out:

• A wildcard is expanded before it is checked, so each match is validated individually and the pattern itself never is. local: 'ci/*' fails when ci/ holds a README.md; constrain it to local: 'ci/*.yml' and the same README.md is ignored. The message names both, so you can find the pattern in your config: local include "ci/*" matched "ci/README.md": path must end in .yml or .yaml.
• A remote URL’s query string or fragment is part of the name GitLab checks. remote: https://.../f.yml?ref_type=heads — what the GitLab UI’s Open raw button gives you — is rejected despite ending in .yml to the eye. The message tells you to strip from the ? on, and only says so when stripping actually fixes it.
• Three kinds of include are exempt. component:, whose location is FQDN/project/name@version and never ends in .yml; anything whose include entry carries rules:, because GitLab discards a rules-gated include before it validates it; and any location still holding an unexpanded $VAR, which glci reports through the fetcher’s own error instead.

This covers top-level include: only. A trigger: include: location is checked when the child pipeline is created, so glci lint passes and the trigger job fails — which is how GitLab behaves too.

A local include is refused as outside the repository#

What you see: CI configuration is invalid: parsing CI config: resolving includes: local include "../shared.yml" escapes project directory, or local include "alias.yml": a symbolic link, which GitLab does not follow. Under GLCI_PREFER_API=1 the middle of that wrapper reads resolving local includes: instead. A trigger: include: produces the same two refusals, the first naming the repository root it resolved from. A wildcard names the pattern you wrote as well as the file it matched: local include "ci/*.yml" matched "ci/alias.yml": a symbolic link, which GitLab does not follow, and a link in a directory component names that component: local include "vendor/ci.yml": "vendor" is a symbolic link, which GitLab does not follow.

Cause: an include: local: names a file GitLab could not read, so glci refuses it rather than reporting a green pipeline you cannot push. Two ways that happens:

• The path leaves the repository. ../shared.yml is the obvious case. A leading slash is not a filesystem absolute — GitLab strips it (Gitlab::Utils.remove_leading_slashes), so /etc/passwd is looked up as <repo-root>/etc/passwd and simply is not found.
• The path goes through a symbolic link. GitLab reads includes out of the git object database, where a symlink is a blob holding the target’s path rather than a followed link. It therefore never traverses through one (link/ci.yml answers Local file 'link/ci.yml' does not exist!) and never follows one, reading the link’s own text and reporting Invalid configuration format. glci refuses both, which is why a link is rejected even when its target is inside the repository. glci reads your working tree, so an untracked link is refused the same way a committed one is.

Fix: move the shared file into the repository and reference it directly, or, for a file that genuinely lives in another project, use include: project: — which is the keyword for that and works on GitLab. Replace a symbolic link with the real path.

One exemption, and only to the symlink half. An include entry carrying rules: is one GitLab discards before it examines it, and glci cannot yet tell whether those rules would have matched — so for such an entry a link is followed rather than refused, since failing a config GitLab accepts is the worse error. Leaving the repository is refused for every include regardless, rules: or not, because that is a question about which files may be opened rather than about agreeing with GitLab.

include: project: fails instead of being skipped#

What you see: project include "my/group" file "ci/child.txt": path must end in .yml or .yaml for an include: project:, even with no GitLab token configured. glci used to print skipping project include ...: no GitLab token configured and carry on.

Cause: GitLab validates an include’s location before it looks at its content or at your access to it, so the extension is wrong whether or not the file could have been fetched.

Fix: rename the file. A token is not the problem here — setting GITLAB_TOKEN will not clear this message, and the same config fails on GitLab.

A component version selector with ~ is rejected#

What you see: component include "gitlab.com/org/proj/comp@~2": version selector "~2" is not valid on GitLab; use "@2" instead, on a config glci accepted before.

Cause: glci used to resolve @~N and @~N.M itself. GitLab does not — ~latest is the only version it spells with a tilde, and anything else is matched as a literal ref, so @~2 fails there with content not found. Resolving it locally hid that until push.

Fix: drop the tilde. @2 and @2.3 are the partial-version selectors GitLab recognises, and they mean what @~2 and @~2.3 used to mean here. @~latest is unchanged.

An unknown interpolation is reported for component.reference#

What you see: a parse aborts with unknown interpolation provided: `reference` in `component.reference` , naming the include the file came from.

Cause: $[[ component.reference ]] reads GitLab’s component context, and a file only gets one when two things are true: its spec: header lists reference under component: (the header being the first document of a multi-document file), and it was reached through an include: component: entry. A .gitlab-ci.yml, or a file pulled in by remote:, template: or project:, has no component context at all, and neither does a header that opts into other keys (component: [version]). glci used to substitute regardless, which meant a config GitLab rejects passed locally.

Fix: use the expression only inside a component, add reference to that component’s spec: component: list, and keep it out of the spec: header itself, where GitLab never interpolates it. A component’s nested local: includes inherit its reference and may use the expression too; a nested project:, remote: or template: include cannot. See component includes.

A component version selector resolves locally but fails on GitLab#

What you see: @~latest or @2 resolves under glci, and the same pipeline on GitLab reports content not found for the component.

Cause: GitLab only accepts a selector for a component published to the CI/CD Catalog. glci resolves against the project’s published releases instead, because the catalog version list is not exposed by the REST API — so a project that has releases but was never published as a catalog resource resolves here and is refused there.

Fix: publish the component project to the CI/CD Catalog, or pin an exact version such as @2.3.7, which needs no catalog entry on either side.

CI configuration errors#

A script: line contains ": " and is rejected#

What you see: CI configuration is invalid: jobs:<name>:script config should be a string or a nested array of strings up to 10 levels deep, for a script line that looks perfectly ordinary:

job:
  script:
    - curl -H "Authorization: Bearer $TOKEN" https://example.com
    - echo "Error: deployment failed"

Cause: this is a YAML rule, not a glci one. A colon followed by a space is the block-mapping indicator, and it is not allowed inside a plain (unquoted) scalar — so YAML reads those two entries as mappings rather than strings:

{'curl -H "Authorization': 'Bearer $TOKEN" https://example.com'}
{'echo "Error': 'deployment failed"'}

The double quotes in the line quote nothing, because they are not the first character of the scalar. GitLab refuses the whole file for this, so glci lint reports it too. Earlier versions accepted the file and dropped the entry without a word, so the command never ran and the job still reported success.

Fix: quote the whole entry, with the quote as the first character. Either form works, so pick whichever does not collide with the quoting inside the command:

job:
  script:
    - 'curl -H "Authorization: Bearer $TOKEN" https://example.com'
    - "echo 'Error: deployment failed'"

Two things that are not the problem:

• A colon with no space after it is fine. echo "ratio 1:2" needs no quoting — 1:2 is not a mapping indicator.
• An anchor alias is fine. - *shared_steps expands to a nested list, which GitLab flattens rather than rejecting, up to ten levels deep.

The same rule covers before_script:, after_script:, default:before_script, default:after_script, services:command: and hooks:pre_get_sources_script — GitLab validates all of them with the same Entry::Commands rule, and so does glci. The message names the exact entry, so default:before_script config should be… means the default: block rather than any one job.

This is reported by glci lint, not enforced by glci run. Like every other config diagnostic glci produces, it does not stop a run today: a pipeline started without linting first still executes the job with the line missing, and still passes. On GitLab the same file is refused at pipeline creation and nothing runs at all. Lint a config you have just edited before you run it.

Rule evaluation issues#

The pipeline did not run#

What you see: run, jobs, show or variables fails with

the pipeline did not run: no workflow:rules rule matched under CI_PIPELINE_SOURCE=merge_request_event, CI_MERGE_REQUEST_IID=1 [rule 1 if: $CI_PIPELINE_SOURCE == "schedule" → if: evaluated to false] — GitLab would not have created this pipeline either; simulate another context with --context, or review the workflow:rules configuration

Cause: the project’s workflow: rules: did not accept the simulated context, so there is no pipeline to plan — the same refusal GitLab answers a push with. The default context is merge_request, which sets CI_PIPELINE_SOURCE=merge_request_event and no CI_COMMIT_BRANCH, so a workflow block written for branch pipelines rejects it.

Fix: run with the context the pipeline expects, and check the bracketed rule list in the message to see which condition failed:

glci run --context branch=main
glci run --context tag=v1.0

To keep from typing it every time, name the context in .glciconfig.toml — a [contexts.<name>] entry you pass as --context <name>, or a context = inside [pipelines.<name>] that applies when you run with --pipeline <name>. See workflow rules.

Jobs unexpectedly skipped#

What you see: no jobs to run: requested [job-name] but none matched after rules evaluation or a job you expected is missing from glci show.

Cause: Rules evaluated to when: never in the simulated context. By default, glci simulates a merge_request context.

A job that declares needs: optional: true on a rules-excluded job is no longer one of the reasons — that edge is dropped and the dependent runs, as Optional needs describes.

Debugging steps:

# See which jobs are included in the default context
glci show

# Compare with a different context
glci show --context branch=main
glci show --context tag=v1.0

# Check a specific job
glci jobs   # lists all jobs and their when: status

Common reasons jobs are excluded:

An if: expression that does not parse is no longer one of the reasons — glci rejects that config outright, the way GitLab does, rather than treating the rule as false and dropping the job. glci lint names the job and exits non-zero, and glci run refuses to start. The same check applies to a child or cross-project pipeline’s config, where it fails the trigger job with the reason in that job’s log and spawns no downstream pipeline, as GitLab creates none of its jobs. It also applies when the daemon resumes a pipeline after a crash: if the CI config (or a file it includes) was edited in between into one GitLab rejects, the resume is refused and the pipeline is marked failed, exactly as when the edited file no longer parses. Nothing will run those jobs now, so every job that had not already finished is marked failed with the pipeline and the run gets a finish time: glci history shows it as a completed failure rather than an open one, and glci show <id> no longer lists its jobs as pending. Long conditions written as a YAML block scalar (if: >, if: >-, if: |) parse normally; whitespace between tokens, newlines included, is ignored – except inside a quoted string, which cannot span a newline.

Jobs unexpectedly included#

What you see: A job that should be skipped (e.g., deploy jobs) runs anyway.

Cause: The default merge_request context may match rules that your real CI wouldn’t. Or rules: changes: matches everything because glci has no diff context.

Fix:

# Simulate the exact context you want
glci run --context branch=feature-x

# Skip specific jobs by name
glci run --skip "deploy*"

A rule testing for an unset variable started or stopped matching#

What you see: a job gated on $VAR == null, $VAR != null, $VAR == "" or $VAR != "" appears or disappears, with no change to the pipeline.

Cause: glci used to resolve an undefined variable to the empty string, and to read the bare word null as the null literal wherever it appeared. Both are now correct: an undefined variable is nil, a variable defined as empty is "", and nil equals nothing but nil. Seven comparisons reversed:

The common guard rules: [{ if: '$SOME_VAR != null' }], meaning “this variable was set at all, even to empty”, was inverted for any variable declared with an empty default — glci skipped jobs GitLab runs, and ran jobs GitLab skips, exiting 0 either way.

Fix: nothing, if the rule was written for GitLab’s semantics — it now behaves as it does upstream. If a rule was written against the old behaviour, != "" is the one to re-read: it is not a presence test. Use the bare $VAR for “set to something non-empty”, != null for “defined at all”.

glci variables <job> prints each condition with its operands expanded. It renders an unset variable and one defined empty identically, as (), so it will not by itself tell those two apart — check whether the name has a row in the variables table below the trace. That distinction now decides ==, != and bare truthiness as well as =~.

A rule using =~ started or stopped matching#

What you see: after upgrading, a job guarded by =~ or !~ is included where it used to be skipped, or skipped where it used to run — with no change to the pipeline.

Cause: the right-hand side of =~ is a regular expression only when it is written as one, /pattern/. Anything else is a substring test in which the right-hand side is the text being searched, and an unset right-hand side never matches at all. glci used to compile every right-hand side as a regex and match it against the left, so comparisons that did not use /pattern/ evaluated backwards, and an unset one became the empty pattern and matched everything.

Fix: decide which of the two the rule meant.

# A pattern: write it as one.
- if: '$CI_COMMIT_BRANCH =~ /^(main|release)$/'

# A membership test: the right-hand side is the haystack, which is what
# GitLab does — "main" occurs inside "main,release".
- if: '$CI_COMMIT_BRANCH =~ $BRANCHES'

# A presence test: the bare variable, for "set to something non-empty".
# Not `!= ""` — an undefined variable is nil, and nil != "" is true.
- if: '$DEPLOY_PATTERN'

A variable that is defined but empty is not the same as one that was never set: the first is searched and matches only an empty left-hand side, the second never matches. See the right-hand side of a match.

glci variables <job> prints each condition with its operands expanded, which shows which side held what. It renders an unset variable and one defined empty identically, as (), so it will not by itself tell those two apart — check whether the name has a row in the variables table below the trace.

A rule with a backslash in it is now invalid, or quietly matches differently#

What you see: after upgrading, a pipeline that linted clean is refused with jobs:<name>:rules:rule if invalid expression syntax – or it still lints clean, and a rule whose literal contains a backslash now matches the other way. The sharpest form of the second case is a when: never guard that stops firing, so a job it used to block starts running.

Cause: a quoted string in an if: expression is taken verbatim. There are no escape sequences, so a backslash is ordinary data. glci used to decode escapes; GitLab never has, so the two disagreed about what the literal even was, and this brings glci in line. Four shapes come out of it:

• $FOO == "a\"b" – the literal ends at the escaped quote and b" is left over, which is refused.
• $A == "x\" && $B == "y" – the literal ends early too, but the leftover is a valid continuation, so it lints clean and quietly reads as $A == "x\" and $B == "y".
• $WORKDIR == "C:\\temp" – the literal is now the eight characters C:\\temp rather than the decoded C:\temp, so a guard written this way no longer matches. If it was a when: never rule, the job runs.
• $A == "x\" == "y" – the leftover begins with a comparison operator, and this is the one shape where glci is stricter than GitLab. GitLab lexes $A, ==, x\, ==, y and its parser builds the left-associative ($A == 'x\') == 'y', which is valid configuration whose chained rule is constant (always false for == and =~, always true for != and !~). glci does not chain comparisons, so it refuses a file GitLab accepts, with the same ... if invalid expression syntax as every other refusal (the parser’s unexpected trailing input: "== \"y\"" appears in a rule trace, not in the lint verdict). A leftover beginning !=, =~ or !~ is refused the same way. Rewrite the rule rather than relying on either reading.

A literal that spans a newline is refused outright; that has its own entry directly below.

In the second and third cases the rule was already behaving this way on GitLab – a guard comparing against "C:\\temp" never fired there either, so the job was always running in real CI and only glci was blocking it. Check what the rule was meant to compare before assuming the old local result was the correct one.

Fix: stop escaping quotes and use the other quoting style instead; write a backslash as itself.

# Refused: the literal ends at \" and b" is left over.
- if: '$FOO == "a\"b"'

# Write the other quote around it.
- if: "$FOO == 'a\"b'"

# \n is two characters, not a newline, so this matches SEP: 'a\nb'.
# Keep the variable's own YAML single-quoted so YAML does not decode it either.
- if: '$SEP == "a\nb"'

# A backslash stands for itself: match WORKDIR: 'C:\temp' as it really
# is, rather than doubling it.
- if: '$WORKDIR == "C:\temp"'

A literal needing both kinds of quote cannot be written at all – that is true on GitLab too.

glci variables <job> prints each condition with its variables expanded, which tells you what the operands held. It renders the expression as you wrote it, so it will not show you where the literal actually ended – count from the opening quote to the next quote of the same kind, ignoring any backslash in between, and that is the string being compared.

Only glci lint and glci run refuse an invalid config. glci show, glci jobs and glci variables plan around it instead: the job disappears from the listing with a warning on stderr and the command still exits 0. So if a job goes missing after an upgrade, run glci lint to get the verdict rather than inferring one from an empty table.

A string literal in a rule spans a newline#

What you see: glci lint fails with jobs:<name>:rules:rule if invalid expression syntax and glci run refuses to start, on a pipeline that ran locally before the upgrade. The rule itself looks ordinary – the newline is inside a quoted string that was written across two lines of a YAML block scalar, which is easy to miss.

Cause: Lexeme::String’s pattern is built from ., and Ruby’s . does not match a newline, so a literal spanning one matches no lexeme at all and GitLab rejects the configuration. glci used to evaluate it instead, so the same file ran here and was refused there. This is the shape most likely to be in an existing pipeline already: the backslash shapes above need somebody to have deliberately written \" or \\, while a literal newline only needs a long condition broken across lines in a | block.

# Refused: the literal opens on one line and closes on the next, so the
# string itself contains the newline.
release:
  script: ./deploy.sh
  rules:
    - if: |
        $CI_COMMIT_MESSAGE == "release
        candidate"

Fix: keep each literal on one line. Whitespace between tokens, newlines included, is still ignored, so a long condition can be broken anywhere outside the quotes:

    - if: |
        $CI_COMMIT_MESSAGE == "release candidate" &&
        $CI_COMMIT_BRANCH == "main"

A folded scalar (if: >, if: >-) folds a single newline into a space rather than keeping it, so a literal broken across lines there is accepted and compares against release candidate with a space in it – not a syntax error, but not what was written either. A blank line inside a folded scalar does keep a newline, and is refused like the | case. A carriage return is ordinary data: Ruby’s . matches it, so it stays inside the string.

Context simulation not matching real CI#

What you see: Jobs appear in glci show that don’t appear in the GitLab pipeline (or vice versa).

Cause: glci evaluates rules locally with the simulated context. Some differences from real CI:

• $CI_PIPELINE_SOURCE is set to push (for branch/tag) or merge_request_event (for merge_request context) — your real pipeline may have a different source
• rules: changes: may behave differently because glci uses local git diff while GitLab compares against the target branch
• rules: exists: evaluates your local working tree minus gitignored files (tracked + untracked), while GitLab evaluates the committed tree at the pipeline SHA — so uncommitted/untracked files satisfy exists: locally but won’t on GitLab until committed
• Protected/unprotected branch distinctions are not enforced locally — CI_COMMIT_REF_PROTECTED and the two CI_MERGE_REQUEST_*_BRANCH_PROTECTED variables are always false

Fix: Use --context and --env together to match your real CI environment as closely as possible:

glci show --context merge_request --mr-source feature --mr-target main

Performance tips#

Slow image pulls#

The embedded registry acts as a pull-through cache – the first pull is slow but subsequent pulls are instant.

Slow startup due to secrets fetch#

Remote variable fetching adds latency to pipeline startup. If you don’t need remote secrets:

glci run --secrets none      # skip all remote variable fetching
glci run --secrets project   # skip group variables (slower due to pagination)

Reducing disk usage#

glci system df               # check what glci is using
glci system cache clean      # wipe CI cache
glci system prune            # clean unused containers, networks, volumes
glci system prune --all      # also remove registry data and history

Known limitations#

• CI_JOB_TOKEN is forwarded from the host’s GitLab token — permissions may differ from production. Use --no-token to disable.
• include: project: and include: component: require a GitLab API token (other include types work offline). Component version selectors resolve against the project’s published releases: @~latest takes the newest, and a partial version (@2, @2.3) the newest match, never a pre-release; one matching no release falls back to a tag or branch of that name. @~N and @~N.M are rejected, as on GitLab. GitLab selects from CI/CD Catalog versions rather than releases, so a selector against a project that has releases but is not a catalog resource resolves here and is refused there.
• Local include: paths resolve from the repository root (nearest .git above the CI file), matching GitLab — not from the CI file’s directory. Outside a git repository, resolution falls back to the CI file’s own directory, so a subdirectory CI file referencing repo-root paths only works inside a git checkout. That root is also the containment boundary: a path leaving it, or one reached through a symbolic link, is refused rather than read. Paths are normalised before resolution, so ci/../shared.yml and ci//shared.yml resolve here where GitLab looks up the literal blob path and finds neither.
• Protected variables are fetched regardless of branch protection status.
• Kubernetes executor is not supported — only Docker.
• Git submodules — served from the mock for relative URLs and for absolute HTTP(S) URLs on the project’s own GitLab host (SSH URLs only under GIT_SUBMODULE_FORCE_HTTPS); anything on another host is left to the network and needs credentials in the container. Objects come from your checkout, a --project-dir/[projects] mapping, or a clone from GitLab limited to the superproject’s own namespace or below. A submodule glci cannot reach is a daemon: warning: in glci daemon logs and the job’s checkout fails. Child and cross-project pipelines are served too, each against its own project path, and a project path is served once per pipeline prefix — the first walk wins. Up to 10 levels deep, 64 repositories per walk. See Git submodules.
• retry: does not cover job timeouts — glci only re-runs a job after the runner reported a result, so retry: when: job_execution_timeout and retry: when: runner_system_failure never fire locally. A runner that picks a job up and then goes silent does trip the mock server’s stuck timer, so retry: when: stuck_or_timeout_failure works. allow_failure is honoured for all of these. See Tolerated failures.
• Child pipelines — trigger: include: supports up to 2 levels of nesting (matching GitLab). A child’s own config resolves include: (including project:/component:) against the same GitLab instance and token as the parent, and from the repository root, whatever directory glci was run from.
• Merge request variables describe a merge request that does not exist, so CI_MERGE_REQUEST_IID, _ID, _DIFF_ID, _TITLE and the _DRAFT/_SQUASH_ON_MERGE/_PROTECTED flags are fixed values rather than a real merge request’s. CI_MERGE_REQUEST_LABELS, _MILESTONE, _ASSIGNEES and _APPROVED are never published at all, because GitLab defines them only once the merge request carries a label, a milestone, an assignee or an approval — a rule that branches on them takes the unset path locally. CI_MERGE_REQUEST_EVENT_TYPE is always detached: merged-result and merge-train pipelines run on a commit GitLab creates on the fly and have no local equivalent. CI_MERGE_REQUEST_DIFF_BASE_SHA and CI_MERGE_REQUEST_REF_PATH name a commit and a ref in your own repository, which the single synthetic commit glci serves does not carry, so git commands against either fail inside a job. See Context simulation.
• CI_DEFAULT_BRANCH comes from [defaults] default_branch, then refs/remotes/origin/HEAD, then main — so on a repository whose default branch is master and whose clone never recorded it, $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH rules diverge from GitLab. Set default_branch = "master" in the project’s .glciconfig.toml to fix it for everyone who clones the repository, or run git remote set-head origin -a to fix it for this clone. If the name you typed is not a usable branch name, glci says so on stderr and keeps the next answer in the order, so check that line before assuming the key took effect. The served repository also carries the default branch as a second name for the pipeline’s commit, so git diff origin/<default>...HEAD in a job is empty rather than a real diff. See Context simulation.
• Dynamic child pipelines — trigger:include:artifact runs a child pipeline whose YAML a generator job produced. The generator must run before the trigger job (earlier stage or needs:); otherwise the trigger fails with artifact include "…": job "…" produced no artifacts. Each artifact-sourced config file is capped at 5 MiB and an empty file is rejected.
• Cross-project triggers — trigger: project: resolves from local directories or clones from GitLab. A cloned target has only its .yml/.yaml files extracted (the only thing an include: can name), so its rules: exists: sees no other file and its config must live at .gitlab-ci.yml; map the project locally when either matters. More than 64 MiB of YAML is refused.
• parallel: matrix: declaration order is read from the YAML, so shard names and CI_NODE_INDEX match GitLab. Two shapes carry no order for glci to read and fall back to alphabetical variable order: !reference to a path that is not a job’s parallel: matrix: (matrix: !reference [.matrices, deploy], where the target is a bare list — !reference [.template, parallel, matrix] keeps its order), and a matrix set through parallel in .glciconfig.toml, where TOML tables are unordered. Inline, extends:, include:, YAML anchors and merge keys (<<:) all keep their declared order. See Parallel job variables.
• A merge key (<<:) written after a key it also defines keeps the locally written value in glci, which is what the YAML specification prescribes and what the Go YAML parser implements. GitLab’s parser (Psych) applies the merge at the position of the <<, so there the merged value wins instead. Affects any key, not just parallel: matrix:; write <<: first — as nearly every config does — and both agree.
• Some parallel: shapes GitLab rejects are accepted locally: parallel: 2.5 (truncated to 2 shards), parallel: "3", parallel: {}, and an unknown key beside matrix: are ignored rather than reported. The diagnostics glci does mirror are the count range (1-200), the 200-job matrix cap, a matrix that is not a list of mappings, an entry with no variables, and a value that is not a string or an integer.
• rules: if: comparisons do not chain. GitLab’s parser reads ==, !=, =~ and !~ left-associatively, so $A == "x" == "y" compares the first comparison’s true/false against "y"; glci refuses it, reporting the leftover as trailing input. A string literal reaches that shape on its own, because \" does not escape a quote: $A == "x\" == "y" closes the literal at that quote and leaves == "y" over. GitLab accepts such a file and glci rejects it, so glci lint exits non-zero and glci run refuses to start. See operators and operands and String literals.
• Crash recovery requires both mock server and runner containers to survive. Secrets are held only in daemon memory (mlocked, never written to disk) and are lost on daemon restart, so resumed pipelines always re-fetch remote variables from the GitLab API. Remote include: files are the exception: those come from ~/.glci/cache/includes on resume rather than being re-fetched.