How It Works

glci sits between your .gitlab-ci.yml and the official gitlab-runner. Instead of talking to a real GitLab instance, the runner talks to a mock GitLab API server running as a daemon-scoped container. Jobs execute in real containers with the same shell scripts, image pulls, and lifecycle that GitLab CI uses in production — no simulation, no approximation.

glci drives a container engine by shelling out to its CLI (docker or podman). Both Docker and Podman are supported; the engine is selected via [docker] engine in config (per-runner override with [runners.<name>] engine). When engine is unset (omitted), glci uses Docker, or auto-falls back to Podman if Docker is unreachable and Podman is — printing a one-time notice to stderr; set engine = "podman" to silence it. An explicit engine = "docker" opts out of that probing and never falls back. The choice of engine is orthogonal to the gitlab-runner executor: the generated runner config always uses executor = "docker" even under Podman, because gitlab-runner has no podman executor — glci simply points it at Podman’s Docker-compatible API socket. See Container engine for the engine setting.

Component overview#

Data flow#

The mock server is the central hub — the only component both the daemon and the runner communicate with. Jobs push/pull artifacts, cache, and container images directly to/from the mock server via extra_hosts (host-gateway) with FF_NETWORK_PER_BUILD. When runners target a Docker daemon on a different machine, a relay proxy bridges the gap (shown in dashed lines).

Infrastructure#

Daemon-managed (shared across pipelines)#

• glci-mock — mock server container (API + embedded OCI registry on port 39741 by default, configurable via mock_server_port). Started once by the daemon; jobs reach it via extra_hosts (host-gateway) with FF_NETWORK_PER_BUILD.

• Runner containers — persistent gitlab/gitlab-runner containers managed by RunnerManager. Three tiers:

  • Shared runner (glci-runner-{suffix}) — handles projects with default config; jobs without matching named-runner tags go here.
  • Per-project runner (glci-runner-{suffix}-{project}) — created when [runner] config_template is set for the project, in either config file. Isolated from the shared runner.
  • Named runner (glci-runner-{suffix}-{project}-{name}) — one per [runners.<name>] config entry. Jobs whose tags match the runner name are routed here via run_untagged=false.

  Each runner container polls the mock server for jobs via tag-based dispatch. Config is written to a Docker volume and reloaded via SIGHUP. When a config_template produces multiple [[runners]] blocks, glci derives sub-tokens and registers each runner separately. Named runners can target different Docker daemons via docker_host (endpoint URL) or docker_context (Docker context name). Remote runners get their own bus network on the remote daemon and reach the mock server through a local TCP proxy with host-gateway extra_hosts.

All containers are long-lived. They start on first use and stop when the daemon shuts down. If a container is missing (manually removed, Docker restart), the daemon recreates it on demand. Per-project and named runners are created lazily when a pipeline first needs them.

Per-pipeline#

• glci-net-{id} — Docker bridge network. The shared mock and runner containers are connected to it for the pipeline’s lifetime.
• DinD jobs are serialized via the _glci_dind resource group to prevent concurrent DinD jobs from conflicting on the docker service alias.
• runner-* — Transient job execution containers, spawned by the runner’s Docker executor. One per job, destroyed after the job finishes.

Pipeline execution lifecycle#

This is the sequence of events when you run glci run [jobs...]:

Detailed steps#

1. CLI sends StartPipeline to daemon via Unix socket (daemon auto-starts if not running). If another pipeline is already being prepared for the same directory, the request is queued instead of rejected — the daemon holds the HTTP connection open and starts the queued pipeline automatically when the current preparation finishes. Cancelling the request (Ctrl+C) dequeues it. A maximum queue depth of 3 prevents resource exhaustion.
2. Daemon parses .gitlab-ci.yml — offline parser resolves includes, extends, !reference tags; evaluates workflow/job rules; expands parallel: jobs into shards; builds execution plan (stages, DAG, auto-include upstream deps)
3. Daemon creates per-pipeline Docker network (glci-net-{id})
4. Daemon prepares source code — creates a bare git repo from the working directory (including uncommitted/untracked files in dirty mode), tar.gz for upload
5. Daemon ensures mock container is running — connects to pipeline network, registers per-pipeline state (secret), configures S3 cache credentials, uploads git repo. When a job asks for submodules, one additional bare repo per submodule is prepared and served alongside it, each pinned to the commit the superproject records
6. Daemon ensures runner container is running — connects to pipeline network, Docker socket or DOCKER_HOST for remote Docker
7. Scheduler dispatches jobs — concurrent within stages, DAG-aware, respects resource_group exclusions. Per job: queue on mock server with HMAC token, persistent runner polls for job, clones source, downloads artifacts, executes scripts, uploads artifacts, streams trace, reports completion
8. Trigger jobs spawn child pipelines sharing the same mock and runner containers (max depth: 2). The child’s config is parsed with the parent’s GitLab credentials and include cache, so its own include: project:/component: resolve the same way the top-level config’s do. A child gets its own submodule walk when it asks for more than the pipeline that spawned it served, and a cross-project target’s submodules are read from that project’s own commit
9. Daemon persists artifacts to ~/.glci/projects/{key}/pipelines/{id}/
10. EventBus streams events to all connected CLI clients

Mock server API#

The mock server (pkg/mockserver/) is the central hub. It implements the subset of the GitLab API that gitlab-runner needs, plus infrastructure services and an internal control API consumed by the daemon.

GitLab runner API#

These endpoints are consumed by the runner, which thinks it is talking to a real GitLab instance:

Infrastructure services#

Internal API (daemon to mock server)#

These endpoints are consumed by the daemon to manage pipelines. They are not authenticated via the runner protocol — they go through the docker exec transport:

Docker exec transport#

The daemon communicates with the mock server container exclusively through docker exec, which enables remote Docker daemon support (TCP via DOCKER_HOST, Docker contexts, SSH tunnels, cloud Docker).

Three transport implementations exist in pkg/daemon/transport.go:

The persistentExecTransport keeps a single subprocess alive, multiplexing all requests over shared stdin/stdout with the muxproto binary protocol. This avoids the overhead of spawning a new docker exec for every API call.

Relay proxy for cross-machine Docker daemons#

When a named runner targets a Docker daemon on a different machine (docker_host or docker_context pointing to a non-localhost address), job containers on the remote host can’t reach the mock server via host-gateway — that routes to the remote machine’s own host, not back to where the mock lives. glci solves this by deploying a relay proxy container on the remote daemon that bridges HTTP requests back through docker exec.

How it works:

1. The daemon starts a glci-proxy-* container on the remote daemon using the resolved glci image — normally the registry-qualified tag, since glci:local only exists on the machine that built it — connected to the runner’s bus network with the glci-mock network alias
2. Inside the container, docker exec -i runs glci internal relay-proxy --port 39741, which starts an HTTP server
3. Runner job containers reach glci-mock:39741 via Docker DNS on the bus network — this resolves to the relay container
4. The relay serializes each HTTP request into a muxproto binary frame and writes it to stdout
5. On the daemon machine, mockRelayLoop reads frames from the docker exec stdout pipe, forwards them as HTTP to the local mock server, and writes muxproto responses back to stdin
6. The relay reads the response frame from stdin and returns it as an HTTP response to the job container

All HTTP headers (including Job-Token for authentication) are forwarded through the relay chain. The relay serializes concurrent requests through a mutex since the muxproto pipe is single-threaded. No direct TCP connectivity is needed from the remote machine back to the daemon host.

Muxproto binary protocol#

pkg/muxproto/ implements a length-prefixed binary protocol for multiplexing HTTP-like request/response pairs over a byte stream:

Request:  int64(methodLen) method int64(pathLen) path int64(ctLen) contentType int32(headerCount) [int64(keyLen) key int64(valLen) val]... int64(bodyLen) body
Response: int32(statusCode) int64(ctLen) contentType int32(headerCount) [int64(keyLen) key int64(valLen) val]... int64(bodyLen) body

All integers are big-endian. Max frame size is 1 GB; a body that would exceed it is rejected rather than truncated, so a partial payload never reaches the peer looking complete. Payloads that legitimately outgrow a frame — artifact archives, in particular — are transferred in 32 MiB chunks over /internal/artifact/{jobID} using offset/limit query parameters. The transport auto-restarts on stream corruption.

Event streaming#

The mock server emits events on /internal/events as a hybrid binary/JSON stream:

• Control events (job_result, job_dispatched) are JSON lines
• Trace data uses a binary frame format to avoid JSON overhead for high-volume output: \x00 | uint64(pipelineID) | uint64(jobID) | uint32(offset) | uint32(dataLen) | data

Both event types flow on a single FIFO channel per subscriber (buffer 4096), so a job_result never overtakes an earlier trace frame. Trace drops are tolerated and reconciled via GET /internal/trace. Control drops fall back to /internal/status polling. The daemon also polls /internal/status every 5 seconds as a safety net for results missed during event stream reconnection.

Each job’s trace writer is guarded by its own lock rather than the client-wide event lock, so one job’s slow writer no longer holds up the daemon’s SetJobTraceWriter / RemoveTraceWriter / FlushTrace calls for other jobs. It does not isolate the jobs themselves: the event stream is read and dispatched on a single goroutine, so a writer blocked on disk stalls frame delivery for every job until it returns, whichever lock guards it. Two things depend on the per-job lock. The reconciliation reads its position after the fetch instead of carrying a copy across it, so frames the stream delivers while the request is in flight aren’t written to the trace file a second time. And unregistering a writer waits for any write already in flight, so the executor’s close cannot race the event-stream goroutine — a write landing on a closed trace file would still be published to the event bus, reaching live viewers but never disk.

The daemon tracks that position using each frame’s absolute offset, in the same coordinate space GET /internal/trace serves, rather than by summing the lengths of frames it received. A sum cannot express a hole, and the bus drops whichever frame is publishing when a buffer fills — not necessarily the last one. Reconciliation therefore resumes from the first byte the writer is missing: the gap gets filled, and the bytes after it are written twice, the same trade the trace handover makes.

EventBus#

The daemon’s EventBus (pkg/daemon/eventbus.go) manages in-memory event storage and streaming for CLI consumers:

• Two-tier storage: Status events (stage_start, job_status, pipeline_done) are kept permanently. Log events (job_log) are bounded by byte size (default 10 MB) with slice compaction.
• SubscribeWithReplay: Atomically returns replay history and registers a subscriber under a write lock, preventing event loss between replay and subscription.
• Unbounded subscriber queues: Each subscriber gets an eventQueue backed by a mutex-protected slice and a drain goroutine. Publish never blocks and never drops events. The drain goroutine feeds a 64-element output channel the consumer reads from.

Trace handover: disk before bus#

A client attaching to a running job (GET /api/logs/{pipelineID}/{jobName}) subscribes, sends the job’s trace file as a snapshot, and then streams live events, discarding those stamped strictly before a cutoff so the snapshot isn’t repeated wholesale (an event stamped exactly at the cutoff is kept, since a coarse clock can collapse the two readings and a duplicate beats a drop). Two rules keep that handover lossless:

• history.TeeWriter writes each trace line to the file before publishing it to the bus (the file is deliberately unbuffered, since the reader snapshots it with os.ReadFile).
• The handler stamps the cutoff before reading the file, not after sending it.

Together these guarantee that a line missing from the snapshot is stamped after the cutoff and therefore survives the filter. The residual race runs the other way — lines appended while the file is being read are in the snapshot and pass the filter, so they are displayed twice — which is the intended trade. When the pipeline finishes while the snapshot is in flight, the handler re-reads the trace and emits the tail before returning on the done check.

Scheduler#

The scheduler (pkg/scheduler/) provides DAG-aware concurrent job dispatch with resource_group support. It dispatches each job as soon as its dependencies are satisfied; how many run in parallel is governed by the gitlab-runner concurrent setting, not the scheduler.

Scheduling modes#

Job lifecycle states#

Jobs move through these states: pending -> running -> done (or skipped). Manual jobs (when: manual) start in a separate manual state and move to pending only when triggered.

Skip logic#

• on_success jobs are skipped when the pipeline has failed and their dependencies are complete
• on_failure jobs are skipped when the pipeline has not failed and no running jobs remain that could cause a failure
• Cascading skips: if a job’s dependency was skipped, the job itself is skipped (DAG mode only)
• resource_group: only one job per resource group runs at a time

Authentication#

Pipeline-level security without a real GitLab instance:

1. Pipeline secret: 32 bytes from crypto/rand, generated per pipeline
2. Job tokens: HMAC-SHA256(secret, jobID) — each job gets a unique token
3. Runner dispatch: job routed to the runner presenting the matching token (prevents FIFO dispatch races when multiple jobs are queued)
4. Git auth: HTTP Basic with job token
5. Registry auth: Docker-standard token auth
6. Cache (S3): AWS Signature V4 with daemon-secret-derived credentials (stable across pipeline runs)

Crash recovery#

The daemon performs recovery on startup (pkg/daemon/recovery.go):

1. Find recoverable pipelines: Scan ~/.glci/active-workdirs/ markers, read execution state from ~/.glci/projects/{key}/pipelines/{id}/execution.json, check if both mock and runner containers are still alive via docker inspect.
2. Preserve resources: Build sets of container names, volume names, and network names belonging to recoverable pipelines.
3. Clean orphaned resources: Remove any glci-* containers, volumes, and networks not in the preserve sets. Also clean stale temp files.
4. Finalize stuck pipelines: Mark pipelines without surviving containers as failed in history.
5. Resume: Reconnect to containers via docker exec, query ground-truth state via GET /internal/status, reconcile with persisted state, and re-dispatch pending jobs.

Execution state is written atomically (temp file + rename) on every job state transition. It contains container names, job states, runner tokens, pipeline secret, and the original StartPipelineRequest for re-parsing CI config during resume.

CI config parsing#

glci includes a full offline .gitlab-ci.yml parser as the primary parsing path (pkg/config/). It implements GitLab’s processing pipeline in Go:

1. Load YAML with yaml.Node-based parser preserving !reference tags
2. Interpolate inputs ($[[ inputs.key ]]) before and after include resolution
3. Resolve all includes — local: (disk), remote: (HTTP), template: (GitLab CDN), project: (API), component: (API). Component version selectors (@~latest, @~N, @~N.M) are resolved to the latest matching semver tag via the GitLab Tags API before fetching. Includes are deduplicated within a parse and cached on disk under ~/.glci/cache/includes across parses — for an hour, or indefinitely when the ref: names a version tag. Every parse route shares that cache and the same GitLab credentials: the CLI (glci lint/jobs), the daemon’s pipeline prepare, crash-recovery re-prepare, and the parse of a child pipeline or cross-project trigger target. Entries are keyed by GitLab instance and by a fingerprint of the token they were fetched with, so a project pointed at a self-managed host never reads another instance’s copy, and a template pulled with one credential is never served to a parse holding a different one (or none).
4. Resolve extends: — recursive deep merge with circular/depth detection (10 levels max)
5. Resolve !reference tags — path lookup with cycle detection, sequence flattening
6. Inject edge stages (.pre/.post)
7. Apply default: inheritance with inherit: control (true/false/[keys])
8. Parse workflow: rules with auto_cancel: support
9. Parse all job fields into enriched types (trigger, release, retry, environment, hooks, id_tokens, etc.)
10. Expand parallel: jobs — parallel: N into job 1/N … job N/N, parallel: matrix: into one job per permutation. Each shard records its 1-based index and the shard total, which the runner emits as CI_NODE_INDEX and CI_NODE_TOTAL

Also handles: rules: changes: (git diff matching), rules: exists: (non-ignored file glob: tracked + untracked, minus .gitignore), YAML anchors/aliases, multi-document YAML.

API-assisted mode (optional)#

Set GLCI_PREFER_API=1 to use the GitLab CI Lint API for YAML resolution. This requires a GitLab token but delegates include:, extends:, and !reference resolution to the server. The offline parser is the default because it requires no network access and handles the vast majority of CI configurations.

API-assisted mode does not apply to downstream pipelines. A child pipeline’s config is a synthetic root glci merges from the trigger’s include files, and a cross-project target’s config is a tree of .yml/.yaml files extracted from its bare repo into a private temporary directory (the whole tree, so the target’s own include: local: entries resolve from its repository root) — neither is a file the server can see, so linting it would have GitLab resolve the include entries itself against the committed copies in the repository and hand back a different pipeline. Child and cross-project configs are therefore always parsed offline, GLCI_PREFER_API notwithstanding. Their include: project:/component: entries are still fetched from the API as usual.

API Lint response — field inventory#

When using API-assisted mode, glci calls POST /projects/:id/ci/lint with include_jobs: true, dry_run: false. The API returns merged_yaml (fully resolved YAML) and a jobs array. These fields are available from each source:

Daemon API#

The daemon runs as a background process and exposes an HTTP/JSON API over a Unix socket (~/.glci/daemon.sock). The CLI is the primary consumer.

The daemon auto-starts on the first glci run and auto-stops after 30 minutes of idle. A version guard auto-restarts it if the CLI/daemon build versions mismatch. It also resolves its container-engine endpoint once at startup and reports it via /api/health; a command whose shell selects a different endpoint gets a warning rather than a restart, since the daemon is shared across shells and projects.