GitLab Runner

You’ll run GitLab CI jobs inside a cluster environment. The ready-made app is the runner manager; each pipeline job starts its own short-lived CI job instance.

Choose this when

• Projects on GitLab.com or a self-hosted GitLab need Kubernetes runners
• Next.js, Node, or Docker image builds should run next to the apps they deploy
• You want one place in Xooshe to size concurrency and temporary disk so jobs are not killed mid-build

What you get

Two layers, sized separately:

Layer	What it is	What sizes it
Runner manager	Long-running instance that polls GitLab and starts jobs	Resources step (Tiny / Starter / …) and copies
CI job instances	One instance per GitLab job (build + helper containers)	Custom configuration → concurrent and job

The Resources step does not set disk for CI jobs. A Next.js npm ci / next build writes several Gi to temporary disk on the node (Kubernetes ephemeral-storage). If that request stays 0, the node can evict the job when disk is low.

Before you start

• A GitLab project or group runner token (glrt-… or a registration token) and the GitLab URL
• Permission to create an app (roles)
• Enough node disk for (concurrent × job temporary-disk request) plus about 10% free for the kubelet

Create the runner in GitLab (Settings → CI/CD → Runners) and copy the token before you deploy.

Deploy a runner

1. Select the cluster → New App → ready-made GitLab Runner.
2. Pick the environment (often a shared ci or ops environment).
3. Size the manager with Starter (250m CPU, 512Mi memory) and 1 copy unless many projects share one manager.
4. After create, open the app → Custom configuration.
5. Set gitlabUrl and runnerRegistrationToken, then set concurrent and job from the tables below.
6. Save so Helm upgrades the runner.

Keep the token in Custom configuration, not in git.

Size the runner

Peak load is copies × concurrent jobs at once. Raising copies doubles managers; it does not give each job more disk. For slow queues, raise concurrent only after node disk can hold the extra jobs.

Manager (Resources step)

The manager only talks to GitLab. It does not compile your app.

State	Resource card	CPU	Memory	Copies
Idle / few projects	Tiny	80m	128Mi	1
Typical team CI	Starter (recommended)	250m	512Mi	1
Many projects, high poll load	Standard	500m	1Gi	1–2

CI jobs (Custom configuration)

Match one GitLab job, then set concurrent so several of those jobs can land on the same node without filling disk.

State	concurrent	Job CPU (request / limit)	Job memory (request / limit)	Temporary disk (request / limit)
Lint, unit tests, small images	4	250m / 1	512Mi / 1Gi	1Gi / 4Gi
Next.js / Node npm ci + next build	2–4	500m / 2	1Gi / 4Gi	4Gi / 10Gi
Docker / Kaniko / nx image builds, large caches	1–2	500m / 2	1Gi / 4Gi	8Gi / 20Gi

Xooshe defaults new runners to the Kaniko / nx row (concurrent: 4, 8Gi request / 20Gi limit). A typical next build in a Node image uses about 3.5Gi. Kaniko image builds are larger: extracting a cached RUN layer and then snapshotting node_modules can briefly keep two copies on disk and overflow a 10Gi limit.

Helper container (GitLab’s helper sidecar) stays small: about 50m CPU, 64–256Mi memory, 100–512Mi temporary disk.

Capacity check

On each node, keep roughly:

concurrent × ephemeralStorageRequest ≤ usable temporary disk

Usable disk is what remains after other apps and the kubelet’s free-space reserve (often ~10% of the node disk). Example: two Kaniko jobs at 8Gi each need ~16Gi free on one node if they all schedule there. Requests make the scheduler spread jobs; a request of 0 does not. For image builds, keep concurrent at 1–2 even if the JSON still says 4.

Use GitLab tags (next, heavy) so disk-heavy pipelines pin to a runner sized for them, and keep a second runner on Tiny + 1Gi disk for lint jobs.

Custom configuration

Key	Purpose
gitlabUrl	GitLab instance (https://gitlab.com or your server)
runnerRegistrationToken	Runner token from GitLab
concurrent	Max jobs this manager runs at once
checkInterval	Seconds between polls (default 3)
tags	GitLab runner tags, comma-separated
job.cpuRequest / job.cpuLimit	CPU for the build container
job.memoryRequest / job.memoryLimit	Memory for the build container
job.ephemeralStorageRequest / job.ephemeralStorageLimit	Temporary disk for the build container
job.ephemeralStorageRequestOverwriteMax / job.ephemeralStorageLimitOverwriteMax	Max a single GitLab job may set with KUBERNETES_EPHEMERAL_STORAGE_REQUEST / _LIMIT (defaults 20Gi / 40Gi)
job.helper*	Same fields for the GitLab helper container
runnersConfig	Full runner TOML override (advanced; skips generated job settings)

Example for Kaniko / nx image builds:

{
  "gitlabUrl": "https://gitlab.com",
  "runnerRegistrationToken": "glrt-xxxxxxxxxxxxxxxxxxxx",
  "concurrent": 2,
  "tags": "kubernetes,heavy",
  "job": {
    "cpuRequest": "500m",
    "cpuLimit": "2",
    "memoryRequest": "1Gi",
    "memoryLimit": "4Gi",
    "ephemeralStorageRequest": "8Gi",
    "ephemeralStorageLimit": "20Gi"
  }
}

Save after edits. Changing the JSON numbers is what raises the cap — saving the same 10Gi limit does not. Existing runners also keep their old concurrent until you change it. Older catalog defaults used concurrent: 10 with no job disk request (kubelet eviction) or a 10Gi job limit (Kaniko kill). Wait for the runner manager instance to restart before retrying a pipeline.

Keep it healthy

• Prefer GitLab distributed cache (S3 / MinIO) over huge local node_modules / Docker layer caches on the node.
• Split runners by tag: light jobs vs Next.js vs image builds.
• After a failed job, check the runner app Logs and GitLab’s job log. Monitor and debug apps.
• Watch node disk from resource monitoring. Image pulls and build caches accumulate even when jobs finish.
• Leave unregisterRunners: true so a deleted Xooshe app does not leave stale runners in GitLab.
• terminationGracePeriodSeconds (default 3600) should exceed your longest job so an upgrade does not kill in-flight pipelines.

If jobs fail

Signal	What it means	What to do
Container build exceeded its local ephemeral storage limit "10Gi"	The job is still running; Kubernetes killed only the build container because it wrote more than the limit. Common in Kaniko when a cached layer is extracted and node_modules is snapshotted again	Set job.ephemeralStorageLimit to 20Gi (request 8Gi), set concurrent to 1–2, save, wait for the manager to restart, retry. Saving the same 10Gi JSON does not raise the cap. One heavy job can also set KUBERNETES_EPHEMERAL_STORAGE_LIMIT: 20Gi in .gitlab-ci.yml
TerminationByKubelet + ephemeral-storage + request is 0	Node temporary disk is below the kubelet threshold; the CI job had no disk request so it was evicted first	Set job.ephemeralStorageRequest / Limit (Kaniko: 8Gi / 20Gi), lower concurrent, save the app, retry
Same kubelet error but usage is already several Gi (for example build was using 3604328Ki)	next build, npm ci, or image layers filled the workspace; several concurrent jobs piled onto one node	Use the Next.js or Kaniko row above; add tags so heavy jobs skip the lint runner
Job pending / not scheduled	Requested temporary disk (or CPU/memory) does not fit any node	Lower job requests, lower concurrent, or add node disk
system failure / pod disrupted, no disk in the message	Node pressure, preemption, or the runner manager restarted	Check runner Logs, node events, and wait for terminationGracePeriodSeconds on upgrades
Runner never picks jobs	Wrong URL/token, or tags on the runner do not match .gitlab-ci.yml	Fix Custom configuration; confirm the runner is online in GitLab
Image pull errors in the job	Job image not reachable from the cluster	Use a private registry the environment can pull

A Kaniko kill (limit too small) looks like:

INFO[1004] Found cached layer, extracting to filesystem
INFO[1132] Saving file app/dist/apps/website-v2 for later use
INFO[1144] Saving file app/node_modules for later use
WARNING: Event retrieved from the cluster: Container build exceeded its local ephemeral storage limit "10Gi".
ERROR: Job failed (system failure): pod "runner-…-concurrent-2-…" status is "Failed"

That is not the same as node-wide eviction. The 10Gi figure is the runner’s job.ephemeralStorageLimit. Raise it, then confirm the runner manager instance restarted.

A typical Next.js eviction (request 0) looks like:

Job failed (system failure): pod is disrupted: reason "TerminationByKubelet",
message "The node was low on resource: ephemeral-storage. ...
Container build was using 3604328Ki, request is 0, has larger consumption of ephemeral-storage.
Container helper was using 48Ki, request is 0, ..."

3604328Ki is about 3.4Gi of temporary disk in the build container — normal for next build — with request 0. Set a job disk request, drop concurrent from 10 toward 2–4, save, and re-run.

You’re done when

• The runner shows online in GitLab.
• A tagged pipeline runs to success (lint on the light runner, Kaniko/nx on the sized runner).
• A second parallel image build does not kill the first on the 10Gi cap.

Full catalog steps: How to use ready-made apps.

NextHow to use ready-made apps

BackgroundReady-made catalog

If stuckSupport tickets