🛢️ GitLab CI Services: Running Databases and Sidecars Inside Your Jobs

My integration tests used to mock the database, which meant they passed beautifully and then production fell over on a query the mock never simulated. The fix was running tests against a real Postgres in CI — and GitLab’s services: keyword makes that almost trivial, once you understand how the networking and timing actually work. The “almost” is where people lose an afternoon, so this is the guide I wish I’d had: how services: works, the gotchas that bite everyone, and where AI saves you time without leading you off a cliff.

What services: actually does

A services: entry starts an additional container alongside your job’s main container, on the same network, before your script runs. Your test code talks to it over the network by hostname.

integration-test:
  stage: test
  image: node:20
  services:
    - postgres:16
  variables:
    POSTGRES_DB: app_test
    POSTGRES_USER: runner
    POSTGRES_PASSWORD: runner
    DATABASE_URL: "postgres://runner:runner@postgres:5432/app_test"
  script:
    - npm ci
    - npm run test:integration

The key detail: the service is reachable at the hostname postgres — the image name with slashes turned to dashes. Your DATABASE_URL points at postgres:5432, not localhost. This trips up nearly everyone the first time; the database is in a sibling container, not your job’s container.

Multiple services and custom aliases

You can run several. For a job that needs Postgres and Redis, plus a custom hostname:

api-test:
  image: python:3.12
  services:
    - name: postgres:16
      alias: db
    - name: redis:7
      alias: cache
  variables:
    POSTGRES_DB: app
    POSTGRES_PASSWORD: secret
    DATABASE_URL: "postgres://postgres:secret@db:5432/app"
    REDIS_URL: "redis://cache:6379/0"
  script:
    - pytest tests/integration

The alias lets you pick the hostname instead of using the derived one, which is cleaner and survives an image change. Note the password here is a throwaway test credential, not a real secret — that distinction matters, and I’ll come back to it.

The timing trap: services aren’t instantly ready

The single most common flaky-test cause with services: the container is started before your script, but the database inside it may not be accepting connections yet. Your first query races the boot and intermittently fails. The fix is to wait for readiness explicitly:

  script:
    - until pg_isready -h db -U postgres; do echo "waiting for db"; sleep 1; done
    - pytest tests/integration

A readiness loop turns a flaky pipeline into a reliable one. AI will draft the service block correctly but routinely omits this wait — it’s the first thing I add back, every time.

Pro Tip: For services without a clean readiness probe, hit the port with a loop like until nc -z cache 6379; do sleep 1; done. A few seconds of waiting beats hours of debugging intermittent CI failures that you can’t reproduce locally.

Docker-in-Docker as a service

Building images inside CI uses the same mechanism — the Docker daemon is a service:

build-image:
  image: docker:27
  services:
    - docker:27-dind
  variables:
    DOCKER_HOST: "tcp://docker:2376"
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: "1"
    DOCKER_CERT_PATH: "/certs/client"
  script:
    - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

The dind service provides the daemon; your job’s docker client connects over TLS. Getting the DOCKER_HOST and DOCKER_TLS_* variables right is fiddly and version-dependent, which makes it a good thing to have AI draft from a known-good template — then verify against your runner’s Docker version, because these settings changed across Docker releases and AI will happily mix versions.

Passing config and entrypoint commands

Some images need a command or extra config. services: supports command and entrypoint:

  services:
    - name: postgres:16
      alias: db
      command: ["postgres", "-c", "max_connections=200"]

This is handy when your test load exceeds the default connection limit — a real failure mode I hit on a busy parallel test job.

Pulling service images from a private registry

The default examples pull service images from Docker Hub, which has rate limits that will bite a busy CI fleet — toomanyrequests failures that look like flakiness but are really throttling. The fix is to pull service images from your own registry, and to authenticate the pull. GitLab reads DOCKER_AUTH_CONFIG to authenticate service and image pulls:

api-test:
  image: "$CI_REGISTRY/mirror/python:3.12"
  services:
    - name: "$CI_REGISTRY/mirror/postgres:16"
      alias: db
  variables:
    DATABASE_URL: "postgres://postgres:secret@db:5432/app"
  script:
    - pytest tests/integration

The DOCKER_AUTH_CONFIG itself is a masked, protected CI/CD variable — a JSON blob with registry credentials — set once in project or group settings, never in the YAML. Mirroring upstream images into your registry also gives you a pinned, scanned supply chain for your test dependencies, which matters more than people think: a compromised postgres image in CI runs with whatever access the job has.

This is a place where the throwaway-versus-real-secret distinction gets subtle. The POSTGRES_PASSWORD in the service is disposable. The DOCKER_AUTH_CONFIG that lets you pull the image is a real credential and gets the full protected-variable treatment. Conflating the two — putting registry auth inline because “it’s just CI” — is exactly the mistake that leaks a credential into your git history.

The secrets rule, applied to services

Here’s the nuance worth stating plainly: the credentials in a service block — POSTGRES_PASSWORD: runner — are ephemeral test credentials for a throwaway container that lives for the duration of one job. Those are fine to commit. What is never fine is putting a real secret — your production database password, a real API token — into a service or job variable in the YAML. Real secrets live in masked, protected CI/CD variables, and they never touch a service definition or a chat window.

When I ask AI for help with a flaky service, I share the YAML and the log, and the only credential in there is the disposable test one. I never paste a real connection string to “let it test the query.” That’s the line.

Where AI helps and where to verify

services: config is well-documented and repetitive, which makes it ideal fast-junior-engineer work. AI reliably:

• Generates the service block and the matching connection-string variables.
• Remembers the hostname-is-the-image-name rule (mostly).
• Drafts the dind TLS variable set from a template.

It reliably gets wrong:

• Omitting the readiness wait, causing flaky tests.
• Mismatching dind variables across Docker versions.
• Using localhost instead of the service hostname in older examples it learned from.

So I draft fast and read every line before merge, adding the readiness loop and checking hostnames against the real run. For a careful pass over a CI change that introduces real service dependencies, the code review dashboard catches the subtle ones. And when integration tests start failing in CI but not locally, the incident response dashboard keeps the triage structured.

My reusable prompt: “Write a GitLab CI integration-test job using services: for Postgres 16 and Redis 7. Set connection-string variables using the service hostnames, and include explicit readiness waits before the tests run. Use throwaway test credentials only.” That last clause keeps real secrets out of the YAML by design. Variants live in my prompt library and the testing prompt packs.

Conclusion

services: gives every job a real Postgres, Redis, or Docker daemon, so your integration tests exercise the actual thing instead of a mock that lies. Mind the sibling-container networking, always wait for readiness, and keep the distinction between throwaway test creds (fine in YAML) and real secrets (never in YAML, never in chat). Let AI draft the verbose config, then verify it yourself. More guides in the GitLab CI/CD category.