🚨 GitLab CI Error Guide: 'connection refused' to a CI Service — Fix services: Networking

Overview

This failure appears when a job tries to reach a database, cache, or other dependency declared under services: and the connection is rejected. The Go/tooling variant looks like this:

dial tcp 127.0.0.1:5432: connect: connection refused

Client libraries print their own wording for the same underlying condition:

psql: error: connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
curl: (7) Failed to connect to postgres port 5432: Connection refused

The error is not authentication and not DNS resolution — the target host answered by refusing the TCP connection, which means either nothing is listening on that host/port yet, or the job is dialing the wrong address for how the runner networks service containers.

Symptoms

• The job connects fine on your laptop with docker compose but fails in CI with connection refused.
• Integration tests fail immediately at startup while trying to reach Postgres, MySQL, Redis, or a mock server declared under services:.
• The error is intermittent: it passes when the service happens to start quickly and fails when it is slow (a race, not a config error).
• With the Kubernetes executor, connecting to localhost/127.0.0.1 fails, while the same config works on the Docker executor.
• getaddrinfo / “could not resolve host” would indicate a name problem; here the name resolves but the port refuses, pointing at readiness or the wrong host alias.

Common Root Causes

• Service not ready yet — the job’s script: starts before Postgres/MySQL finishes initializing; the port is not listening at the instant of the first connection.
• Wrong hostname for the executor — on the Docker executor a service is reachable by its image/alias hostname (e.g. postgres), not always localhost; on the Kubernetes executor services share the pod network so they are reached via localhost, and the image-name alias does not resolve.
• Wrong port — connecting to the default port when the service image listens on a different one, or the app expects a port the service never exposes.
• Service failed to start — a bad command:, a missing required env var (e.g. POSTGRES_PASSWORD), or the image crash-looping, so nothing ever listens.
• Alias mismatch — a custom alias: was set but the client still uses the default image-derived hostname.
• App binds to the wrong interface — a service that binds only to its own loopback, unreachable from the job container on the Docker executor.

Diagnostic Workflow

First make the executor and the service explicit in .gitlab-ci.yml, then prove readiness before the real work runs. On the Docker executor, reach the service by its alias; add an explicit wait loop so a slow start does not race the test:

integration-test:
  stage: test
  image: python:3.12
  services:
    - name: postgres:16
      alias: db            # reach it at host "db"
  variables:
    POSTGRES_DB: appdb
    POSTGRES_USER: app
    POSTGRES_PASSWORD: app-ci-password
    # App/client must point at the service alias, not localhost, on Docker executor:
    DATABASE_URL: "postgresql://app:app-ci-password@db:5432/appdb"
  script:
    - apt-get update && apt-get install -y postgresql-client
    - |
      for i in $(seq 1 30); do
        pg_isready -h db -p 5432 -U app && break
        echo "waiting for postgres ($i/30)..."; sleep 2
      done
    - pg_isready -h db -p 5432 -U app        # fail loudly if still down
    - pytest tests/integration

On the Kubernetes executor, the service runs as another container in the same pod, so it is reachable on localhost — change the host, not the logic:

integration-test-k8s:
  stage: test
  image: python:3.12
  services:
    - name: postgres:16
  variables:
    POSTGRES_DB: appdb
    POSTGRES_USER: app
    POSTGRES_PASSWORD: app-ci-password
    DATABASE_URL: "postgresql://app:app-ci-password@localhost:5432/appdb"  # pod-local
  script:
    - apt-get update && apt-get install -y postgresql-client
    - until pg_isready -h localhost -p 5432 -U app; do sleep 2; done
    - pytest tests/integration

Validate the file with CI Lint (Pipeline editor → Validate) before pushing. To see why a service never listened, inspect the service container startup — run the runner with service logging so the crash reason surfaces:

# On a self-managed runner host, enable service/health logs
gitlab-runner --debug run
# and in config.toml under [runners.docker]:
#   [[runners.docker.services]]  # or set the feature flag below

variables:
  # Surface service container logs into the job trace to see startup failures
  CI_DEBUG_SERVICES: "true"

With CI_DEBUG_SERVICES: "true", a crashing Postgres will print its own error (for example a missing POSTGRES_PASSWORD) in the job log, turning a silent “connection refused” into an actionable message.

Example Root Cause Analysis

A team’s integration job passed roughly half the time and failed the rest with:

dial tcp 127.0.0.1:5432: connect: connection refused

They ran on the Docker executor and their app read DATABASE_URL=postgresql://app:app@localhost:5432/appdb. Two problems compounded. First, on the Docker executor the Postgres service is a separate container, not localhost, so 127.0.0.1 inside the job container had nothing on 5432 — it only worked when a leftover local Postgres happened to exist on some runners. Second, there was no readiness wait, so even after fixing the host, a cold start still lost the race.

Enabling CI_DEBUG_SERVICES: "true" confirmed Postgres started fine but on its own container. The fix was to point the app at the service alias and add a pg_isready wait loop:

  services:
    - name: postgres:16
      alias: db
  variables:
    DATABASE_URL: "postgresql://app:app@db:5432/appdb"
  script:
    - until pg_isready -h db -p 5432 -U app; do sleep 2; done
    - pytest tests/integration

The job became deterministic and the intermittent refusals disappeared.

Prevention Best Practices

• Know your executor: on Docker reach services by their alias/image hostname; on Kubernetes reach them via localhost (shared pod network).
• Always add a readiness wait (pg_isready, redis-cli ping, wait-for-it, nc -z) before the first real connection — never assume the service is up when your script starts.
• Set every required service env var (e.g. POSTGRES_PASSWORD) so the service actually starts and listens.
• Set an explicit alias: and use that exact name in your connection string so host and client never drift.
• Turn on CI_DEBUG_SERVICES: "true" while debugging so a crash-looping service prints its startup error into the job trace.
• Pin the service port explicitly and confirm the image listens on it, rather than relying on a default the image may not use.

Quick Command Reference

# Readiness probes to gate before connecting
pg_isready -h db -p 5432 -U app          # Postgres
redis-cli -h redis -p 6379 ping          # Redis -> PONG
nc -z db 5432                            # generic TCP port open?
until nc -z db 5432; do sleep 2; done     # wait-loop pattern

# Reach services correctly per executor
# Docker executor:      host = service alias  (e.g. "db")
# Kubernetes executor:  host = localhost      (shared pod network)
variables:
  CI_DEBUG_SERVICES: "true"   # print service container logs into the job trace

Conclusion

connection refused to a services: container is a networking-and-readiness problem, not auth: the name resolved but nothing was listening at that address yet. Confirm which executor you run on — dial the alias on Docker and localhost on Kubernetes — ensure the service has every env var it needs to start, and always gate the first connection behind a readiness loop like pg_isready. Turn on CI_DEBUG_SERVICES to expose a crashing service, and the intermittent refusals become deterministic, passing jobs.