📡 Offline Sigstore: Verifying Signed Images in Air-Gapped Clusters With Cosign Bundles

Keyless signing with Sigstore feels like magic right up until you try to verify in an environment that cannot phone home. The whole keyless model leans on live services: Fulcio mints a short-lived signing certificate, and Rekor records an inclusion proof in a public transparency log. Verification, by default, wants to reach those services again to check the proof. In a regulated, disconnected, or egress-locked cluster, none of that is reachable — and the usual reactions are to punch a hole in the firewall or quietly give up on verification entirely. Neither is acceptable. The good news is that Sigstore has a real offline story, and it hinges on capturing everything you need at signing time into a self-contained cosign bundle.

Why the Default Path Breaks Offline

When you run cosign verify the optimistic way, it does more than check a signature. It validates the signing certificate against Fulcio’s trust root, and it confirms the artifact’s entry exists in Rekor by querying the log. Both of those are network calls to the public Sigstore infrastructure. In an air-gapped runtime, the calls time out, and depending on your flags you either fail closed (now nothing deploys) or fail open (now you are not really verifying anything).

The conceptual fix is to move all the verification material to signing time, when you presumably do have connectivity, and carry it with the artifact. A cosign bundle packages the signature, the signing certificate, and the Rekor inclusion proof into one object. At verify time, cosign verify --offline checks that bundle against a mirrored trust root with no outbound calls. The transparency-log entry is verified from the proof you already captured, not from a live lookup.

Capturing the Bundle at Build Time

Your build environment is where connectivity lives, so that is where you sign and capture. The key is to persist the bundle alongside the image so it travels into the disconnected environment together:

# In the connected build pipeline (keyless, OIDC identity)
cosign sign \
  --bundle checkout.bundle \
  --yes \
  ghcr.io/example/checkout@sha256:abc123...

# The bundle now contains the signature, cert, and Rekor inclusion proof.
# Store it as an artifact and/or push it to the internal registry alongside the image.

The discipline here is that every artifact destined for the air-gapped runtime must carry a bundle. A single image that slips through without one becomes the exception that forces a fail-open rule, and a fail-open rule across the board is the same as no verification.

Mirroring the Trust Root

Offline verification still needs to trust something — specifically the Sigstore TUF root that anchors Fulcio’s CA and Rekor’s signing key. You mirror that root into your environment once, distribute it through whatever channel you already trust (your internal artifact store, configuration management), and pin verification to it:

# In the connected environment: initialize and export the TUF trust root
cosign initialize --mirror https://tuf-repo-cdn.sigstore.dev --root root.json

# Bundle up the local TUF cache and ship it into the air-gapped environment,
# then point verification at the mirrored root rather than the public CDN.

The trust root is the one thing that does expire and rotate, so mirroring is not a one-time act — it is a recurring obligation. A stale mirrored root is the most insidious failure mode in the whole design, because it can silently break verification or, depending on how rotation is handled, accept what it should reject.

Verifying With Zero Outbound Calls

With the bundle captured and the root mirrored, verification runs fully offline. Crucially, you still constrain who signed it — offline does not mean unconstrained:

cosign verify \
  --offline \
  --bundle checkout.bundle \
  --certificate-identity "https://github.com/example/checkout/.github/workflows/release.yml@refs/heads/main" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  ghcr.io/example/checkout@sha256:abc123...

The --certificate-identity and --certificate-oidc-issuer flags are doing the real security work. Without them, you would accept anything signed by any identity Sigstore trusts — which is most of the open-source world. Pinning the expected workflow identity and issuer is what makes the verification mean “our release pipeline signed this,” not just “someone signed this.”

Enforcing It at Admission

Once manual verification works offline, push it to the cluster so unsigned or untrusted images cannot run. A Kyverno or policy-controller rule does the same offline check at admission, against the mirrored root, and fails closed. Roll it out the way you would any blocking control — audit mode first, confirm every pipeline emits bundles, then enforce. Flipping fail-closed admission on before coverage is complete is the fastest way to block a legitimate deploy and lose the room. This is the same staged-rollout discipline that makes keyless image signing safe to adopt in connected clusters; the air-gapped case just raises the stakes on the trust-root half.

Using AI to Pressure-Test the Design

The failure modes of offline verification are easy to miss because the happy path looks identical to the broken path until something rotates. This is a good place to have a model enumerate the edge cases against your specifics before you commit:

Prompt: “Review this offline Sigstore design for an air-gapped Kubernetes cluster. List every failure mode where verification could silently pass when it should fail, or fail when it should pass. For each, say whether it must fail closed and how I’d detect it. Inputs: bundles captured at build, mirrored TUF root refreshed monthly, Kyverno admission with —certificate-identity pinned.”

Output (excerpt): “Stale trust root: if the monthly refresh is missed and the public root rotates, your mirror may reject valid new signatures (fail closed — acceptable) OR, if you disabled expiry checks to avoid breakage, accept signatures it shouldn’t (fail open — unacceptable; never disable expiry to paper over a missed refresh). Detection: alert when the mirrored root’s expiry is within 7 days, and fail the build that produces an image whose bundle can’t be verified against the current mirrored root, so coverage gaps surface at build time, not deploy time.”

Treat that as a checklist to verify against your real configuration, not as a verdict. The model is good at surfacing the rotation-and-staleness traps that are hard to hold in your head; you confirm each one against the actual deployment.

The One Rule That Holds It Together

Offline Sigstore works, but it trades a live trust anchor for a mirrored one, and that trade only pays off if you own the mirror’s freshness as seriously as you own the signatures. Capture a bundle for every artifact, pin the signing identity, fail closed at admission, and alert before the mirrored root expires. Do those four things and you get the integrity guarantee of keyless signing without the network dependency that disconnected environments cannot satisfy. If you are designing this from scratch, the air-gapped verification prompt walks the bundle capture, mirroring, and rotation runbook step by step so you do not discover the stale-root failure mode in production.