s0sta
EN NL عربي
Filters
How to search
  • Plain words find names, texts and examples.
  • "two words" finds the exact phrase.
  • --name or -x finds an option.
  • #word looks only inside commands.
  • @grep finds lessons and labs that use grep.
  • Filters: cat:network · in:labs · in:notes · level:rhcsa · ch:8 · root:yes. Put ! before a word to leave it out. OR between words means one of them.
  • ↑ ↓ to move · Enter to open · Esc to close · / to search

Git & CI/CDShellSec series

Four deploy failures

Four failures account for most bad deploys, and none of them is exotic. Each one is a check somebody skipped, and each one is cheaper to catch before the rollout than after it.

Quick reference

Wrong context

  • The runner kept a kubeconfig from the last job.
  • Production manifests land on a staging cluster.
  • Every health check passes, on the wrong cluster.

Stale image

  • The tag was rebuilt; the name never changed.
  • A node reuses the image it already pulled.
  • The pod is Ready and runs the previous code.

Failed smoke

  • Health returns 200 while checkout returns 500.
  • One retry hides a backend that flaps.
  • The pipeline never calls a paying endpoint.

No rollback

  • revisionHistoryLimit: 0 deleted the undo.
  • The previous image was pruned from the registry.
  • A migration moved the schema forward.

Note: Each one is cheaper to catch before the rollout than after it.

Explained

Wrong context

The runner keeps the kubeconfig from the previous job, or someone runs the prod manifest without switching. The release lands on staging, passes every health check, and production stays on the old version. The context check is first because it decides which cluster answers every later question.

Stale image

The tag v2 is rebuilt, the name never changes, and a node reuses the image it already pulled. The pod reports Ready while running the previous code. A digest pin is the only reference that cannot be silently rewritten.

Failed smoke

Health returns 200 because the process is listening; checkout returns 500 because a dependency is not configured. One retry hides a backend that flaps. A smoke test has to assert a status and a body on the endpoint that earns money.

No rollback

revisionHistoryLimit was set to 0 to keep the cluster tidy, the previous image was pruned from the registry, or a migration moved the schema forward and the old code cannot read it. An undo needs a target, an image and a compatible schema.

The honest caveat: these four checks do not cover a bad migration or a slow leak, which is why the gate is a floor, not a ceiling.

The lesson: deploy failures are check failures that arrived late.

In one line

None of these are surprises. Each one is a check that did not run.

The first check

kubectl config current-context

More notes: Git & CI/CD

Open in the notes app: search, filters, copy This note on GitHub