The Kubernetes Deployment Checklist

A real Kubernetes deployment checklist is the difference between a rollout that drops zero requests and one that sheds a burst of errors every single time you ship. Kubernetes will happily run a deployment that has no resource requests, lying probes, and no graceful shutdown, and it will punish you for all three under load. This checklist is the set of items that make a deployment production-safe, ordered by how often teams skip them and pay for it.

Most production incidents on Kubernetes are not exotic. They are a missing PodDisruptionBudget during a node drain, or a service that never handled SIGTERM, surfacing on an ordinary Tuesday deploy. The checklist exists to make those non-events.

Why a deployment checklist matters

Kubernetes gives you enormous power and very few guardrails. A deployment manifest with none of the safety items still deploys; the failure shows up later, as dropped requests on rollout, a node drain that takes the whole service down, or a memory leak that kills a node. A checklist turns those latent failures into things you handled before they fired.

The throughline of every item below is the same: make the deployment behave correctly during the disruptions that are normal in Kubernetes, deploys, scale events, node drains, and pod evictions. This post is part of the Kubernetes operations series and ties together several of its deep dives.

What should a Kubernetes deployment checklist include?

A production deployment needs resource requests and limits, the three probe types, a safe rolling-update strategy, a PodDisruptionBudget, graceful shutdown, autoscaling, externalized config and secrets, and observability. The items most often missing, and most often the cause of deploy-time incidents, are graceful shutdown and PodDisruptionBudgets.

Here is the full checklist, grouped:

Area	Item	Why it matters
Resources	CPU/memory requests	Scheduling + no CPU starvation
Resources	Memory limit	Contains a leak before it kills the node
Health	Readiness probe	Gates traffic to ready pods only
Health	Liveness probe	Restarts a wedged process (lenient)
Health	Startup probe	Covers slow boots without false kills
Rollout	Rolling update strategy	Gradual replace; set maxUnavailable/maxSurge
Rollout	PodDisruptionBudget	Caps simultaneous voluntary disruptions
Lifecycle	Graceful shutdown	Drains in-flight requests on rollout
Scaling	HPA (right signal)	Responds to real load
Config	Externalized config + secrets	No rebuild to change config; no secrets in image
Ops	Metrics, logs, traces	Debuggable when the deploy goes wrong

You do not have to add all of these on day one, but a service that takes real traffic should have every one before you call it production-grade.

How do you deploy to Kubernetes without downtime?

Zero-downtime deploys come from four items working together: a rolling update (replace pods gradually), accurate readiness probes (send traffic only to pods that can serve), graceful shutdown (drain in-flight requests before a pod exits), and a PodDisruptionBudget (stop too many pods going down at once). Remove any one and a rollout starts dropping requests.

The sequence on a rollout should be: a new pod starts, passes its readiness probe, and joins the endpoints; an old pod is sent SIGTERM, immediately fails its readiness probe so it stops receiving new requests, finishes its in-flight work within the termination grace period, then exits. When that choreography is correct, a deploy is invisible to users. When readiness or shutdown is wrong, every deploy sheds a burst of errors.

What is the most commonly missed item in Kubernetes deployments?

Graceful shutdown, by a wide margin. Many deployments never handle SIGTERM and have no preStop hook, so when Kubernetes terminates a pod on rollout, in-flight requests are cut mid-flight. The fix is to fail readiness immediately on shutdown so no new traffic arrives, then drain existing work within the termination grace period before exiting.

The related subtlety is the termination grace period: it must be long enough for your longest in-flight request to finish, or the pod is force-killed mid-request anyway. Set it to comfortably exceed your real request duration, and make the application actually stop accepting new work the moment it receives SIGTERM.

Resource requests, limits, and the CPU-limit nuance

Always set resource requests, because they drive scheduling and guarantee the pod is not starved of CPU during normal operation or startup. Limits deserve more thought. A memory limit is valuable: it contains a memory leak so a single pod cannot take down the whole node. A CPU limit is double-edged: it caps a pod’s CPU and can throttle a latency-sensitive service at exactly the wrong moment.

The pragmatic stance most operators land on is: requests always, memory limits usually, CPU limits cautiously or not at all for latency-sensitive services. The reason is that CPU is compressible (throttling slows you down) while memory is not (running out kills you), so the two limits protect against different severities of failure and deserve different defaults.

How do you roll back a bad Kubernetes deployment?

Roll back fast and automatically where you can. Kubernetes keeps a deployment’s revision history, so a manual rollback to the previous known-good revision is a single command, but the better posture is to catch a bad rollout before it fully ships. A progressive rollout that watches health and halts on regression turns “page, diagnose, roll back” into “the rollout stopped itself.”

The baseline capability is the built-in rollout history: every deployment update creates a revision, and you can return to the prior one quickly. That is your floor, and every team should know the command and have tested it. The failure mode it guards against is the deploy that looked fine in CI and degrades under real traffic, which is exactly when seconds matter and nobody wants to be reconstructing the previous config by hand.

Above that floor, progressive delivery shrinks the blast radius. A canary or blue-green rollout exposes the new version to a slice of traffic first, watches the golden signals, and only proceeds if they hold, rolling back automatically if they do not. This pairs directly with the observability you already need: the same metrics that page you can gate a rollout. The goal is that a bad deploy is contained to a fraction of traffic for a few minutes, not shipped to everyone and then frantically reverted.

A pre-ship deployment checklist

Before a deployment serves production traffic:

• Resource requests set; memory limit set; CPU limit chosen deliberately.
• Readiness, liveness, and startup probes configured and meaningful (not fixed sleeps).
• Rolling update strategy with sane maxUnavailable / maxSurge.
• A PodDisruptionBudget so node drains cannot take the service down.
• Graceful shutdown: SIGTERM handled, readiness fails first, grace period exceeds longest request.
• Autoscaling configured on the right signal.
• Config and secrets externalized; nothing sensitive baked into the image.
• Metrics, logs, and traces wired so a bad rollout is immediately debuggable.

Why do the same items get missed every time?

Deployment checklists exist because a small set of items is skipped repeatedly, and it is worth understanding why — because the reason predicts which items need automation rather than reminders.

The pattern: the missed items are the ones with no symptom in normal operation. A missing readiness probe causes no problem until a rollout. An absent PodDisruptionBudget causes nothing until a node drain. Unset resource requests are invisible until the cluster is under pressure. A service works perfectly in staging and in production, and the omission only surfaces during an event that happens weeks later — by which point nobody connects it to the deployment.

That is why exhortation does not fix them. The feedback loop between the omission and the consequence is too long for anyone to learn from, and every individual deploy that skipped the item appeared to succeed.

The items in this category, in rough order of how often they bite:

• Resource requests and limits. Without requests, the scheduler cannot place pods sensibly and eviction ordering is effectively arbitrary.
• Readiness probe correctness. Not its presence — its truthfulness. A probe returning 200 unconditionally is worse than none, because it makes failure look like health.
• A preStop delay. The single line that makes rolling deploys stop dropping requests, and almost always absent.
• PodDisruptionBudget. Nothing notices until a cluster upgrade takes the service down.
• Anti-affinity or topology spread. Three replicas on one node is three replicas of nothing when that node fails.
• A tested rollback. Everyone believes they can roll back; far fewer have done it under load.

The fix is enforcement, not documentation. An admission policy that rejects a Deployment without resource requests, or a CI check that fails a manifest lacking a readiness probe, converts a checklist item into something that cannot be forgotten. That is the same escalation described elsewhere in this series: move a rule from documented to guardrail, and it stops depending on anyone remembering.

For the handful that genuinely cannot be automated — has the rollback been rehearsed, does the probe tell the truth — a short pre-ship review with a named owner is the fallback. But automate everything that can be automated first, because a checklist of twenty items where fifteen are machine-checkable trains people to skim the whole thing.

What belongs in the deployment manifest itself?

Separating what must be in the manifest from what belongs elsewhere prevents both an unreviewable wall of YAML and the more common failure of a manifest missing the fields that matter.

Every production Deployment should specify:

Field	Why it matters
resources.requests	The scheduler places pods on this; without it, placement is guesswork and eviction order is arbitrary
resources.limits.memory	Bounds the blast radius of a leak; a pod without one can take down its node
readinessProbe	Gates traffic; must reflect real ability to serve
startupProbe for slow boots	Lets steady-state probes stay tight without killing pods during startup
lifecycle.preStop	The delay that lets endpoint removal propagate before SIGTERM
terminationGracePeriodSeconds	Must exceed preStop plus the longest in-flight request
strategy.rollingUpdate	maxUnavailable and maxSurge govern your own deploys, which a PDB does not
topologySpreadConstraints	Stops all replicas landing on one node or one zone
securityContext	Non-root, read-only root filesystem, no privilege escalation

The CPU limit is the deliberate omission and the most contested item. A memory limit is protective, because memory is incompressible and an unbounded pod can kill a node. A CPU limit is different: CPU is compressible, and a limit causes throttling that shows up as latency spikes on a pod that has capacity available to it. Set CPU requests always; set CPU limits only when you specifically need to cap a workload, and expect to see throttling metrics if you do.

Two structural points. Keep environment-specific values out of the base manifest and apply them through an overlay, so the base never encodes a production-only value — the pattern in Environment Config Overlays for Kubernetes. And secrets come from a secret manager, never from a literal in the manifest, since manifests end up in Git and in every developer’s checkout.

How do you make a rollback something you trust?

Everyone lists rollback on the checklist. Far fewer have a rollback they have actually exercised, and the gap surfaces during the incident where it matters most.

Four properties separate a real rollback from a hoped-for one:

It is one command, and people know it. kubectl rollout undo for the simple case. If your rollback requires reconstructing a previous state by hand, it is not a rollback, it is a rebuild.

It has been performed in production. Not in staging. Roll back a real deploy deliberately, during business hours, and time it. The number you get is your actual recovery time, and it is usually longer than assumed because of image pulls and startup.

Database migrations do not block it. This is where most rollbacks genuinely fail. A migration that drops a column makes the previous version unable to run, so the code rollback is unavailable exactly when needed. The discipline is expand-and-contract: add the new column, deploy code writing both, migrate, deploy code reading the new one, and only remove the old column after the previous version is no longer deployable. Every schema change should be backward-compatible for at least one release.

Enough history is retained. revisionHistoryLimit controls how many previous ReplicaSets survive. Set too low, the version you want is gone.

The framing worth adopting: rollback capability is a property of your migration discipline, not of your deployment tool. Kubernetes will happily roll back the pods; whether the previous version can actually run against the current database is decided weeks earlier, by whoever wrote the migration.

How do you deploy without dropping requests?

Zero-dropped-requests during a rollout is achievable and it is not the default. Four things must be true together, and missing any one produces the small error burst that most teams accept as normal.

The pod must outlive its own removal from the endpoints. SIGTERM and endpoint removal happen in parallel, so a pod that exits promptly dies while proxies still route to it. A preStop sleep of several seconds covers the propagation window. This is the single most impactful item and the most frequently missing.

The application must drain rather than exit. On SIGTERM: stop accepting new connections, finish in-flight requests, then exit. A process that exits immediately on the signal drops whatever it was holding.

The grace period must accommodate both. terminationGracePeriodSeconds has to exceed the preStop delay plus the longest realistic in-flight request, or the kubelet sends SIGKILL mid-drain and undoes the work.

Readiness must be truthful on the way up. The replacement pod must not receive traffic before it can serve. A probe that returns ready early produces errors on every rollout that look like a load-balancer problem.

The verification is straightforward and worth doing once per service: run a steady load generator and perform a rolling restart. Any non-zero error count means one of the four is wrong. Zero errors under load during a rollout is an achievable bar, and until you have measured it you do not know which side of it you are on.

One addition for larger deployments: rollout speed interacts with capacity. maxUnavailable: 25% on a service running near its limit removes a quarter of your capacity mid-rollout, and the remaining pods may not absorb it. For anything close to saturation, prefer maxUnavailable: 0 with a positive maxSurge, which adds capacity before removing any — at the cost of briefly needing headroom for the extra pods.

A closing note on how to keep the checklist alive: prune it. A list of forty items is skimmed and therefore useless; a list of ten that are genuinely load-bearing gets read. Every time an item becomes machine-enforced, remove it from the human list — that is the point of automating it. And every time an incident traces back to a missed item, add it. A checklist that only grows is one that stops being read, which returns you to the situation it existed to prevent.

The discipline that keeps this honest is treating the checklist itself as something with an owner and a review cadence, exactly like the runbooks and dashboards it sits alongside. Artefacts without owners decay in the same way regardless of how useful they were on the day they were written.

Name the owner in the document itself, so the question of who maintains it never needs to be asked.

An unowned checklist is indistinguishable from an out-of-date one within about two quarters, and the failure is silent because nothing announces that a document has stopped being true. Reviewing it after each incident that touches a deployment is usually cadence enough.

That cadence has the advantage of being triggered by something that already commands attention, rather than depending on a calendar reminder everyone eventually declines.

It also keeps the list grounded in what has actually gone wrong in your environment, rather than in what goes wrong generically, which is what makes a borrowed checklist feel irrelevant and get ignored.

What I’d do differently

The lesson I keep relearning is that the boring items, graceful shutdown, PodDisruptionBudgets, are the ones that actually cause production pain, while the exciting ones get all the attention. A team will spend a week on a fancy deployment strategy and never add a preStop hook, then wonder why every deploy sheds errors.

If I were standardizing deployments again, I would encode this checklist as a template and a CI policy check, so a deployment literally cannot ship without requests, probes, a PDB, and graceful shutdown. Making the safe path the default path is the only way these items stop being “things we meant to add.” The checklist is cheap; the incidents it prevents are not.

Sources

• Kubernetes, Deployments and rolling updates: kubernetes.io/docs/concepts/workloads/controllers/deployment
• Kubernetes, Pod termination and graceful shutdown: kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination
• Kubernetes, Resource management for pods: kubernetes.io/docs/concepts/configuration/manage-resources-containers