DevOps Practices

DevOps isn’t a job title or a tool — it’s a set of practices for delivering software quickly and reliably, by treating the whole path from commit to production as one engineered system. This guide covers the practices that matter, but with a twist most “best practices” lists skip: for each one, why it matters, how teams get it wrong, and when it’s not worth the cost. A practice adopted as cargo-cult ritual (microservices because Netflix, Kubernetes for three services) does more harm than the problem it was meant to solve. Adopt deliberately.

Foundation practices

Version control for everything

Why: Git isn’t just for app code — infrastructure, pipeline config, docs, and policies all belong in version control, giving you history, review, and rollback for the entire system. Common mistake: long-lived feature branches that drift for weeks and merge in pain. Maturity looks like: trunk-based development — short-lived branches merged to main daily behind feature flags — which is strongly correlated with high delivery performance. When not: trunk-based needs solid automated testing and feature flags to be safe; a team without those should build them first rather than merge broken code to main. See Git Commands.

Infrastructure as code

Why: Provision infrastructure from version-controlled, reviewable code (Terraform/OpenTofu, Pulumi) instead of console clicks — so environments are reproducible, auditable, and recoverable. Common mistake: manual “quick fixes” in the console that cause drift, so the code no longer matches reality and the next apply reverts a production hotfix. Maturity looks like: all changes through IaC, drift detection in CI, and modules that encode your standards. When not: IaC has a learning curve; for a genuine one-off throwaway, the console is fine — just don’t let throwaway become production.

Immutable infrastructure

Why: Replace servers rather than patching them in place — build a new image/container and swap it, so every instance is identical and reproducible, and configuration drift is impossible. Common mistake: SSHing in to “just fix one thing,” reintroducing the snowflake servers immutability was meant to kill. When not: stateful systems (databases) can’t be casually replaced; immutability applies cleanly to the stateless tier and needs careful data handling elsewhere.

Delivery practices

CI/CD

Why: Continuous integration (every commit built and tested automatically) catches breakage early; continuous delivery (every green build is deployable) makes releases boring and frequent. Small, frequent deploys are lower risk than big-bang releases — less changes each time, so failures are smaller and easier to pinpoint. Common mistake: a slow, flaky pipeline that engineers learn to ignore or bypass — a pipeline nobody trusts is worse than none. Maturity looks like: fast (under 10 min) reliable pipelines, deploys many times a day, automated rollback. See GitHub Actions.

Automated testing

Why: Tests are what make frequent deployment safe. The test pyramid — many fast unit tests, fewer integration tests, a handful of end-to-end tests — gives fast feedback without a brittle, slow suite. Common mistake: the inverted pyramid (mostly slow, flaky E2E tests) that takes an hour and fails randomly, so people stop trusting it. When not: 100% coverage is a vanity metric; test behavior that matters and code that’s risky, not trivial getters. Diminishing returns are real.

Progressive delivery, feature flags, and rollback

Why: Decouple deploy from release. Feature flags let you ship code dark and turn it on for a cohort; canary/blue-green deployments expose a new version to a slice of traffic and watch metrics before full rollout; automated rollback reverts fast when a canary regresses. Together they make the blast radius of a bad change tiny. Common mistake: flags that never get cleaned up, accumulating into unmaintainable conditional spaghetti. Maturity looks like: every risky change behind a flag, canaries gated on SLO metrics, one-click (or automatic) rollback. When not: a small team shipping low-risk changes may not need full canary infrastructure yet — but feature flags are cheap and worth adopting early.

Operations practices

Observability: metrics, logs, traces

Why: You cannot operate what you cannot see. Metrics tell you that something’s wrong (error rate up), logs tell you what (the exception), traces tell you where (which service in the chain). Common mistake: logging everything and alerting on nothing actionable — or the reverse, alert fatigue from noisy thresholds. Maturity looks like: structured logs, RED/USE-method dashboards, distributed tracing, and alerts tied to user-facing symptoms, not internal noise. See System Design and DevOps Tools.

SRE: SLIs, SLOs, and error budgets

Why: An SLI is a measured indicator (p99 latency, success rate); an SLO is the target (99.9% success); the error budget is the allowed shortfall (0.1%). This reframes reliability as a budget to spend: within budget, ship features fast; budget exhausted, freeze features and fix reliability. It turns “how reliable should we be?” from an argument into a number. Common mistake: chasing 100% — the last nines cost exponentially more and users can’t tell 99.9% from 99.99% for most products. When not: a pre-product-market-fit startup may rationally choose to spend its error budget freely; SLOs matter most once users depend on you.

Incident response and blameless postmortems

Why: Incidents are inevitable; how you respond and learn is what compounds. A clear on-call rotation, defined severities, and an incident commander role turn chaos into process. Blameless postmortems focus on the system that let the failure happen, not the person who pushed the button — because in a blame culture people hide problems, and hidden problems recur. Common mistake: a postmortem that names a “root cause” as human error and closes — instead of finding the missing guardrail, alert, or test. Maturity looks like: every significant incident produces concrete, tracked action items that remove a class of failure.

Disaster recovery, capacity, and cost

• DR — know your RPO/RTO, back up across regions, and test restores and failover. An untested DR plan is a liability dressed as a control.
• Capacity planning — provision for peak with headroom, autoscale on real signals, and load-test before big events rather than discovering limits in production.
• Cost optimization — treat cloud spend as an engineering metric (FinOps): right-size, use spot/committed capacity where appropriate, and attribute cost to teams so it’s visible. Common mistake: optimizing cost so aggressively you sacrifice reliability or engineer time worth more than the savings.

Organizational practices

Platform engineering and developer experience

Why: As infrastructure complexity grows, expecting every developer to be a Kubernetes and cloud expert doesn’t scale. A platform team builds an internal platform (golden paths, self-service, paved roads) that lets product engineers ship without deep infra knowledge — treating developers as customers. Common mistake: building a platform nobody asked for, or one so rigid it blocks the teams it’s meant to serve. When not: a 10-person company doesn’t need a platform team; this is a scaling practice that earns its keep once cognitive load on product teams becomes the bottleneck.

Automated security, secrets, and policy as code

Why: Shift security left — scan dependencies, code, IaC, and images in the pipeline so issues surface in the PR, not in production (or a breach). Manage secrets in a dedicated store with rotation, never in code. Encode guardrails as policy as code (OPA, Kyverno, Sentinel) so compliance is enforced automatically rather than by review heroics. Common mistake: security as a gate at the end that engineers route around; it has to be automated and early to stick. See Cloud Security.

Documentation and automation as culture

Why: Runbooks, architecture decision records, and onboarding docs are what let a team scale beyond the people who built the system — and what get you through a 3am incident when the expert is asleep. The broader principle: automate the toil. If a human does a repetitive operational task more than a couple of times, it’s a candidate for automation. Common mistake: documentation that’s written once and rots; keep it in version control next to the code, reviewed like code. When not: don’t automate a process you don’t yet understand or that changes every time — automate the stable, repetitive toil first.

How to adopt (without cargo-culting)

You don’t adopt all of this at once, and you shouldn’t. A sane order for most teams:

1. Version control + CI + automated tests — the non-negotiable foundation.
2. IaC + CD — reproducible environments, frequent safe deploys.
3. Observability + on-call + blameless postmortems — see and learn from production.
4. SLOs + progressive delivery — manage reliability deliberately; shrink blast radius.
5. Platform engineering + policy as code — once scale makes cognitive load the bottleneck.

Each layer makes the next safe. Skipping ahead — Kubernetes and microservices before you have CI and observability — is how teams end up with distributed monoliths they can’t operate.

Frequently asked questions

What actually measures whether our DevOps is working? The four DORA metrics: deployment frequency, lead time for changes, change-failure rate, and time to restore service. They capture speed and stability together, so you can’t game one at the other’s expense. Track them over time, not against other companies.

What is an error budget and how do we use it? It’s the allowed unreliability under your SLO (e.g. 0.1% for a 99.9% target). While you’re within budget, prioritize shipping features; when you’ve burned it, prioritize reliability work. It makes the speed-vs-reliability trade-off an explicit, shared decision.

Do we need Kubernetes / microservices to “do DevOps”? No. DevOps is practices (automation, fast feedback, shared ownership, observability), not a specific stack. Plenty of high-performing teams run monoliths on VMs or managed platforms. Adopt Kubernetes/microservices only when a concrete need justifies the operational cost.

What makes a postmortem “blameless,” and why does it matter? It examines the system and process that allowed the failure, not the individual who triggered it. It matters because blame drives problems underground — people stop reporting near-misses — whereas a blameless culture surfaces weaknesses so you can fix them before they recur.

Where should a team just starting out begin? Version control, CI, and automated tests first — everything else builds on reliable, tested, frequently-integrated code. Then IaC and CD, then observability and incident practices. Adopt in an order where each layer makes the next one safe.

Related resources

• Guides: GitHub Actions (CI/CD), System Design (the architecture these practices operate), Cloud Security and Kubernetes Security (shift-left security).
• Guide: DevOps Tools — the toolchain that implements these practices.
• Academy & tools: the free DevOps tools and Docker Skills Academy for hands-on practice.