Kubernetes
Environment Config Overlays for Kubernetes
One config that differs per environment is where drift starts. Base plus overlay keeps dev, staging, and prod honest without duplicating manifests.
Colson
Articles by Colson
Distributed Systems
GDPR Deletion in Event-Driven Systems
Deleting a user across many services is a fan-out problem with a nasty twist: what your deletion event means. Requested or executed changes everything.
Distributed Systems
Kafka Partitions, Retention, and Compaction
Three Kafka settings decide throughput, cost, and whether replay works: partition count, retention, and cleanup policy. How to choose each deliberately.
Distributed Systems
Kafka Topic Naming and Governance
Ad-hoc Kafka topics become an unnavigable mess. A naming convention plus a single canonical registry keeps an event bus legible as it grows past 50 topics.
Distributed Systems
The Transactional Outbox Pattern
Writing to your database and publishing an event are two operations that can't share a transaction. The outbox pattern fixes that, and when to skip it.
Engineering Strategy
Alerts That Don't Wake You at 3 AM
Most on-call pain is bad alerts, not bad systems. Alert on symptoms users feel, page only on what needs a human now, and kill the rest. The rules.
Distributed Systems
Structured Logging at Scale
Plain-text logs are ungreppable at scale. Structured logging makes logs queryable, but volume and cost bite back. How to log structured, useful, and affordable.
Engineering Strategy
SLOs and Error Budgets That Hold Up
An SLO you never enforce is a number on a wiki. How to set SLOs from user experience, turn them into an error budget, and actually act on the burn rate.
Distributed Systems
OpenTelemetry Sampling Strategies
Tracing everything is too expensive; tracing 1% randomly drops the traces you need. OpenTelemetry sampling strategies, head vs tail, and how to choose.
Distributed Systems
Taming Prometheus Cardinality
High-cardinality labels are what kill Prometheus, not query volume. What causes a cardinality explosion, how to spot it, and the label rules that prevent it.
Kubernetes
Do You Actually Need a Service Mesh?
A service mesh gives mTLS, traffic control, and observability, at real complexity cost. When a mesh is worth it, when it isn't, and the lighter alternatives.
Kubernetes
PodDisruptionBudgets That Actually Protect You
A PodDisruptionBudget keeps your service up during node drains and upgrades, but a wrong value blocks drains or protects nothing. How to set PDBs correctly.
Kubernetes
HPA on Queue Depth, Not CPU
For queue-driven workers, CPU-based autoscaling reacts too late. Scale your Kubernetes HPA on queue depth or lag instead. Why CPU lies, and how to switch.
Kubernetes
The Kubernetes Deployment Checklist
A production Kubernetes deployment checklist: resource limits, probes, rollout strategy, PodDisruptionBudgets, graceful shutdown, and the items teams skip.
Kubernetes
Cutting Kubernetes Pod Startup Time
Slow Kubernetes pod startup hurts autoscaling, deploys, and recovery. The five things that make pods slow to start and how to fix each, in priority order.
Engineering Strategy
Microservice Incident Response That Works
In microservices, the hard part of incident response is locating the fault across services. The triage order, the tools, and how to stop cascades fast.
Microservices
Modular Monolith vs Microservices: How to Choose
Modular monolith vs microservices: start with a modular monolith and extract services only when a real force demands it. The decision framework and the signals.
Engineering Strategy
Platform Service Design for Internal Teams
Platform service design means building internal services as products for your own engineers. The golden-path model, the self-service rule, and what to avoid.
Microservices
Shared Libraries in Microservices: The Real Cost
Shared libraries in microservices promise reuse but quietly recouple services through a versioned dependency. When they help, when they hurt, and the rules.
Microservices
Breaking Service Dependency Cycles
Service dependency cycles make microservices impossible to deploy, test, or reason about in isolation. How to detect them and four ways to break them.
Engineering Strategy
Incident Postmortems for Staff Engineers
A blameless incident postmortem fixes the system, not the person. The structure, the root-cause discipline, and the action items that actually get done.
Engineering Strategy
Writing RFCs That Survive Design Review
An RFC that survives design review states the problem, the options, and a justified decision, not just a chosen design. The structure and the failure modes.
System Design
System Design Interviews for Senior Engineers
System design interviews for senior engineers reward judgment, not memorized diagrams. The framework, the tradeoff signals interviewers grade, and the traps.
Distributed Systems
Designing Leaderboards at Scale
Designing leaderboards at scale: why a sorted database query dies under load, how a Redis sorted set fixes it, and how to shard ranking past one node.
Distributed Systems
Notification Fanout Architecture
Notification fanout comes down to fan-out-on-write vs on-read, and the celebrity problem that breaks the naive choice. The hybrid design that scales, explained.
System Design
WebSocket Capacity Planning for Social Products
WebSocket capacity planning for social products: budget memory, file descriptors, and fan-out per connection so you scale before connections, not CPU, break first.
Distributed Systems
Backpressure Design for Real-Time Systems
Backpressure keeps real-time systems alive under load by making producers slow down instead of drowning consumers. Strategies, tradeoffs, and a checklist.
Distributed Systems
Timeout Budgets Across Service Chains
A timeout budget is one deadline split across a service chain so the whole request fails fast instead of piling up doomed work. How to set and propagate it.
Distributed Systems
Idempotency Keys for Distributed Systems
Idempotency keys make retried requests safe so a timed-out payment or duplicate POST applies exactly once. The design, storage, and TTL decisions that matter.
Distributed Systems
Sharding Strategies Before You Need Them
Pick database sharding strategies before traffic forces your hand. Shard keys, hash vs range vs directory, online resharding, and the traps that bite early.
Engineering Strategy
Grafana Dashboards for Operators, Not Executives
Most Grafana dashboards are decoration. An operator dashboard answers one question fast during an incident. How to design dashboards that speed up debugging.
Distributed Systems
Jaeger Tracing for Cross-Service Debugging
Jaeger turns a slow request across many services into one visual trace. How distributed tracing works, what to instrument, and the sampling tradeoff that bites.
Kubernetes
K3s on Hetzner: What to Trust and Isolate
Running K3s on Hetzner is the cheapest path to real Kubernetes, but bare-metal means you own the security boundaries. What to trust, what to isolate, and how.
Engineering Strategy
Bazel Monorepo Builds for Large Systems
Bazel makes large monorepo builds fast through hermetic, cached, incremental builds, but the cost is up-front rigor. When it pays off, and how to adopt it.
Distributed Systems
Kafka Replay Strategy Without Duplicate Events
Replaying a Kafka topic re-delivers events, so duplicates are guaranteed unless consumers are idempotent. The safe replay playbook: dedup, offsets, and isolation.
Kubernetes
Readiness Probes That Don't Lie
Most Kubernetes readiness probes lie: they return 200 because the process started, not because the service can serve. How to write probes that tell the truth.
Kubernetes
Kubernetes Namespace Strategy for SaaS Platforms
Kubernetes namespace strategy for SaaS: namespace-per-tenant vs cluster-per-tenant vs shared. The isolation, cost, and blast-radius tradeoffs, with a decision table.
Microservices
Bounded Contexts in Real Microservice Systems
Bounded contexts are how you find real microservice boundaries: split where the same word means different things. The practical guide, not the DDD theory.
Microservices
Service Ownership Boundaries in Microservices
Service ownership boundaries decide whether microservices scale your org or stall it. Draw them by data ownership and team, not by code size. The rules.
Distributed Systems
Protobuf Schema Evolution Without Breaking Clients
Protobuf schema evolution has clear rules: keep field numbers stable, reserve removals, add not mutate. The safe-vs-breaking change cheat sheet and why.
Distributed Systems
gRPC Across Languages: Production Lessons
gRPC across languages promises neutral RPC. In production the gaps are real: load balancing, deadlines, status codes, and schema evolution. The fixes.
Distributed Systems
Building Rust Hot Path Services in Production
Rust hot path services hold their latency target only if you set 4 defaults right: panic strategy, allocator, Tokio runtime, and bounds. The production checklist.
Microservices
Why Language Boundaries Break Polyglot Microservices
Polyglot microservices break at the seams, not inside services. The 5 cross-language failure modes (deadlines, cancellation, errors, types, connections) and fixes.
Microservices
FastAPI Microservices Architecture: A Production Guide
FastAPI microservices done right: where they belong in a polyglot fleet, the async model that decides performance, and the worker math most teams skip.
Distributed Systems
Go vs Rust for Microservices: When to Choose Which
Go vs Rust for microservices: use Go by default, reach for Rust only on hot paths where tail latency or memory is a measured need. The decision framework.