Braxton J. Schafer

The problem. Source control is the one piece of infrastructure no engineer can route around. Cloudflare’s ran on Bitbucket, and replacing it company-wide meant changing the daily workflow of every engineer in the company at once, with no option to fix it in a later release.

Constraints. A net-new deployment rather than a lift-and-shift, so nothing inherited the old system’s operational shape. It had to survive the load of the entire engineering organization and reach parity with a system people had years of muscle memory for, all while the old one stayed up.

What I did. I architect and lead the build: GitLab deployed on our Kubernetes from the ground up, operated like the rest of the platform: declarative configuration, GitOps delivery, real observability, and rollouts that halt on a health signal instead of a phone call.

Where it stands. Done. It’s the source control system, full stop, and every engineer at the company is on it every day.

The problem. GitLab’s stock runner manager is a process that reaches out and creates job environments. On Kubernetes that’s a poor fit: it holds state the cluster already knows how to hold, it fails in ways the cluster can’t see, and it doesn’t reconcile: when reality drifts from intent, nothing walks it back.

The design. A ground-up reimplementation in Go as a Kubernetes controller. Job capacity becomes a declared resource; the controller’s job is to make the world match the declaration and keep it matched. Reconciliation turns a recurring class of “the manager got confused, restart it” incidents into a control loop that corrects itself, and it means the platform’s own primitives (scheduling, health, eviction, autoscaling) apply to CI capacity instead of being duplicated inside a sidecar process.

The tradeoff. Owning a fork of critical-path infrastructure is a real cost: upstream keeps moving, and we carry the maintenance. It’s defensible because the seam is narrow, and the alternative, permanently working around a component that can’t express what we need, costs more over time.

The problem. Every CI platform accumulates the same failure mode: a red pipeline means either the platform broke or someone’s code broke, the dashboard can’t tell you which, and the on-call learns to ignore the signal. Once that happens the platform has no feedback loop, and regressions get found by annoyed engineers instead of by us.

What I built. The observability layer for the new CI platform:

• Per-project and per-component pipeline metrics. A regression is attributed to the component that caused it instead of averaged into a platform-wide number that never moves.
• Queue-depth and error dashboards. Capacity starvation and correctness failures look identical from a single job’s perspective and have to be separated at the fleet level.
• Alerting that distinguishes platform failure from user-caused job failure. An alert that fires on someone’s failing test suite is worse than no alert, because it trains people to ignore the channel.
• Health-gated rollouts of CI component changes. A bad shared component gets stopped by a health signal instead of by the volume of complaints.

Why it mattered. The dashboards weren’t the point. The point is that the team can now change shared CI components at all: you can’t roll out to every pipeline in the company without a signal telling you when you’re breaking them.

The problem. Kubernetes YAML doesn’t compose. Run the same workload across several clusters and you end up with several near-identical manifest trees that drift apart one urgent fix at a time, until nobody can say what’s actually deployed where. Templating engines mostly move the problem into a language that’s harder to debug.

The approach. Drove CDK8S adoption across engineering as the standard for codifying Kubernetes infrastructure: manifests as real, typed, testable code, where shared behavior is a construct rather than a copied block, and a change lands in one place.

The hard part. Almost none of it was technical. Adoption isn’t a rollout; it’s a series of individual teams deciding the switch is worth it, so the standard has to be good enough that people choose it on their own, and the migration incremental enough that nobody has to stop shipping to do it. Most of the effort went into making the default easy, not into the abstractions.

Outcome. CDK8S is the standard for codifying Kubernetes infrastructure across engineering, and cross-cluster manifest duplication is gone.

The problem. AI coding tools showed up as a per-engineer thing: everyone experimenting privately, nobody’s setup portable, and useful techniques stuck in whichever harness the person who found them happened to use. The configuration (skills, agents, commands) lived in scattered dotfiles and got rewritten from scratch every time someone tried a different tool.

What I built. A tool that manages and syncs skills, agents, and commands from multiple sources into any AI harness. Harness independence is the design constraint: tying the shared configuration to one vendor’s format would defeat the point. Sources compose, so a team-wide baseline and a personal layer coexist without copying either.

Adoption. Tooling alone doesn’t move a team. I write internal posts on the techniques that hold up in practice, so they spread instead of staying with whoever found them.

The symptom. Our Atlantis instance took about 30 minutes to restart. Restarts happen constantly (credential rotations, onboarding), roughly 100 times a month. Everyone had absorbed the delay as the cost of doing business and scheduled around it.

The root cause. Not Atlantis. By default, Kubernetes recursively chowns every file on a volume to match the pod’s fsGroup on each mount; on a large persistent volume, that walk is the entire startup time. fsGroupChangePolicy has existed since Kubernetes 1.20 and takes a second value, OnRootMismatch, which checks the root directory’s ownership and skips the walk when it already matches.

The fix. One line in the pod spec. Restarts went from ~30 minutes to ~30 seconds. At roughly 100 restarts a month that was over 50 hours of blocked engineering time every month: a little over 600 engineer-hours a year, returned by a single field.

Why it’s worth remembering. The fix was trivial, and that’s the point: the work was noticing that a 30-minute restart had been accepted as normal, not knowing the flag. I wrote it up so the next person wouldn’t have to rediscover it.