Braxton J. Schafer

Epic used a Subversion monorepo for all shipped code. The company had built a ton of custom tooling and systems around it and largely abstracted it away from developers.

I led the migration effort to not only convert to Git, but to stand up a brand-new cloud-native hybrid GitLab cluster scaled to support 5,000 active developers.

Cloudflare was using a singleton Bitbucket instance that couldn’t scale or be resilient. We needed something that could be resilient, highly available, and support our swift engineering organization.

I led the buildout of GitLab, fully on Kubernetes, distributed across three primary sites and built for scale.

The problem: GitLab’s stock runner manager was designed back in the day when your CI runner was one big server running all the time. While it doesn’t not support running jobs as Kubernetes pods, it struggles. The runner manager is one stateful app, and it fails in a wide variety of fun ways.

The design. A ground-up reimplementation in Go as a Kubernetes controller. Job capacity is handled as a first-class constraint. The reconciler resolves the legacy problem of “the manager got confused, all jobs are orphaned”. Controller restarts don’t interrupt running jobs; all jobs talk directly to GitLab now.

The problem. Red pipelines are an inevitability, but telling at a glance what happened (or if there’s a bigger problem) shouldn’t have to be. If the platform doesn’t have a good feedback loop, that means that infrastructure regressions get found by annoyed engineers when things break, instead of by us.

What I built. The observability layer for the new CI platform:

• Per-project and per-component pipeline metrics. A regression is attributed to the component that caused it instead of averaged into a platform-wide number that never moves.
• Queue-depth and error dashboards. Capacity starvation and correctness failures look identical from a single job’s perspective and have to be separated at the fleet level.
• Alerting that distinguishes platform failure from user-caused job failure. An alert that fires on someone’s failing test suite is worse than no alert, because it trains people to ignore the channel.
• Health-gated rollouts of CI component changes. A bad shared component gets stopped by a health signal instead of by the volume of complaints.

Why it matters. Pretty dashboards aren’t the point. What matters is that the team can now change shared CI components at all: you can’t roll out to every pipeline in the company without a signal telling you when you’re breaking them.

The problem. Kubernetes YAML doesn’t compose. Run the same workload across several clusters and you end up with several near-identical manifest trees that drift apart one urgent fix at a time, until nobody can say what’s actually deployed where. Templating engines mostly move the problem into a language that’s harder to debug. Plus, who likes dealing with whitespace issues?

The approach. Drove CDK8S adoption across engineering as the standard for codifying Kubernetes infrastructure: manifests as real, typed, testable code, where shared behavior is a construct rather than a copied block, and a change lands in one place.

The hard part. Almost none of it was technical. Adoption isn’t a rollout; it’s a series of individual teams deciding the switch is worth it, so the standard has to be good enough that people choose it on their own, and the migration incremental enough that nobody has to stop shipping to do it. Most of the effort went into making the default easy, not into the abstractions.

Outcome. CDK8S is the standard for codifying Kubernetes infrastructure across engineering, and cross-cluster manifest duplication is gone.

AI tools took the world by storm, but they came on so suddenly that there was no cohesive design or usage for them. I noticed two main problems:

1. Skills, agents, and config accumulates on each person’s machine, and upstream skills ended up stale.
2. The moat between per-engineer usage and techniques was massive.

I solved the first problem by building tooling to support syncing AI configs from any Git source, and keeping them up to date. As we allowed different harnesses, it added support to keep everything in sync between any harness an engineer chose to use.

The second problem involves fostering a sharing culture. With the tooling to support sharing skills, we can then share mindset. I do this both asynchronously via frequent internal blogposts sharing tips and tricks, and synchronous sessions letting engineers share their AI-related wins and get input on their struggles.

The symptom. Our Atlantis instance took about 30 minutes to restart. Restarts happen constantly (credential rotations, onboarding), roughly 100 times a month. Everyone had absorbed the delay as the cost of doing business and scheduled around it.

The root cause. Not Atlantis-related at all. By default, Kubernetes recursively chowns every file on a volume to match the pod’s fsGroup on each mount; on a large persistent volume, that walk causes the entire startup time. fsGroupChangePolicy has existed since Kubernetes 1.20 and takes a second value, OnRootMismatch, which checks just the root directory’s ownership and skips the walk when it already matches.

The fix. One line in the pod spec. Restarts went from ~30 minutes to ~30 seconds. At roughly 100 restarts a month that was over 50 hours of blocked engineering time every month: a little over 600 engineer-hours a year, returned by a single field.

Why it’s worth remembering. The fix was trivial, but deciding to investigate the problem is the interesting bit: noticing that a 30-minute restart had been accepted as normal, not knowing the flag. I wrote it up so the next person wouldn’t have to rediscover it.