Four years of CI, two market cycles, and the thing we still haven’t solved

The pattern

There’s a conversation that happens in every cycle. Different quarter, different context, same shape. Something is expensive or broken or inefficient. Everyone senses it. The urgency peaks, then dissolves. The moment passes and we move on. Next cycle, the conversation returns — same contours, different numbers.

After enough cycles you stop being surprised by the pattern and start asking a different question: what would it look like to get ahead of it instead of reacting to it?

CI Insights started as our answer to that question.

The invisible burden

We were early GitHub Actions adopters. Monorepo, self-hosted runners, autoscalers, custom VM pools built from scratch. GitHub became our backbone, and everything coupled to it. At some point the cost of that backbone became our problem — the platform team’s, the SRE’s — while the engineers who generated the cost opened PRs, watched the light turn green, and merged.

The feedback loop was clean. The cost was invisible. Nobody above us was asking. In bull market, we could pay for everything, so we did. Speed mattered. Quality mattered. Cost was someone else’s line item.

That someone else was always us.

Building before anyone asked

The first version came together during a stable period — not bull, not bear, just a window with enough space to think. We were orchestrating test runners with goroutines as a multiplexer, and we wanted visibility into which tests were the real bottlenecks, where the waste was hiding. We used gotestsum to surface trends. Rough, but it worked.

Nobody put it on a roadmap. We built it because we’d seen the cycle enough times to know that when the moment came — when someone above us finally cared about the numbers — we’d need data we didn’t have yet. Better to have the foundation before the urgency, not scramble for it during.

We rebuilt it properly in February. The intent was narrow: track SLA for our runner fleet, because GitHub ARC gave us nothing usable. That’s why we built our own ETL. Once we held the data, everything else became possible — cost attribution, SCAD integration, per-team visibility. The tool grew into what the data allowed.

When the moment arrived

Transaction volume started falling in January. Roughly halving each month. By March and April, cost was the conversation in every room.

We found out CI was 25 to 30 percent of our entire cloud spend. In bull, that number made sense. In bear, it was the first thing anyone wanted to explain.

This was the moment the tool became political. Not because it improved — it had been sitting there, quietly useful, for months. The org was finally in enough pain to look at what we’d built.

We demoed it. The response was what we hoped for: this could be the foundation of an internal developer platform, a way to answer questions with data instead of gut feel. Convincing. Real. And then — the wind wasn’t going that direction. The moment was real but the organizational will to act on it fully wasn’t there yet.

So the tool exists. The data exists. And we wait for the conditions that turn insight into policy.

What visibility actually does — and doesn’t

Some things changed. Engineers who already cared started testing locally before opening PRs. Wasted runs that used to vanish into a cloud bill now had names and owners. The people who were already good became more intentional about their footprint.

But visibility only works on people who are already looking. The engineers who didn’t care before saw the numbers too. They just didn’t feel compelled to act on them — because there was no consequence for looking away.

Visibility shifts ownership for people who already care. To shift it for everyone, the cost has to land somewhere it hurts. That’s not a technical problem. That’s a political one.

Real enforcement means budget attribution per team. Quotas. Someone with authority saying: your team spent X this sprint, and that number is yours to own. That requires organizational will we can surface but not create. And organizational will follows the market, not the merit of the idea.

We built the mirror. We can’t make anyone look into it. That part was never ours to do.

What four years actually taught us

Good ideas don’t ship on merit. They ship when the moment is ready for them. We had the insight early. We built the tool when the pressure arrived. We’re still waiting for the enforcement that would make it complete.

That’s not a failure. That’s just what it looks like to work inside an organization — building things that are right before they’re politically possible, and learning to hold both without collapsing one into the other.

We built the mirror. We can’t make anyone look into it. That part was never ours to do.

But we’ll be ready when it is.