For more than five years, we built on top of GitHub. CI, automation, deployments, operational workflows — everything you can name. It was a good platform for that. We didn’t have to build another UI, invent another permission model, or teach everyone a different way to run things. Everything used the same language: GitHub Actions workflow YAML.

That consistency made onboarding easier. It also meant more and more of our ability to operate depended on one platform being available.

The question wasn’t just what happens when GitHub goes down. It was what happens when GitHub goes down at the same time we need to act. A hotfix. A deployment. A database schema change or an operational query. We had put all of those behind the same door.

During Q2 and Q3 of 2026, we had GitHub disruptions that paused our ability to ship for roughly two to five hours at a time. The impact on us had been limited. Some happened over the weekend or outside working hours in Indonesia. We were lucky with the timing. That didn’t make the dependency any smaller.

We had already invested too much in the existing system to treat this as a fresh start. Hundreds of services, daily deployments, tooling that people already knew. A migration would introduce its own risk.

So we started with a narrower problem: a break-glass delivery path. Keep GitHub as the everyday platform, but give ourselves another way to execute when it wasn’t available.

We wanted to keep the execution environment, tooling, and infrastructure we maintained. Change the route into that work, not the work itself.

Over a month, we built that route around our existing internal developer platform, Temporal, Gitea, and Kubernetes Jobs.

Owning the runners wasn’t enough#

In an earlier post, I wrote that GitHub had become our backbone, and everything coupled to it. That story was about cost. This one is about availability.

We already managed our own runner fleet. But owning the machines didn’t mean we could give them work independently of GitHub. Self-hosted runners still execute jobs from GitHub Actions. Moving the compute into our infrastructure hadn’t removed that dependency.

When the affected part of GitHub Actions couldn’t dispatch work, having available runners wasn’t enough. Our PR checks — tests, lint, security checks — couldn’t run. Those checks were required, so the normal merge path stopped too.

We had already hit this with a hotfix: the change needed to move, but CI wasn’t running. Even getting the code onto main wouldn’t solve the whole problem. Creating the production artifact also depended on GitHub Actions. Many of our operational workflows and deployments started through workflow_dispatch.

That was the Actions side. Git availability was another dependency. If we couldn’t push or fetch, Argo CD couldn’t retrieve new desired state from the manifest repository either. Running workloads wouldn’t disappear, but our normal route for changing them would be blocked.

We owned the execution capacity. We still depended on GitHub to reach it.

Our workaround was to start a pod and perform the release from there. It worked, but the process was manual and scattered. Running the release was only part of it. We also had to backfill the manifests and create the Git tag against the exact commit, so the result lined up with what the normal workflow would have produced.

That was the painful part. We could perform the work, but we had to reconstruct the workflow around it by hand.

We didn’t need another everyday platform#

We designed the fallback for a GitHub outage that could include both its API and Git service.

Normal development would stay on GitHub. The break-glass path would start from our IDP, and access was restricted to the infrastructure team while the capability was experimental.

We considered a more direct replacement: Gitea as a backup platform, including its CI/CD capabilities. Gitea Actions is mostly compatible with GitHub Actions, which made it worth considering. But compatibility wasn’t the only cost. We would still have two platforms to maintain, two permission setups to keep aligned, and another operational surface alongside our existing GitHub Enterprise setup.

A different CI system had a similar problem. GitHub was also where people went to run operational work, so replacing it meant teaching them another system. Extending the IDP let us keep the tooling people knew and give the platform a way to execute work.

We kept the initial scope to three operations: create a release, deploy it, and roll back. Rollback used the same deployment path, just targeting a specific previous version. Database operations were part of the wider dependency problem, not part of this initial scope.

The useful work already existed#

Mason, our internal CLI, gave us a useful starting point.

In The floor, not the ceiling, I wrote about Mason becoming the interface for building, testing, packaging, and deploying across the monorepo. It takes service definitions — JSON files describing how a service is built and configured to run — and performs the work.

Our GitHub composite actions invoked Mason. The break-glass path could invoke those same commands, so the build and deployment logic could stay where it was.

Mason could also run locally, but some production dependencies were only accessible inside our internal network. It needed the toolchains baked into our runner images too. We reused those images and the node pools provisioned for GitHub Actions runners, including the environment I covered in the runner caching post.

The IDP began as a service catalog: ingest the service definitions from the repository into PostgreSQL, then expose services and ownership. Later, we added release and deployment tracking through webhooks from GitHub. We could put the new operations on those service pages.

Not everything was portable just because the commands were. Our Go monorepo depended on shared modules and a separate API contract repository. Our deployment process pushed manifests into a GitHub repository for Argo CD to consume. Those dependencies still needed an answer.

We mirrored the monorepo, its dependency repositories, and the Kubernetes manifest repository into Gitea.

A different route to the same execution#

The IDP accepted requests to create a release, deploy, or roll back. Temporal coordinated the work, and Kubernetes Jobs ran the existing tooling. Gitea supplied the mirrored repositories; GitHub synchronization happened separately.

diagram source — renderer unavailable
flowchart TD
    accTitle: The break-glass delivery path
    accDescr: An infra operator requests a release, deploy or rollback from the IDP service page. Temporal runs Kubernetes Jobs on the existing node pools, which take their image from ECR, check out from the Gitea mirrors, and run Mason to build and deploy. Mason writes release artifacts to object storage and reads them back, then applies them to the target cluster through Kustomize. GitHub mirrors into Gitea in the background, and deferred work backfills tags and releases to GitHub afterwards.
    operator["Infra operator"] --> idp["IDP service page<br/>Access controls"]:::accent
    idp -->|"Release · deploy · roll back"| temporal["Temporal<br/>workflows"]:::accent
    temporal -->|"Kubernetes Jobs on<br/>existing node pools"| preflight["Bash preflight<br/>Clone and set up"]:::accent
    preflight --> mason["Mason<br/>Build and deploy"]:::accent

    gitea["Gitea mirrors<br/>Source, deps,<br/>manifests"]:::store -->|"Checkout"| preflight
    ecr["ECR<br/>Runner images"]:::store -->|"Job image"| preflight
    mason <-->|"Release artifacts<br/>written, then read"| storage["Object storage"]:::store
    mason -->|"Kustomize apply"| cluster["Target cluster"]

    github["GitHub<br/>Primary repos"]:::external -.->|"Background mirroring"| gitea
    temporal -.->|"Deferred work"| backfill["Backfill to GitHub<br/>Tags and releases"]:::deferred

The solid connections cover immediate execution and its inputs. The dotted connections cover background mirroring and deferred GitHub updates. The scheduled mirror synchronization also ran through Temporal, alongside Gitea’s built-in mirroring.

Creating a release#

An operator opens the service page in the IDP and requests a release. They can provide a specific commit SHA from trunk. If they leave it empty, we resolve the latest commit available on the Gitea mirror’s trunk and pass that SHA to both architecture builds.

Each build runs on the matching node architecture. We stitch the resulting image digests into one multi-architecture image reference, then generate secrets and Kubernetes manifests and upload the release artifacts to object storage.

diagram source — renderer unavailable
flowchart TD
    accTitle: Creating a release
    accDescr: One commit SHA from the Gitea trunk feeds an AMD64 build and an ARM64 build. Their image digests are stitched into a single image reference, secrets and manifests are generated, and the release artifacts are uploaded to object storage. The Git tag and GitHub Release are backfilled when GitHub returns.
    sha["One commit SHA<br/>from Gitea trunk"] --> amd["AMD64 build<br/>Mason on AMD64"]:::accent
    sha --> arm["ARM64 build<br/>Mason on ARM64"]:::accent
    amd --> stitch["Stitch digests<br/>One image ref"]:::accent
    arm --> stitch
    stitch --> artifacts["Generate secrets<br/>and manifests"]:::accent
    artifacts -->|"Upload"| storage["Object storage<br/>Release artifacts"]:::store
    artifacts -.->|"When GitHub returns"| backfill["Backfill Git tag<br/>and GitHub Release"]:::deferred

The IDP showed progress as a timeline. Operators could also open the Temporal UI to inspect workflow progress. Raw job logs stayed in Kubernetes; we hadn’t streamed them into either interface. Instead, we wrapped failures in human-readable errors, so an engineer didn’t need to be familiar with Kubernetes just to understand what had gone wrong.

Gitea remained a mirror. A separate backfill activity created the Git tag and GitHub Release once GitHub was available, with immutability controls enforced there. Deployment could proceed before that backfill completed.

Why Temporal here#

We used Temporal for business workflows elsewhere in the company and had experience operating it. Here, we needed it to coordinate the Kubernetes Jobs and track the work that would outlive the original request.

Backfill was a concrete example. A deployment could finish while GitHub was unavailable, leaving repository updates to complete afterward. Temporal coordinated that deferred work and its retries. Otherwise, we would be back to the manual checklist from the pod-based workaround.

Deploy now, reconcile afterward#

Normally, deploying meant pushing manifests to GitHub and letting Argo CD apply them. That couldn’t be our fallback too. If we still needed a successful GitHub push, we hadn’t removed the blocker. We’d just moved it to the end of the workflow.

The break-glass path checked the target cluster’s API server, then disabled auto-sync for the application. It combined the release manifests from object storage with the mirrored manifest repository and applied them through Kustomize.

The repository contained overlays and patches that weren’t in the generated release artifacts. Applying those artifacts alone would miss part of the configuration. That’s why we needed the manifest repository mirrored too.

After a successful deployment health check, the workflow started reconciliation. The reconciler pushed the manifest changes back to GitHub. Once the repository reflected the deployed state, auto-sync could be re-enabled.

diagram source — renderer unavailable
flowchart TD
    accTitle: Deploying, then reconciling back to GitHub
    accDescr: The workflow checks the target cluster, disables auto-sync, combines the release artifacts from object storage with the overlays from the mirrored manifest repository, and applies them through Kustomize. A healthy deployment starts reconciliation, which pushes the manifests once GitHub is reachable and then re-enables auto-sync. An unhealthy one alerts, re-enables auto-sync and leaves Argo CD to self-heal, which was never verified in a full outage.
    start["Check cluster<br/>Disable auto-sync"]:::accent --> prepare["Combine manifests<br/>with overlays"]:::accent
    gitea["Gitea mirror<br/>Manifest repo"]:::store --> prepare
    storage["Object storage<br/>Release artifacts"]:::store --> prepare
    prepare --> apply["Apply through<br/>Kustomize"]:::accent
    apply --> health{"Deployment<br/>healthy?"}
    health -->|"Yes"| recon["Start reconciliation"]
    recon -.->|"When GitHub returns"| push["Push manifests<br/>to GitHub"]:::deferred
    push --> restore["Git matches cluster<br/>Auto-sync back on"]
    health -->|"No"| alert["Alert<br/>Auto-sync back on"]
    alert --> recovery["Argo CD self-heals<br/>Not verified in<br/>a full outage"]

If health checks failed, we alerted and re-enabled auto-sync. Our applications had self-healing configured. Disabling automated sync suspended that behavior too; re-enabling it let Argo CD reconcile live changes back toward the previous desired state.

That recovery path still depended on Argo CD being able to obtain the previous desired manifests. We hadn’t verified that behavior during a complete GitHub outage, so I wouldn’t call it a guaranteed rollback under those conditions.

What stayed inside the job#

A Bash script injected into each Kubernetes Job handled preflight: clone the source from Gitea, prepare the workspace, then invoke the Mason command for that stage. It served the same purpose as the checkout step in our GitHub workflow. Once the source was ready, Mason did the work.

A fallback can inherit the same failure#

Moving the trigger out of GitHub was only part of the work. We also had to account for what the job fetched, how it authenticated, and which revision was actually available.

Source availability#

We kept Gitea’s built-in mirroring enabled, but observed lag and added a scheduled Temporal workflow to trigger synchronization every minute. The five-minute drift threshold was for alerting; it didn’t block releases. These were separate from the post-deployment reconciler that wrote manifests back to GitHub.

Once GitHub became unreachable, we could only use revisions that had reached the mirror. Creating a new fix during the outage was out of scope. The supported operations were releasing mirrored code, deploying it, and rolling back to an existing version.

Identity and credentials#

Our IDP had RBAC built by my peers and token-based authentication for machine access. People signed in through company SSO, independently of GitHub.

We also ingested GitHub data into PostgreSQL through an ETL pipeline. When live API calls weren’t available, the platform could use that stored data. It could be stale, but the service pages didn’t need a successful GitHub API call to remain useful.

The jobs used Kubernetes service accounts linked to IAM roles rather than static credentials. Build and deployment jobs had separate service accounts, with permissions scoped to the work each needed to perform. The break-glass path didn’t need to retrieve credentials from GitHub to run.

Everything the job downloads#

For Go dependencies, we had Athens as our module proxy and a seeded GOMODCACHE in the pods, populated from the modules our monorepo used. A warm cache avoided downloads. On a miss, modules using the proxy went through Athens.

Private modules covered by GOPRIVATE used a separate path. We mirrored those repositories into Gitea too. As I remember it, private repository fetches were redirected to the mirror; I no longer recall the exact rewrite configuration.

A module missing from the pod cache still needed to be available through Athens or the private mirrors. Athens could itself need an upstream fetch for something it hadn’t cached.

The runner images lived in ECR, and generated release artifacts lived in object storage. Neither depended on GitHub for storage or retrieval.

The failure boundary we chose#

This path shared our execution infrastructure. It still needed the IDP, its database, Temporal, Gitea, the clusters, and artifact storage to be healthy. It wasn’t a separate disaster-recovery environment.

Access was restricted to infrastructure operators. We didn’t rerun tests and security checks during the break-glass release: with trunk-based development, those checks ran before code reached main. We used the same assumption as the normal release path — a trunk revision had passed the required checks. Scheduled regression runs existed too, though they weren’t release gates.

What worked, and what it opened up#

We successfully created releases, deployed them, and rolled back through the platform. GitHub was reachable during those runs but unused by the immediate execution path; its updates were handled through backfill. We hadn’t run a full outage drill with GitHub blocked or unavailable.

For me, the experience made this worth considering beyond emergencies. An operator could open a service page, request a release, and follow its progress there. That felt better than leaving the platform to find and dispatch a GitHub workflow. It could become the everyday deployment path, although we hadn’t made that transition.

The month of work built on Mason, the runner environment, and the IDP. We still took on maintenance: operating Gitea, watching mirror drift, maintaining workflows, and dealing with reconciliation failures. Self-hosting the mirror meant owning its freshness too.

I saw the orchestration work as an investment in future self-service operations. We now had a way to execute work from the service catalog, using tools we knew how to operate.

By the time I left, we hadn’t established a routine for exercising the fallback. Keeping it ready as the rest of the system changed was still work to do.