Building a Kubernetes operator for S3

When I advised internal teams as an SRE consultant, I saw the same pattern again and again. Most outages started with a manual step.

One of them began with a firewall rule. Somebody fixed it by hand and forgot to update the code. The next deployment brought the old rule back. The connections dropped again.

So we tried building a Kubernetes operator for the firewall rules. But the workflow had manual approval stages. We had to feed every approval back into the events of Kubernetes. An approval belongs in the code review, before the operator runs. That workflow did not let us build it that way.

Storage was worse. Most operations were CLI commands inside a pod. Some of them did harm when they ran twice. And Kubernetes never knew the state of the storage.

Houston is my answer for ObjectGateway. Teams write their storage as code, and their review process approves it. Houston keeps all the components in their desired and expected state. To avoid the 3 a.m. accident, the ogw CLI and the console refuse changes that are managed as code.

First, a prototype

The first version of Houston was a prototype: about 39,000 lines of code, 18 custom resources, and 5 processes and Jobs. It was built ahead for features that Core did not have yet, and that made it bloated. The features that waited for missing APIs were never tested.

So the rebuild starts from what Core can do today.

Where the prototype went wrong

The prototype used one key for everything. Even the requests of teams ran with the key of the system account. The operator had to keep the teams apart to control their permissions.

Getting keys to the teams was the hardest part. Each key was encrypted and passed through several helper programs. When one step broke, it was hard to repair.

It was fragile in daily work too. Deleting a namespace often got stuck, and someone had to clean it up by hand. A webhook checked every change before Kubernetes accepted it. When the webhook was down, nobody could change anything.

What changed

The new Houston keeps what worked and drops the rest.

• The gateway keeps the teams apart, not the operator. Each namespace has its own service account in its own account. The requests of one team never reach the account of another.
• A key goes straight into a Kubernetes Secret, and Core rotates it. Applications read it with the standard AWS variables and need no change.
• Deletion keeps the data by default. A parent object waits until its children are deleted.
• The schema checks every field automatically when you apply it. This also works while Houston is down.
• Houston has no S3 client, and its own keys cannot read or write objects. Versioning and lifecycle rules stay in S3, with the tools that teams already use.

A Kubernetes operator works best with one source of truth. The state should not live in two places. Without a guard, the CLI and the console could change the same resources. So I added a field to the resources of ObjectGateway that names who manages them. The CLI and the console read it and leave those resources alone.

Since Core supports OIDC, teams can set up their identity providers directly in Houston too.

Simpler to run

I simplified the architecture of the prototype. Houston now has half the code. It needs 9 custom resources instead of 18. And it runs as 1 process instead of 5 processes and Jobs.

For you, that means fewer moving parts. There is one process to watch and one log to read. When something goes wrong, there is only one place to look.

The landing page shows what the resources of Houston look like.