Why does nobody build S3 like this?

I designed the storage cluster for a friend’s company. They had about 3 PB of video on a cloud account, and its provider was about to close it. So we built the cluster in their own data center, with brand-new drives and everything that I had learned at work.

Then the provider closed the account, and there was no way back. Every file now had to live on the new cluster. And it could not keep up.

For weeks, I tried everything that I knew. Consultants and experts from my network tried too. Every change looked good on paper and did nothing in practice. The disks stayed busy, and the latency stayed high.

When something is slow, I find out why. This time I could not do anything about it.

The worst part was that I had seen this before.

It was not the first time

In my day job, I was a site reliability engineer for a data lake of several petabytes at a large telecom company. One week, a batch job that usually ran for 6 hours ran for 18. Every step of the pipeline after it waited.

A team had just moved its workload from Hadoop to our new S3 storage. Hadoop ran the work next to the data. Our new storage sent every byte over the network, several times.

Each fix showed us the next problem. More gateways made the jobs faster, until the gateways stopped answering, because their own cleanup kept them busy. Then we found the metadata on the same slow hard disks as the data. To move it, we had to change the layout of the cluster, and each change started days of rebalancing.

Several consulting companies had approved that design. We had a whole team that did nothing but storage, and it still took us weeks. How was a small company supposed to manage that?

What were we doing wrong?

Nobody could tell me, and I could not let it go. So I started to build my own prototypes, and I trashed each one. They were too complex or too slow, or I was not able to test them well enough. And almost everywhere I looked, I found replication, erasure coding, and many moving parts again. Exactly what had caused me so much trouble in production.

Then I asked a different question: what is the simplest design that can deliver S3? Between the HTTP request and the disk, there is only the gateway. Buckets behave like directories, and objects behave like files.

It was the first prototype that worked. And then the doubt started. If it is this simple, why did nobody smarter than me build it? Why does nobody build S3 like this? I still ask myself that today.

I talked with many people in my network who run storage. Their answer was simple. S3 is boring, and nobody cares about it until it stops working.

Everything depends on the metadata

Both clusters failed because of the metadata. So in ObjectGateway, my own S3 storage, it belongs to the file. It sits in an extended attribute or in a small file next to it. As long as the filesystem keeps its promises, data and metadata cannot drift apart.

The gateway never makes copies of data on its own. That job belongs to the storage below it, or to Orbit, a separate product. The catch is that one gateway runs on one machine, and the filesystem has to pass our tests. In return, running it is as simple as running the filesystem underneath.

Today, ObjectGateway is a product of Quadrant Systems, the small software company in Switzerland that I founded. We run it ourselves, among other things to host this website. It is the storage that I was missing back then: S3 that stays simple and runs without a storage expert. And that hopefully stops waking people up at night.

If you want to follow where this goes, subscribe to the product updates.