ObjectGateway
How fast is S3 from where your app runs?
By Marc Singer
Updated 7 min read
In my last post, two storage clusters could not keep up. The storage was too far from the work and too slow for its own metadata. When I came up with the architecture of ObjectGateway, performance became one of three things it has to get right, along with staying up and speaking S3 the way your software expects it.
So, how fast is ObjectGateway then? The gateway is only half of the answer. The other half is where your workload runs. The same gateway answers a small read in 2 ms from the same data center and in 20 ms from across a border. A batch job that reads 10,000 small objects one after the other finishes in 20 seconds next to the storage and needs more than 3 minutes from the other country. The underlying storage is the same in both cases.
The benchmark
Leo (@leodev) published storage-bench. The benchmark measures what applications do every day. I chose it because he used it to build his benchmark of public S3 offerings.
His version only knows the providers of his comparison. I adapted storage-bench so it takes any S3 service. Then I ran it from four cloud zones in Frankfurt, Munich, Geneva and Vienna against our two gateways in Bern and Nuremberg. Both sites run our current stable release v0.7.4. The raw results are online.
What distance costs
How long a typical download of a small 1 KB object takes, by where the workload runs:
Show as a table
| Workload | Bern | Network | Nuremberg | Network |
|---|---|---|---|---|
| Same machine | 1.8 ms | 0.2 ms | 1.8 ms | 0.3 ms |
| Same data center | 2.2 ms | 1.1 ms | 3.0 ms | 1.4 ms |
| Frankfurt | 9.4 ms | 6.5 ms | 5.4 ms | 3.8 ms |
| Munich | 13.3 ms | 10.3 ms | 11.4 ms | 9.2 ms |
| Geneva | 8.3 ms | 4.1 ms | 19.4 ms | 16.2 ms |
| Vienna | 20.0 ms | 16.2 ms | 16.1 ms | 13.2 ms |
From the same data center to a neighboring country, every single request for a small object becomes about ten times slower.
I wanted to know how much of that time is really the gateway. So I took the network out of the picture. At both sites I measured how long the network alone takes, on the same machine as the gateway. Both times, the gateway and the client needed about 1.5 ms on top of it. From the cloud zones it was a little more, depending on the client machine. Everything else in the chart is distance.
Beyond the gateway itself
The metadata from my last post shows up here too. In Bern the gateway runs in a virtual machine and reaches its ZFS disks through an extra layer. Small reads and writes do not notice. On the same machine, a small read takes 1.8 ms at both sites. Listing 1,000 objects takes 77 ms in Bern and only 25 ms in Nuremberg. Without any network in between, that gap comes from the storage.
Every layer between the gateway and the disks costs performance. But I run the gateway in Bern like this on purpose. On our own sites I try the setups that customers might run, supported or not. That way I learn what a setup costs and where to optimize it before a customer runs into it.
In Nuremberg the benchmark tripped over a filter in the network in front of the gateway, most likely the DDoS protection of my hosting provider. The benchmark starts 32 uploads at the same moment, and the first time it did, a few of them never arrived. The gateway itself shows no errors. From another server in the same data center, every burst got through. When too many new connections arrive at once, the filter only lets part of them through. Since applications reuse their connections by default, they usually never notice.
At 100 MB the test itself hits its limit. From Frankfurt both sites deliver about 160 MB/s, exactly as much as the network port of the small test machine allows. On the same machine, the gateways deliver 360 to 410 MB/s.
Run it next to your workload
I designed ObjectGateway to be one gateway on the POSIX filesystems you already run and more importantly right where your workload runs. That can be your own data center or a European cloud provider. It needs about 100 MiB of memory at about 1,000 requests per second across one million objects, so it easily fits beside the application. Running your storage in another country adds that distance to every single request.
Our numbers can only show what our own sites do. The adapted benchmark shows how fast S3 is for your applications in your own environment. It works with any S3 service. If you try it, feel free to let me know your results.
As I deploy new sites with new deployment models, this post will grow with more numbers so you get a better picture of what to expect. Subscribe to the product updates to follow along.