GlusterFS

Helsinki is clearly (and logically) having higher latency to the German nodes. This adds up enormously in time.

We are using 6 virtual nodes at Hetzner, all the cheapest variant: 2 GB RAM, 20GB NVMe SSD disk, 1 Skylake IBS CPU core. All are on gbit. GFS 1+2+3 are all in the same location (Nuremberg). The rest speak for themselves.

Copies. If the amount of copies equals the amount of servers, all servers have a local copy of the data. Then no read latency occurs, as everything can be served locally. Additionally, all other nodes in the gluster can go down and data can still be read. However, if there are less copies than servers, read latency occurs as data needs to be fetched over the network with every read, and all servers must be alive for it to function.

This test was simple: Extract WordPress:

cd ~
wget https://wordpress.org/latest.tar.gz
tar -zxf latest.tar.gz -C /path/to/volume/mount

This extracted/created 193 directories, filled with a total of 1835 files. A total of 2028 objects. 42695316 bytes total, 21052.92 bytes per file on average. Or 20.6 KiB/object. Negligible sizes (and big overhead).

Vol Set-up Highest Lat. 1 Replica Full replicas Difference Per object (1) Per object (all)
gv0 Nuremberg 0.060 ms 5s 5s 0s 2.47 ms 2.47 ms
gv1 Nuremberg + Falkenstein 3.635 ms 39s 97s +58s 19.23 ms 47.83 ms
gv2 Nuremberg + Helsinki 25.441 ms 227s 595s +368s 111.93 ms 293.39 ms
gv3 Nuremberg + Falkenstein + Helsinki 26.262 ms 179s 597s +418s 88.26 ms 294.38 ms
gv4 GFS 1 0.159 ms 5s 5s 0s 2.47 ms 2.47 ms
gv5 GFS 1 + GFS 2 0.995 ms 14s 34s +20s 6.90 ms 16.77 ms
gv6 GFS 1 + GFS 3 1.128 ms 12s 31s +19s 5.92 ms 15.29 ms
gv6 GFS 1 + GFS 2 + GFS 3 1.128 ms 15s 34s +19s 7.40 ms 16.77 ms

Extracting WordPress locally on a tmpfs-disk takes 0.6s. Let's say 1 second. Rounding decimals to full seconds.