Disk Latency Testing with fio for Game Save Patterns

A vendor spec sheet says 6,500 MB/s sequential and 1,000k random IOPS. Your Palworld server still freezes for two seconds every fifteen minutes when it autosaves. The two facts are unrelated, because game save I/O is neither sequential nor random-at-depth: it is a burst of 8–64 KB writes to a handful of files, issued by one thread at queue depth 1, followed by an fsync. Benchmark that exact shape or the number you get back tells you nothing.

A fio methodology that reproduces the save patterns of Minecraft, Palworld and Rust, the per-class values to expect, and the thresholds that justify new hardware rather than a config change. Test on the hardware you actually deploy — figures here come from build validation on our dedicated server builds.

What save I/O actually looks like

Capture it before benchmarking it. Three games, three shapes: Minecraft writes annotated region files in 4 KB-aligned chunks after a defrag pass; Palworld writes a compressed UE save to a single .sav file of 200–800 MB; Rust writes small bincode snapshots per entity group plus a larger world file. All three call fsync when they finish, and all three block the tick loop while doing it.

Game	File pattern	Write size	Burst size	Queue depth	Frequency
Minecraft (Paper)	N region files, interleaved	4–32 KB	80–400 MB	1–2	every 5 min + player saves
Palworld (UE5)	1 large .sav, sequential	128 KB–1 MB	200–800 MB	1	every 15 min
Rust	many small entity files + world	8–64 KB	150–600 MB	1	every 10–15 min
Valheim	one .fwl + .db pair	64 KB	50–300 MB	1	every 20 min
Source (CS2/FiveM assets)	many small metadata writes	4–16 KB	low	1–4	continuous

The important shared property is queue depth 1. Enterprise NVMe numbers are quoted at QD32 or higher, where parallelism hides controller latency. A single save thread at QD1 sees the raw per-I/O latency of the drive and filesystem, which on a cheap consumer drive is 15–40× worse than the quoted IOPS figure implies.

The fio job files: three patterns worth running

Run each on a freshly mounted filesystem, after a full write pass to establish the SSD’s steady state. Do not benchmark an empty drive: SLC cache absorbs the first few hundred gigabytes and produces numbers you will never see in production. Drop caches between runs, and run each job for at least 120 seconds so the cache is exhausted.

# save-burst.fio  -- replicate a 400 MB game save, QD1, then fsync
[global]
ioengine=libaio
direct=1
fsync=1
size=400M
ramp_time=10
runtime=120
time_based=1
group_reporting

[write-burst]
rw=write
bs=32k
iodepth=1
numjobs=1
filename=/srv/bench/save.img

# p99 matters more than throughput here:
# fio --output-format=json | jq '.jobs[0].write.clat_ns.percentile["99.000000"]/1e6'

The second job measures the thing players actually feel: the fsync completion, not the write throughput. Use lat_percentiles=1 and read clat percentiles; the mean will look fine while the p99 is the pause you are hunting.

# save-mixed.fio -- save writes competing with 30 players loading chunks
[global]
ioengine=libaio
direct=1
time_based=1
runtime=120
group_reporting
lat_percentiles=1
percentile_list=50:90:99:99.9

[chunk-read]
rw=randread
bs=16k
iodepth=4
numjobs=2
filename=/srv/bench/world.img
size=8G

[save-write]
rw=write
bs=32k
iodepth=1
numjobs=1
fsync=1
filename=/srv/bench/save.img
size=400M
rate=180m

# third job: metadata churn, the hidden cost of many region files
[meta]
rw=randwrite
bs=4k
iodepth=1
numjobs=1
filename=/srv/bench/meta.img
size=1G
rate_iops=400

The mixed job is the one that predicts production behaviour. A drive that does 400 MB/s in isolation but drops to 40 MB/s and 60 ms p99 when a 16 KB read stream is present will produce exactly the “server lags during save” reports that isolation benchmarks never reproduce.

Expected results by drive class

These are steady-state figures from our build validation, measured at QD1 on ext4 with data=ordered. Treat anything more than 30% better than these as a cache artefact until proven otherwise by a repeated run.

Drive class	Save burst write	fsync p99	Mixed 16K read p99	Sustained after 400 GB writes
SATA consumer SSD	420 MB/s	12–25 ms	18–30 ms	Collapses to 90–150 MB/s
SATA DC SSD (5300 Pro)	510 MB/s	4–7 ms	6–9 ms	Holds ~480 MB/s
Gen3 NVMe consumer	1,600 MB/s	3–9 ms	5–8 ms	Drops to 500–800 MB/s
Gen3 NVMe DC (PM983)	1,750 MB/s	1.2–2.0 ms	1.8–2.6 ms	Holds ~1,600 MB/s
Gen4 NVMe DC (PM9A3)	4,100 MB/s	0.6–1.0 ms	0.9–1.4 ms	Holds ~3,900 MB/s
Gen4 NVMe DC RAID1	3,900 MB/s	0.7–1.1 ms	1.0–1.5 ms	Same; survives one drive failure

Read the last column first. Consumer drives are fast until their pseudo-SLC cache fills; a 400 GB write is enough to exhaust it on most 1–2 TB models. After that, writes go to TLC or QLC NAND directly and throughput falls by 60–85%. On a nightly backup window with three game servers saving at once, that is precisely when players are online.

Turning fsync latency into a tick budget

The translation is direct. A Minecraft save of 400 MB at a steady 400 MB/s takes 1.0 s; add fsync p99 of 15 ms and metadata overhead and the tick loop is blocked for 1.1–1.4 s. On the same burst at 4,100 MB/s it is 0.10–0.13 s. Since most servers freeze the tick loop during save, that difference is the visible lag spike, not a theoretical improvement.

Server	Save size	SATA consumer	Gen4 DC NVMe	Player-visible difference
Minecraft, 20 players	120 MB	0.35 s	0.05 s	Barely noticeable
Minecraft, 100 players	500 MB	1.55 s	0.15 s	1.4 s freeze eliminated
Palworld, 16 players	450 MB	1.40 s	0.13 s	Chunk reload hitch gone
Rust, 150 players	700 MB	2.30 s	0.20 s	Prevents mass disconnect on save

The threshold we use: if save-blocked time exceeds 250 ms, either shrink the save (async saves, compression, incremental entity saves) or change the drive. Below 250 ms players do not reliably notice. Above one second, they disconnect.

Filesystem and scheduler settings that change the result

Benchmarking with the wrong mount options measures a configuration you will not ship. These are the settings we validate against for game workloads: ext4 or XFS, noatime, write-back caching with a short dirty expiry so saves are batched rather than dribbled out.

# /etc/fstab: noatime removes a metadata write on every chunk read
/dev/nvme0n1p2 /srv ext4 defaults,noatime,nodiscard 0 2

# batched writes: flush dirty pages every 5s instead of 30s,
# keeping a save's dirty data from accumulating to thousands of pages
sysctl -w vm.dirty_expire_centisecs=500
sysctl -w vm.dirty_writeback_centisecs=500
sysctl -w vm.dirty_ratio=15
sysctl -w vm.dirty_background_ratio=5

# scheduler: none is correct for NVMe, mq-deadline for SATA
for d in /sys/block/nvme*/queue/scheduler; do echo none > $d; done

dirty_ratio=15 is the line most often wrong on 128 GB machines: the default sets 20% of RAM dirty, meaning 25 GB of unflushed write data, so a save burst can stall in writeback for seconds. Lowering the ceiling to 15% and flushing every 5 seconds converts a 2-second stall into a smoother, slightly less efficient write. Disable discard in fstab and run fstrim.timer weekly instead; continuous TRIM on a busy save volume adds latency spikes for no benefit.

Before and after: the report to keep

Save the JSON output of every run alongside the drive model, firmware version, filesystem and mount options. When a player reports lag six months later, the difference between the two reports tells you whether the hardware degraded or the workload grew. Collect these five numbers, nothing else:

write clat p99 at QD1, 32 KB block, after 400 GB of prior writes.
fsync latency p99 (fio fsync=1, or the game’s own save timer log).
mixed-workload 16 KB read p99 while a save is running.
Sustained throughput after cache exhaustion, in MB/s.
SMART Media and Wear_Leveling_Count delta across the test, to catch drive degradation.

If a drive passes all five at purchase and fails one 12 months later, the replacement decision is easy and defensible. If you would rather skip the validation work, the builds on our dedicated server range ship with Gen4 datacenter NVMe in RAID1 and the mount and sysctl settings above already applied, so the benchmark on arrival should match the table in this article within tolerance.

RAID level changes the p99 more than the drive does

RAID1 on NVMe does not double write throughput on a single-threaded save; it doubles the number of devices that must acknowledge each write before fsync returns. On datacenter NVMe with consistent latency, that costs 0.1–0.3 ms of extra p99 and buys you survival of a drive failure. On consumer drives with firmware-level latency outliers, the mirror amplifies them, because the slowest of the two devices gates every flush.

RAID10 across four drives measured 340k random write IOPS at QD32 in our tests but only a marginal p99 improvement over RAID1 at QD1, confirming that save latency is bound by the slowest single I/O, not by aggregate throughput. Choose RAID10 when you run several game servers whose saves overlap, and RAID1 when you run one.

Layout	QD1 save p99	Survives failure	Capacity cost	Best for
Single NVMe	0.6–1.0 ms	No	0%	Dev and staging only
RAID1 (2 drives)	0.7–1.1 ms	Yes	50%	Single-server production
RAID10 (4 drives)	0.8–1.2 ms	Yes	50%	Multiple servers, overlapping saves
RAID5/6	3–8 ms	Yes	25–33%	Never for save volumes

Parity RAID is the one layout to exclude outright. A 32 KB write becomes a read-modify-write of a full stripe, so 32 KB at QD1 turns into several device operations and the save p99 lands in the 3–8 ms range with unpredictable spikes during rebuilds. On a 500 MB save that is seconds of added tick block, which is worse than the SATA SSD you were trying to replace.

Benchmark the backup target too, not just the live volume

The live save volume is only half the I/O path. A nightly backup writes the same data again, and if that write lands on the same physical device it competes with the next save burst. Benchmark the backup destination with a sequential write job and an fsync at the end, and confirm it can absorb a full backup inside your window without stealing latency from the live volume.

# backup-target.fio -- one full server backup, sequential, then fsync
[global]
ioengine=libaio
direct=1
fsync=1
time_based=1
runtime=180
group_reporting
lat_percentiles=1

[backup-stream]
rw=write
bs=1024k
iodepth=4
numjobs=1
size=40G
filename=/mnt/backup/backup.img

If the backup target and live volume share a controller or a PCIe switch, run the mixed job from the previous section with the backup stream added as a third job. Any p99 read degradation above 2 ms means the two paths need separating — a second NVMe pair for backups, or an off-box target on the 1 Gbps link, which moves the write cost onto the network instead of the disk.