Game Server Save Safety: Atomic Writes, fsync and Restore Drills That Hold

Every game server that has been online for six months has a moment where it saved a half-written world. Not because the disk failed, but because the process was killed mid-write by an OOM killer, a panel “restart” button, a power event, or a backup script that copied a live region file. The difference between a five-minute rollback and a wiped community is a combination of atomic write mechanics, fsync discipline, and a retention policy you have actually tested. This guide covers how to make saves crash-safe and how to build a restore drill that proves it. It assumes root on the host — the standard situation when you run on your own bare-metal game server rather than a managed one.

What “Corrupt” Actually Means

• Torn write: the file is partially updated; header points to a length that exceeds the data or a chunk offset that lands mid-record.
• Stale index: metadata was flushed but data was not, so the file references blocks that were never written.
• Logical inconsistency: every byte is valid, but item A is duplicated and item B vanished — typically from a save that raced with game-thread mutation.
• Silent zeroing: filesystem journal replay produced 4 KiB of zeros where data should be after power loss.

Each has a different fix. Only the first two are solved by atomic rename; the third requires the game to snapshot state, and the fourth requires proper fsync plus a UPS.

Atomic Rename Is the Baseline

Write to a temporary file in the same directory, fsync it, then rename() over the original. rename within one filesystem is atomic: readers see either the old file or the new one, never a mix. Then fsync the directory so the rename itself survives a crash.

import os, tempfile

def save_atomic(path, payload):
    d = os.path.dirname(path)
    fd, tmp = tempfile.mkstemp(dir=d, prefix=".save-")
    try:
        with os.fdopen(fd, "wb") as f:
            f.write(payload)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)          # atomic rename
        dfd = os.open(d, os.O_DIRECTORY)
        os.fsync(dfd)                  # persist the rename
        os.close(dfd)
    finally:
        if os.path.exists(tmp):
            os.unlink(tmp)

Most games do not do this. Minecraft writes region files in place; Rust writes .sav with its own scheme; Valheim uses temp files but relies on the game thread finishing cleanly. You cannot patch every game, so the wrapper has to be the backup and restore layer, not the save path.

When Drop-in Save Wrappers Are Worth It

Some servers support LD_PRELOAD interception of write/fsync to add ordering guarantees, and some support storage-level snapshots. Snapshots are the more reliable route:

A ZFS dataset per game instance with fsfreeze -f /srv/game before the snapshot gives you a consistent point-in-time image in under a second. Thaw immediately after; the freeze is shorter than a save and invisible to players.

fsync Discipline and the Storage Stack

The durability of a save therefore depends on three claims you should not take for granted:

• SSD write caches acknowledge writes before they hit NAND. A power loss can lose the last few hundred MB even after fsync, unless the drive has power-loss protection (PLP). Consumer NVMe usually does not.
• RAID controllers with write-back cache and no battery are worse than no controller at all: acknowledged writes vanish on power loss.
• Network storage (NFS) with async export can acknowledge fsync before the server persists it.

Verify your drive’s behaviour rather than trusting the spec sheet. Check nvme id-ctrl /dev/nvme0 | grep -i 'vwc' and confirm whether volatile write cache is present and whether PLP is reported. On hosts without PLP, keep commit low in ext4 (or leave the default) and put the server on a UPS with monitored shutdown.

The Restore Drill

A backup that has never been restored is a hypothesis. Run this quarterly, on a scratch host, with a timer:

• Pick the newest snapshot and the one from 7 days back.
• Restore to a clean directory, start the server on a non-public port, join with a test client, verify spawn chunks, inventory, and at least one structure built after the snapshot date (to confirm RPO).
• Record: time to restore, time to first successful join, and any missing entities.
• Delete the drill server and log the result in your runbook.

If a restore takes 40 minutes, your real downtime will be 40 minutes plus panic. Target under 10 minutes for a 20 GB world on local NVMe; if it is slower, the bottleneck is usually decompression or a network path, both fixable.

Retention That Survives the Operator

Off-host copies need checksums, not just timestamps. Compute a SHA-256 manifest per snapshot at write time and verify a sample on read; silent bit rot on the backup volume is rare but indistinguishable from corruption at restore time unless you check. Keep the manifest alongside the snapshot and test one restore with verification each quarter so the process stays familiar to whoever is on call.

Keep hourly snapshots for 48 hours, daily for 14 days, weekly for 8 weeks, and one monthly offsite. Store at least the monthly tier on a different provider or a different physical location — a host-level compromise or a billing suspension takes the server and its local backups together. Encrypt offsite copies; save files contain player IPs and sometimes chat logs, both of which are personal data in most jurisdictions.

Finally, make the retention policy automatic and monitored: an alert when the newest snapshot is older than two hours catches a broken backup job far earlier than a restore attempt at 2 AM. With atomic writes, PLP-aware hardware, frozen snapshots, and a tested drill, a crash becomes a rollback of minutes rather than the end of a world. If you are planning an upgrade, choose dedicated server storage with power-loss protection and snapshot support and put the drill on the calendar before you need it.