Restarting a game server looks like a single action. It is actually a sequence of five, and doing them out of order is how communities lose an hour of base building. The save flush has to complete before the process is told to stop; the process has to be given a real grace period to close connections; the announcement has to land early enough that players finish what they were doing instead of being dropped mid-fight.
The failure modes compound across a fleet. One server restarting cleanly is easy. Twelve servers restarting on a schedule, some of which are already halfway through an autosave when the timer fires, is where the data goes missing. The orchestration below is designed to be idempotent so that a scheduled run, a manual run, and a retry after a partial failure all converge on the same correct outcome. For hardware and node-count planning around a fleet of this size, our dedicated server comparison table is a reasonable starting point.
Step 1: Announce with a lead time tied to save interval
The warning window should be at least as long as your autosave interval plus a comfortable margin. If the server autosaves every five minutes and a save takes a few seconds, a two-minute warning means some players will be mid-save when the restart hits. Ten minutes of warnings at decreasing intervals is more humane and materially reduces the number of players dropped mid-action.
Announce through the game’s own chat or messaging channel rather than an external system players are not watching. The point is to reach the people in the world, not to produce an audit trail for an incident report.
Step 2: Drain, do not cut
Draining means stopping new connections while leaving existing ones alive. If the game or a fronting proxy supports a drain state, use it. If it does not, the practical equivalent is to remove the node from whatever discovery mechanism players use while leaving the process running, so current sessions finish naturally rather than being severed.
Set a bounded drain window. Some clients will always remain connected — an idle player in a menu, a bot, a stuck session. The drain should shave the population down and then proceed, not wait indefinitely for an empty server that never arrives.
Step 3: Force a save, then wait for the flush to land
Issue the save command through the game’s console or admin interface, then wait for confirmation that it completed. This is the step most automation gets wrong: it sends the save command and immediately sends the stop signal, which means the process is killed while the write is still in flight. The result is a truncated or corrupted world file that the game may or may not detect on next boot.
Confirm the write actually reached stable storage, not just the page cache. A filesystem sync after the save returns is cheap insurance and turns an unverifiable assumption into a fact. If your save path is on a network volume, this matters even more.
Step 4: Terminate with a grace period, then escalate
Send a polite termination signal and give the process time to shut down in its own way — closing sockets, writing final state, releasing locks. Many game servers handle this correctly if they are given ten to thirty seconds. Escalate to a hard kill only after that window expires, and log the fact that you had to. A node that consistently requires hard kills has a shutdown bug worth investigating, and the log is how you find it.
Make the escalation decision on parsed process exit rather than a sleep. A fixed sleep either wastes time when shutdown is fast or kills prematurely when it is slow; waiting on the actual process state is correct in both cases.
Step 5: Verify before rejoining the pool
After restart, do not immediately re-advertise the node. Wait until it reports ready — listening on its port and passing a real query, not merely having a process alive. Then check the world load succeeded, because a server can come up with an empty or fallback world when the save failed to read, and that looks exactly like a healthy start from the outside.
Only after readiness is confirmed should the node rejoin discovery. This single ordering choice converts “server came back empty” from a catastrophic discovery into a caught error.
Making the sequence idempotent
Idempotence here means each step can be run twice without harm. A save command is naturally idempotent. A stop signal to an already-stopped process is a no-op if you check state first. The announcement should be keyed to the run, so a retry does not replay ten minutes of warnings. The readiness check is read-only.
Write a single state machine per node with explicit phases and persist the current phase to disk before each transition. If the orchestrator itself dies mid-run, the next invocation reads the phase, resumes from there, and does not reissue the earlier steps. This is the difference between a retry and a double restart, and on a fleet the difference is visible to every player.
Fleet-wide considerations
Do not restart the whole fleet at once. Stagger nodes so the service stays available, and so the disk and network systems are not all hitting their write bursts simultaneously. If every node in the rack forces a save at the same instant, the shared storage path becomes the bottleneck and a routine restart turns into a latency incident.
Space the staggered starts far enough apart that each node’s load phase completes before the next begins. Startup is the heaviest I/O moment in a server’s lifecycle, and overlapping startups is how a cold boot turns into a timeout cascade. Our game server hosting plans show the disk and RAM per tier, which tells you how much startup burst you can absorb at once.
Test the sequence, not the endpoint
Run the full sequence during a maintenance window with players online, at a low-population hour, and confirm that nobody loses progress. Then deliberately break it: kill the orchestrator after the save step and confirm the rerun resumes correctly rather than starting over. That second test is what actually proves the design.
Save loss during restarts is almost never caused by a bad game server. It is caused by an orchestrator that assumed the save had finished when it had only been requested.
Leave a Reply
You must be logged in to post a comment.