Crash-Loop Triage: Finding the Offending Mod When a Modded Server Restarts Every 90 Seconds

A modded Minecraft server that dies 14 seconds after accepting connections generates about 40 crash reports in ten hours and says nothing useful after the first three. A Rust server in the same loop often writes no crash report at all, leaving a truncated log and a core dump while the supervisor keeps relaunching it. Both sequences start by stopping the loop, not reading the dump.

This assumes shell access, root, and the ability to edit the launch command. The exit codes and filenames below are the real ones from Paper/Forge/Fabric and Rust’s server binary.

Stop the Loop Before You Read Anything

A restarting server destroys evidence: each boot overwrites the latest log, rotates crash reports, and can corrupt region files mid-write. Disable automatic restart, then start once manually in the foreground.

# systemd case
systemctl stop gameserver
systemctl disable gameserver
# start manually, capture everything
cd /srv/server && ./start.sh 2>&1 | tee /tmp/tail.log

Then reproduce twice in a row. One crash is a flaky interaction; the same crash at the same point twice is deterministic and bisectable. If it only happens with players connected, log a test client in at the same moment each run.

Reading a JVM Crash: hs_err_pid and the Log Tail

The hs_err_pid*.log file is written for a native crash, not a Java exception. Its header names the signal and the failing frame, which tells you whether you are debugging Java code or a native library a mod loaded.

head -40 hs_err_pid*.log      # signal, siginfo, problematic frame
# SEGV / SIGBUS in a .so  -> native mod code
# EXCEPTION_ACCESS_VIOLATION + 'Problematic frame: C [libc.so]' -> suspect JNI
# OutOfMemoryError in a thread dump -> heap, not a crash

SIGSEGV with the problematic frame in a .so under mods/ or libraries/ is a native library bug, usually a build compiled against a different glibc or CPU baseline. SIGBUS in a memory-mapped file points at region files, and the fix is a restore, not a mod removal.

With no hs_err file the JVM caught an exception and exited cleanly, which is a Java-level problem: read the log for the first Caused by: chain, not the last. The final exception is usually a shutdown handler failing after the real fault.

Isolating the Offending Mod by Bisection

With a deterministic reproduction, bisect the mod list. Moving half the jars out of mods/ and booting takes two minutes; a 180-mod pack resolves in eight rounds.

mkdir -p /tmp/modstash
ls mods > /tmp/allmods.txt      # record the full list first
# move second half out
sed -n '91,180p' /tmp/allmods.txt | xargs -I{} mv mods/{} /tmp/modstash/

Two caveats break naive bisection. Removing a library mod others depend on produces its own missing-dependency crash at load, so check the signature matches the original. Some packs fail only on a combination, so bisection converges on a pair — if the two halves give different crashes, look in the overlap.

Rust has no loadable mod list in that sense, so the equivalent is plugin bisection under oxide/plugins/. The failure usually surfaces as a .NET exception with the plugin name in the trace — same method, smaller search space.

Memory Settings That Actually Cause Crash Loops

Many ‘mod crashes’ are heap configuration mistakes. Three account for most cases, and all produce a loop that looks like a mod fault because the exception fires inside mod code.

First, -Xmx set close to physical RAM. A 16 GB box with -Xmx14G leaves 2 GB for the OS, metaspace, thread stacks, direct buffers, and page cache. The OOM killer takes the process and there is no hs_err because the kernel did the killing. If dmesg | grep -i 'killed process' shows it, the fix is -Xmx10G, not a mod removal.

Second, heap versus off-heap. Chunk generators and Netty allocate direct buffers outside the heap, so a server at 60% heap usage can still die of native exhaustion at 13 GB RSS. If RSS climbs while heap stays flat, the limit is off-heap and -XX:MaxDirectMemorySize needs an explicit value.

Third, GC pressure misread as a crash. An 8 GB heap running 200 mods at 60 players spends 25% of its time in GC and logs ‘tick took 4500ms’ instead of crashing, until it degrades into an OOM. G1 with a 200–400 ms pause target is the practical default; older collectors produce pause spikes that look like network problems.

Symptom	Likely cause	Check
No hs_err, process gone	kernel OOM killer	dmesg grep killed
RSS high, heap flat	off-heap / direct buffers	RSS vs heap graph
Exception in mod code	dependency or version mismatch	first Caused by
SIGSEGV in .so	native library ABI	problematic frame line

Reproducing Deterministically

Record the conditions: world, player count, the last action taken, and the wall-clock time. Then reproduce with a script placing a bot at the same coordinates and repeating the action.

For chunk-loading crashes, teleporting a bot across a coordinate boundary is enough; for entity-tick crashes, spawning the entity type at a fixed position does it. The test need only be repeatable, because a fix you cannot verify is a guess.

What the Exit Code Tells You

The JVM exit code narrows the search before you open a single file. Exit 0 means a clean shutdown request, usually a stop command or a supervisor signal, not a crash. Exit 1 means an uncaught exception with a stack trace somewhere in the log. Exit 134 is SIGABRT, which on modded servers almost always means the JVM aborted on a fatal error and wrote an hs_err file.

Exit 137 is the important one: 128 + 9, SIGKILL, which is the OOM killer or a container memory limit rather than any mod. Exit 143 is 128 + 15, SIGTERM, meaning the supervisor or the host sent a stop signal mid-startup, often after a health-check timeout. Chasing mods after a 137 is wasted effort — fix the memory ceiling first.

Fixing, Verifying, and Keeping the Server Up

Once identified, prefer in order: update to a version that fixes it, swap for an equivalent, or pin its config to avoid the code path. Removing a mod outright is the last resort on an established world, since it strips blocks and entities from saved chunks permanently.

Verify with three clean reproduction runs, then an hour at normal player load, before re-enabling the supervisor. If it was a native library, confirm the replacement matches your JVM architecture.

Keep the evidence: one crash report per incident, the mod list at the time, and the launch command turn the next incident into a five-minute lookup. Servers with high mod counts also benefit from generous RAM and sustained clock headroom, since a node that is memory-starved or thermally throttled turns a mild mod bug into a hard crash loop — and a written hardware baseline is a useful second reference when the fault lies in the platform rather than the pack.