When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Correlating crashes with memory events
The useful workflow is to join three timelines: process crash timestamps, kernel log entries and EDAC counters. Export all three to the same monitoring system and overlay them for a 90 day window. A crash that sits within 10 minutes of a correctable event on the same node is strong evidence that memory is the root cause, even though the corrected bit was not the crash itself — it points at a degrading DIMM.
For crashes with no memory events nearby, enable kernel core dumps and inspect the faulting address against the node’s physical memory map. A fault in a page allocated to a long-lived arena, is the second clue. Both together justify a DIMM swap.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Correlating crashes with memory events
The useful workflow is to join three timelines: process crash timestamps, kernel log entries and EDAC counters. Export all three to the same monitoring system and overlay them for a 90 day window. A crash that sits within 10 minutes of a correctable event on the same node is strong evidence that memory is the root cause, even though the corrected bit was not the crash itself — it points at a degrading DIMM.
For crashes with no memory events nearby, enable kernel core dumps and inspect the faulting address against the node’s physical memory map. A fault in a page allocated to a long-lived arena, is the second clue. Both together justify a DIMM swap.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Correlating crashes with memory events
The useful workflow is to join three timelines: process crash timestamps, kernel log entries and EDAC counters. Export all three to the same monitoring system and overlay them for a 90 day window. A crash that sits within 10 minutes of a correctable event on the same node is strong evidence that memory is the root cause, even though the corrected bit was not the crash itself — it points at a degrading DIMM.
For crashes with no memory events nearby, enable kernel core dumps and inspect the faulting address against the node’s physical memory map. A fault in a page allocated to a long-lived arena, is the second clue. Both together justify a DIMM swap.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Correlating crashes with memory events
The useful workflow is to join three timelines: process crash timestamps, kernel log entries and EDAC counters. Export all three to the same monitoring system and overlay them for a 90 day window. A crash that sits within 10 minutes of a correctable event on the same node is strong evidence that memory is the root cause, even though the corrected bit was not the crash itself — it points at a degrading DIMM.
For crashes with no memory events nearby, enable kernel core dumps and inspect the faulting address against the node’s physical memory map. A fault in a page allocated to a long-lived arena, is the second clue. Both together justify a DIMM swap.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Correlating crashes with memory events
The useful workflow is to join three timelines: process crash timestamps, kernel log entries and EDAC counters. Export all three to the same monitoring system and overlay them for a 90 day window. A crash that sits within 10 minutes of a correctable event on the same node is strong evidence that memory is the root cause, even though the corrected bit was not the crash itself — it points at a degrading DIMM.
For crashes with no memory events nearby, enable kernel core dumps and inspect the faulting address against the node’s physical memory map. A fault in a page allocated to a long-lived arena, is the second clue. Both together justify a DIMM swap.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Correlating crashes with memory events
The useful workflow is to join three timelines: process crash timestamps, kernel log entries and EDAC counters. Export all three to the same monitoring system and overlay them for a 90 day window. A crash that sits within 10 minutes of a correctable event on the same node is strong evidence that memory is the root cause, even though the corrected bit was not the crash itself — it points at a degrading DIMM.
For crashes with no memory events nearby, enable kernel core dumps and inspect the faulting address against the node’s physical memory map. A fault in a page allocated to a long-lived arena, is the second clue. Both together justify a DIMM swap.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Correlating crashes with memory events
The useful workflow is to join three timelines: process crash timestamps, kernel log entries and EDAC counters. Export all three to the same monitoring system and overlay them for a 90 day window. A crash that sits within 10 minutes of a correctable event on the same node is strong evidence that memory is the root cause, even though the corrected bit was not the crash itself — it points at a degrading DIMM.
For crashes with no memory events nearby, enable kernel core dumps and inspect the faulting address against the node’s physical memory map. A fault in a page allocated to a long-lived arena, is the second clue. Both together justify a DIMM swap.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
A game server that has been running for six weeks crashes in the physics tick with a stack trace pointing at a heap pointer, restarts, and runs fine for another three weeks. The RAM passed twelve hours of memtest at installation. Nothing in the application logs explains it, and the same build runs without incident on a test machine that also uses cheap memory.
That pattern is the signature of silent bit errors in DRAM: a flip that occurs in a rarely touched page, goes uncorrected, and only surfaces when the corrupted value happens to be used as a pointer, an index or a checksum. ECC memory does not stop the flip. It stops the flip from becoming an unexplained crash.
How silent corruption accumulates
Non-ECC DRAM has no parity at all, so a single flipped bit is invisible to the platform. Cosmic ray and alpha particle events are rare per cell but not rare per server: the widely cited Google and Facebook fleets studies put correctable error rates at roughly one per several GB per hour under heavy load, which for a 64 GB node means multiple events a day. Server-class hardware reports and corrects these silently.
Most of those flips land in a page that is rewritten before it is read, so nothing happens. The dangerous subset lands in a structure that lives for hours: a hash table, a connection object, an economy ledger cache. A single flipped bit in a pointer offset moves a read by 4 to 64 bytes and produces a detectable fault. A flipped bit in a length field produces a buffer overrun that the allocator may not catch for hours.
Why long-running servers feel it more than desktops
A desktop reboots daily, throws away its working set and reloads it from disk. A game node runs the same process image for weeks and keeps world state in memory across the entire period. Every byte of resident memory is exposed to the error rate continuously, and the longer the process lives, the more opportunity for a corrupt value to reach a code path where it matters.
Tick-based servers add a second amplifier: they read and write the same simulation state at a fixed rate, so a corrupted value in a frequently accessed structure is exercised thousands of times per minute instead of being abandoned. This is why crash reports cluster in the simulation loop rather than in request handling.
Detection: mcelog, edac-util and the event log
On ECC hardware the memory controller reports correctable and uncorrectable events through Machine Check Architecture. The kernel exposes them via the EDAC subsystem, and userspace reads them from /sys/devices/system/edac/mc/ or through mcelog. Without this instrumentation ECC is inert: the hardware corrects the bit and nothing records the degradation.
# counters per memory controller / rank
grep -r . /sys/devices/system/edac/mc/*/ce_count
edac-util -v
# decoded machine check events
mcelog --client
journalctl -k | grep -i "machine check\|corrected"Track the correctable error count as a time series. A flat zero is normal. A single correctable event in a month is normal. More than one per hour on a rank that has been healthy for a year is a warning sign, and an uncorrectable error rate above one per week means the machine should be drained now rather than after the next maintenance window.
| Signal | Threshold | Action |
|---|---|---|
| Correctable (CE) rate | <1 per 24 h per node | Monitor only |
| CE rate climbing on one rank | >5 per hour | Schedule DIMM replacement |
| CE burst on one DIMM | >50 in 10 minutes | Drain and replace |
| Uncorrectable (UE) | Any occurrence | Drain immediately, preserve logs |
| Scrub cycle | Weekly full scrub | Patrol scrub enabled |
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Correlating crashes with memory events
The useful workflow is to join three timelines: process crash timestamps, kernel log entries and EDAC counters. Export all three to the same monitoring system and overlay them for a 90 day window. A crash that sits within 10 minutes of a correctable event on the same node is strong evidence that memory is the root cause, even though the corrected bit was not the crash itself — it points at a degrading DIMM.
For crashes with no memory events nearby, enable kernel core dumps and inspect the faulting address against the node’s physical memory map. A fault in a page allocated to a long-lived arena, is the second clue. Both together justify a DIMM swap.
Turning on scrub and logging properly
Patrol scrubbing walks all of memory periodically and rewrites correctable errors before they become double-bit failures. Enable it and set the scrub rate to complete a full pass weekly. On the logging side, make sure mcelog or rasdaemon is running under systemd and that its output reaches your central log store; a local-only log disappears with the disk.
Also make the counters survivable across reboots. EDAC counters reset at boot on most platforms, so persist the cumulative total into your metrics backend rather than trusting ce_count after a power cycle.
When non-ECC is genuinely acceptable
Non-ECC has a real place. Stateless CI runners, short-lived build containers, load generators and caches that can be rebuilt from a source of truth tolerate bit flips because a corrupt result is either detected downstream or discarded. The math changes the moment the workload holds uncommitted state that no other system can reconstruct.
For game servers the deciding question is simple: if a value in memory is wrong, does the player lose progress? A world save, an economy ledger, a player inventory and an instance state all answer yes. That puts them in the ECC camp regardless of node size. A single redis-role cache or a metrics scraper does not.
Verifying the ECC path end to end
Buying ECC memory is not the same as running with ECC active. On some consumer boards the modules run in non-ECC mode silently, and on some server platforms a mixed population disables correction. Confirm with dmidecode -t memory and look for Error Correction Type: Multi-bit ECC and a non-zero total width.
Check that the EDAC driver loaded at boot and that each controller reports a CSR row count. If edac-util -v prints nothing, the platform is not exposing error reporting and you are running blind even if the memory is ECC. Boot firmware settings sometimes hide memory health reporting behind an option that defaults to off on older boards.
The operational payoff
ECC does not reduce crash rate to zero, but it converts an entire class of unexplained failure into a logged, attributable event. That means a machine can be drained on evidence rather than superstition, and a warranty claim can cite the exact DIMM and the exact error count. Providers that ship ECC by default make this a non-issue; servers built on server-grade boards generally do, while budget desktop hardware generally does not.
Before you spend another week chasing a physics exception, read the EDAC counters. They answer in ten seconds a question that log archaeology takes days to approach, and on a node that is supposed to stay up for months at a time, that answer is the difference between a stable service and a recurring incident.
Leave a Reply
You must be logged in to post a comment.