{"id":1560,"date":"2026-11-03T02:04:02","date_gmt":"2026-11-03T02:04:02","guid":{"rendered":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/?p=1560"},"modified":"2026-11-03T02:04:02","modified_gmt":"2026-11-03T02:04:02","slug":"hardware-health-monitoring-smart-ipmi-runbook","status":"publish","type":"post","link":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/","title":{"rendered":"Hardware Health Monitoring for Dedicated Servers: A SMART and IPMI Runbook"},"content":{"rendered":"
\n

Bare-metal servers fail in ways that are almost never sudden. A drive starts reallocating sectors weeks before it drops out of an array. A DIMM begins correcting errors silently. A fan slows down and the CPU throttles only under sustained load, which on a game server means the Tuesday-night pattern of stuttering that nobody can reproduce on Wednesday morning. All of this is visible in telemetry the hardware already exposes, and almost nobody collects it. This runbook covers the three data sources worth monitoring — SMART, IPMI or Redfish sensors, and the kernel’s own error counters — plus the alert thresholds that turn them into early warnings instead of postmortems. If you are still specifying the machine, the dedicated hardware reference on this site<\/a> covers which components to watch most closely.<\/p>\n

What each telemetry layer actually tells you<\/h2>\n

SMART is per-device and covers drive health, wear, temperature, and reallocated sectors. It is the earliest indicator of storage failure and the only one that reports endurance consumption on NVMe.<\/p>\n

IPMI, iDRAC, or iLO expose chassis-level sensors: inlet and exhaust temperatures, fan speeds, PSU voltages, and sometimes per-DIMM error counts. Redfish is the modern REST API over the same data and is far easier to script than raw IPMI, though many dedicated providers still give you only IPMI access.<\/p>\n

The kernel exposes what the hardware detected: correctable and uncorrectable machine-check exceptions, AER errors on PCIe links, and thermal throttling transitions. These are the events that explain an otherwise inexplicable latency spike.<\/p>\n\n\n\n\n\n\n\n\n\n\n
Source<\/th>\nKey reading<\/th>\nWarn<\/th>\nAct<\/th>\n<\/tr>\n<\/thead>\n
SMART (NVMe)<\/td>\nPercentage Used \/ Media Errors<\/td>\n> 80% \/ > 0<\/td>\nOrder replacement<\/td>\n<\/tr>\n
SMART (SATA\/HDD)<\/td>\nReallocated sectors<\/td>\n> 0 and rising<\/td>\nReplace before failure<\/td>\n<\/tr>\n
SMART temperature<\/td>\nComposite temperature<\/td>\n> 70 °C sustained<\/td>\nCheck airflow<\/td>\n<\/tr>\n
IPMI inlet temp<\/td>\nChassis inlet<\/td>\n> 27 °C<\/td>\nRaise with provider<\/td>\n<\/tr>\n
IPMI PSU<\/td>\nVoltage out of range<\/td>\nAny deviation<\/td>\nSwap PSU<\/td>\n<\/tr>\n
Kernel<\/td>\nMCE \/ AER counters<\/td>\nAny increase<\/td>\nInspect, plan replacement<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

SMART: beyond the pass\/fail verdict<\/h2>\n

The single most misleading reading on a server is “PASSED”. SMART’s overall verdict is a threshold check with generous margins, and a drive can report PASSED while its reallocated sector count climbs steadily. Read the attributes, not the verdict.<\/p>\n

smartctl -a \/dev\/nvme0 | grep -E \"Percentage Used|Media and Data Integrity Errors|Temperature\"\nsmartctl -A \/dev\/sda | grep -E \"Reallocated_Sector|Pending_Sector|UDMA_CRC|Temperature\"<\/code><\/pre>\n
On NVMe, Percentage Used<\/em> is the endurance estimate and Media and Data Integrity Errors<\/em> is a count of frames that failed internal checks. Any non-zero media error count is a reason to open a support ticket and start planning a swap, not a reason to react immediately.<\/p>\n
On SATA and SAS drives, Reallocated_Sector_Ct<\/em> is the headline. A drive with one reallocated sector is fine; a drive that gains twenty in a week is not. Tracking the rate<\/em> of change is what makes SMART useful, which means you need historical data rather than a point-in-time reading.<\/p>\n
UDMA CRC errors point at the cable or backplane rather than the drive. A clustering of CRC errors on one bay usually means a failing cable, and replacing the drive will not fix it.<\/p>\n
IPMI and Redfish: the chassis view<\/h2>\n
Enumerate sensors first, because names and IDs vary by vendor and are the most common cause of broken monitoring scripts.<\/p>\n
ipmitool sensor list | head -40\nipmitool sdr type temperature\ncurl -sk -u user:pass https:\/\/bmc\/redfish\/v1\/Chassis\/1\/Thermal | jq '.Temperatures[] | {Name, ReadingCelsius}'<\/code><\/pre>\n
Three readings carry most of the signal. Inlet temperature tells you whether the provider’s cooling is doing its job; anything sustained above about 27 °C raises internal component temperatures by a similar margin and shortens drive life. Fan speeds that ramp and de-ramp on a minute cycle suggest a thermal control loop hunting, which frequently precedes a fan failure. PSU voltage readings that drift from nominal indicate a failing unit, and a server with redundant PSUs running on one will behave perfectly until the second fails.<\/p>\n
On many dedicated servers the BMC is on a separate network and reachable only through the provider’s portal or a VPN. Confirm how you will reach it during an incident before you need to, because a BMC you cannot reach is worthless at 3 a.m.<\/p>\n
Kernel error counters that explain mystery latency<\/h2>\n
Hardware that is degrading shows up first as corrected errors in the kernel log. Correctable machine-check exceptions on a DIMM are the classic case: the server keeps running, but memory bandwidth drops as the error handling path retries, and applications see unexplained slow patches.<\/p>\n
journalctl -k -p err --since \"7 days ago\" | grep -Ei \"mce|aer|corrected|throttl\"\ngrep . \/sys\/devices\/system\/machinecheck\/machinecheck*\/ce_count 2>\/dev\/null\ncat \/proc\/interrupts | head -20<\/code><\/pre>\n
Thermal throttling is reported by the CPU itself and is usually visible as frequency collapsing under load while utilization stays flat. If a game server’s tick time degrades only after twenty minutes of full load, thermal throttling is the first hypothesis to test.<\/p>\n
AER correctable errors on a PCIe device — commonly a network card or NVMe controller — indicate a link-training problem. These are usually fixable by reseating the card or moving it to a different slot, but they are worth tracking because they can escalate into a device dropping off the bus entirely.<\/p>\n
Turning checks into a weekly runbook<\/h2>\n
Collect everything on a schedule and store the results, because trends matter more than thresholds. A cron job that appends a timestamped JSON blob once an hour gives you the history that makes threshold alerts meaningful.<\/p>\n
0 * * * * \/usr\/local\/bin\/hw-collect.sh >> \/var\/log\/hw\/smart.jsonl\nsmartctl -j -a \/dev\/nvme0 >> \/var\/log\/hw\/smart.jsonl<\/code><\/pre>\n
Review the data weekly. Look for three things: any drive whose wear percentage moved more than 3 points, any sensor whose typical value shifted by more than 5 °C, and any counter that went from zero to non-zero. Those three patterns precede nearly every hardware failure.<\/p>\n
What to do when a threshold trips<\/h2>\n
For a drive: confirm the alert with a second tool, check for a spare in the chassis, schedule a replacement window, and never rebuild an array while a second member is degraded. For a DIMM, open a ticket with the error output and request a swap — intermittent errors do not resolve themselves. For a PSU, replace it immediately even if the server is running.<\/p>\n
Document each action in the runbook. Hardware monitoring only pays off when the response is a short, known procedure rather than an improvised investigation during an outage. That is the difference between a server that fails predictably and one that fails at the worst possible moment. For more bare-metal operations walkthroughs, see the dedicated server operations guides on bestdedicatedwebhostingserver.com<\/a>.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Bare-metal servers fail in ways that are almost never sudden. A drive starts reallocating sectors weeks before it drops out of an array. A DIMM begins correcting errors silently. A…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":0,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1560","post","type-post","status-publish","format-standard","hentry","category-server-guides-tutorials"],"yoast_head":"\nHardware Health Monitoring for Dedicated Servers: A SMART and IPMI Runbook - Best Dedicated Web Hosting Server Blog<\/title>\n<meta name=\"description\" content=\"Bare-metal servers fail in ways that are almost never sudden. A drive starts reallocating sectors weeks before it drops out of an array. A DIMM begins\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Hardware Health Monitoring for Dedicated Servers: A SMART and IPMI Runbook\" \/>\n<meta property=\"og:description\" content=\"Hardware Health Monitoring for Dedicated Servers: A SMART and IPMI Runbook\" \/>\n<meta property=\"og:url\" content=\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/\" \/>\n<meta property=\"og:site_name\" content=\"Best Dedicated Web Hosting Server Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-03T02:04:02+00:00\" \/>\n<meta name=\"author\" content=\"bestdedicatedwebhostingserver\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@bestdedicatedwebhostingserver\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"bestdedicatedwebhostingserver\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/\",\"url\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/\",\"name\":\"Hardware Health Monitoring for Dedicated Servers: A SMART and IPMI Runbook - Best Dedicated Web Hosting Server Blog\",\"isPartOf\":{\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#website\"},\"datePublished\":\"2026-11-03T02:04:02+00:00\",\"author\":{\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#\/schema\/person\/6eb5f9ea35033fe8e67df44397e089b1\"},\"description\":\"Bare-metal servers fail in ways that are almost never sudden. A drive starts reallocating sectors weeks before it drops out of an array. A DIMM begins\",\"breadcrumb\":{\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Hardware Health Monitoring for Dedicated Servers: A SMART and IPMI Runbook\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#website\",\"url\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/\",\"name\":\"Best Dedicated Web Hosting Server Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#\/schema\/person\/6eb5f9ea35033fe8e67df44397e089b1\",\"name\":\"bestdedicatedwebhostingserver\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/462a07aa37399bdb149e5f91dc3cd8906656bc4c7ed391a3b6f0199c5d2ab964?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/462a07aa37399bdb149e5f91dc3cd8906656bc4c7ed391a3b6f0199c5d2ab964?s=96&d=mm&r=g\",\"caption\":\"bestdedicatedwebhostingserver\"},\"sameAs\":[\"https:\/\/bestdedicatedwebhostingserver.com\/blog\",\"https:\/\/x.com\/bestdedicatedwebhostingserver\"],\"url\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/author\/bestdedicatedwebhostingserver\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Hardware Health Monitoring for Dedicated Servers: A SMART and IPMI Runbook - Best Dedicated Web Hosting Server Blog","description":"Bare-metal servers fail in ways that are almost never sudden. A drive starts reallocating sectors weeks before it drops out of an array. A DIMM begins","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/","og_locale":"en_US","og_type":"article","og_title":"Hardware Health Monitoring for Dedicated Servers: A SMART and IPMI Runbook","og_description":"Hardware Health Monitoring for Dedicated Servers: A SMART and IPMI Runbook","og_url":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/","og_site_name":"Best Dedicated Web Hosting Server Blog","article_published_time":"2026-11-03T02:04:02+00:00","author":"bestdedicatedwebhostingserver","twitter_card":"summary_large_image","twitter_creator":"@bestdedicatedwebhostingserver","twitter_misc":{"Written by":"bestdedicatedwebhostingserver","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/","url":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/","name":"Hardware Health Monitoring for Dedicated Servers: A SMART and IPMI Runbook - Best Dedicated Web Hosting Server Blog","isPartOf":{"@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#website"},"datePublished":"2026-11-03T02:04:02+00:00","author":{"@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#\/schema\/person\/6eb5f9ea35033fe8e67df44397e089b1"},"description":"Bare-metal servers fail in ways that are almost never sudden. A drive starts reallocating sectors weeks before it drops out of an array. A DIMM begins","breadcrumb":{"@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/hardware-health-monitoring-smart-ipmi-runbook\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Hardware Health Monitoring for Dedicated Servers: A SMART and IPMI Runbook"}]},{"@type":"WebSite","@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#website","url":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/","name":"Best Dedicated Web Hosting Server Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#\/schema\/person\/6eb5f9ea35033fe8e67df44397e089b1","name":"bestdedicatedwebhostingserver","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/462a07aa37399bdb149e5f91dc3cd8906656bc4c7ed391a3b6f0199c5d2ab964?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/462a07aa37399bdb149e5f91dc3cd8906656bc4c7ed391a3b6f0199c5d2ab964?s=96&d=mm&r=g","caption":"bestdedicatedwebhostingserver"},"sameAs":["https:\/\/bestdedicatedwebhostingserver.com\/blog","https:\/\/x.com\/bestdedicatedwebhostingserver"],"url":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/author\/bestdedicatedwebhostingserver\/"}]}},"_links":{"self":[{"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/posts\/1560","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/comments?post=1560"}],"version-history":[{"count":1,"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/posts\/1560\/revisions"}],"predecessor-version":[{"id":1563,"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/posts\/1560\/revisions\/1563"}],"wp:attachment":[{"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/media?parent=1560"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/categories?post=1560"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/tags?post=1560"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}