{"id":1707,"date":"2026-11-17T22:35:39","date_gmt":"2026-11-17T22:35:39","guid":{"rendered":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/?p=1707"},"modified":"2026-11-17T22:35:39","modified_gmt":"2026-11-17T22:35:39","slug":"dedicated-server-pcie-nvme-raid-topology-audit","status":"publish","type":"post","link":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/","title":{"rendered":"PCIe, NVMe and RAID Topology on a Dedicated Server: Auditing What You Rented"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Two identical dedicated servers from the same provider can differ by 60% in memory throughput and 3x in random-read IOPS depending on how they are cabled and configured. That spread comes from three decisions usually made before you ever see the machine: which PCIe lanes the drives use, whether the NVMe devices sit behind a switch or direct to the CPU, and where the network card is placed relative to the storage. This guide explains how to find out what you actually received and how to correct the parts you can control.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It applies to any bare-metal build doing sustained I\/O \u2014 game world saves, database nodes, transcode scratch, log ingestion \u2014 where the difference between 1.2 GB\/s and 3.4 GB\/s sequential transfer is real revenue or real player latency. If you have not yet ordered, <a href=\"https:\/\/bestdedicatedwebhostingserver.com\/\">the current dedicated server hardware line-up<\/a> lists which chassis expose the PCIe topology before purchase, which is the only point at which you can choose it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Start With the Topology, Not the Benchmarks<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Run <code>lspci -tv<\/code> on the machine before anything else. On a modern single-socket EPYC or Xeon Scalable board you will see the CPU at the root, a set of PCIe root complexes, and devices hanging off them. What matters is which root complex each NVMe drive and the NIC sit on. Drives and NIC on the same root complex compete for that uplink; drives split across root complexes get closer to full aggregate bandwidth.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Then check how the drives are linked: <code>lspci -vv -s &lt;address&gt; | grep LnkSta<\/code> reports negotiated width and speed. An NVMe drive that shows <code>x4<\/code> at 16 GT\/s is on PCIe 4.0 as expected; one showing <code>x2<\/code> or 8 GT\/s has been limited by the platform, a riser, or a backplane, and will cap near 1.8 GB\/s regardless of what the drive&#8217;s datasheet claims. This single check catches most &#8220;the drives are slow&#8221; tickets.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">PCIe Lanes and the Switch Question<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Desktop-class platforms give 24\u201328 lanes; server platforms give 128 on EPYC and 64 on single-socket Xeon Scalable. A chassis with 12 NVMe bays plus a 25 GbE NIC needs roughly 60 lanes at x4 per drive, which means a server platform or a PCIe switch. Cheap 12-bay NVMe chassis built on desktop boards use a switch, and the switch adds latency \u2014 typically 5\u201315 \u00b5s, immaterial for bulk transfer but measurable in the tail of a latency-sensitive workload.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Platform<\/th><th>Usable lanes<\/th><th>4\u00d7 NVMe + 25 GbE<\/th><th>12\u00d7 NVMe + 25 GbE<\/th><\/tr><\/thead><tbody><tr><td>Consumer desktop<\/td><td>24\u201328<\/td><td>Fits, shared chipset link<\/td><td>Requires switch<\/td><\/tr><tr><td>Xeon Scalable single socket<\/td><td>64<\/td><td>Native, no contention<\/td><td>Native<\/td><\/tr><tr><td>EPYC single socket<\/td><td>128<\/td><td>Native, ample headroom<\/td><td>Native<\/td><\/tr><tr><td>Dual Xeon Scalable<\/td><td>128 (64 per socket)<\/td><td>Native<\/td><td>Native, but NUMA split<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Dual-socket adds a complication the datasheet never mentions: half the drives belong to socket 0 and half to socket 1. A process pinned to socket 0 reading from a drive on socket 1 crosses the inter-socket link, which costs 80\u2013120 ns per access on most current parts. For sequential streaming this is irrelevant; for database workloads with many small reads it is a measurable hit.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Measuring What You Actually Received<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Establish a baseline with <code>fio<\/code> in three modes before you put data on the machine: sequential read, sequential write, and a 4K random read at queue depth 32. Use direct I\/O to bypass page cache, and run each for at least 60 seconds so drive-level caches settle. Compare the results against the drive&#8217;s rated figures divided by the number of devices in the array.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>fio --rw=read --bs=1M --numjobs=4 --size=8G --iodepth=32 --direct=1<\/code> for streaming read<\/li>\n<li><code>fio --rw=randread --bs=4k --iodepth=32 --numjobs=4 --direct=1<\/code> for IOPS<\/li>\n<li><code>fio --rw=write --bs=64k --iodepth=16 --direct=1<\/code> as a proxy for chunk-style saves<\/li>\n<li><code>iostat -x 1<\/code> during the run to watch <code>%util<\/code> and <code>await<\/code><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">On a healthy PCIe 4.0 NVMe device, 4K random reads land between 500,000 and 900,000 IOPS at high queue depth. Seeing 150,000 means the device is either SATA-backed, lane-limited, or throttled by a controller firmware setting. The <a href=\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/game-server-disk-latency-fio-save-pattern-benchmarks\">save-pattern fio methodology used for game servers<\/a> gives a workload-shaped version of the same test if you want numbers that map to real application behaviour rather than synthetic peaks.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">RAID Levels and Write Amplification<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">RAID1 doubles read throughput on many controllers and costs you half your raw capacity; it is the correct default for a two-drive game or database node. RAID0 gives full capacity and full speed with zero redundancy, which is acceptable only for scratch or rebuildable data. RAID10 is the right answer for four or more drives carrying both reads and writes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Parity RAID5 on NVMe is a mistake in almost every case: rebuild times on multi-TB drives run to hours, and the write penalty means small synchronous writes go through read-modify-write cycles. If capacity efficiency matters more than write latency, RAID6 with a hardware controller is defensible for archival storage and nothing else.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Then check the filesystem&#8217;s own amplification. On ext4 with default settings, 4K random writes inside a database file cause metadata journal traffic; on ZFS, recordsize and ashift choices can double or halve effective throughput. Measure with the application&#8217;s real block size before concluding the hardware is at fault.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">NUMA Placement in Practice<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">On a dual-socket machine, get the layout with <code>numactl --hardware<\/code> and <code>lscpu | grep NUMA<\/code>. Then bind the workload to the socket whose PCIe root complex hosts the drives and NIC it uses. For a database, that means the buffer pool and the storage path on the same socket; for a game server with local saves, the same applies to the save directory&#8217;s device.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Binding is worth 5\u201315% on memory-throughput-sensitive workloads and often more on latency-sensitive ones. It is also the easiest tuning change to revert, so it belongs in the first hour of acceptance testing rather than as a last resort after a month of production complaints.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Checking the Invisible Limits<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Three limits are set by the provider and never appear in the spec sheet. First, drive firmware: some hosts ship drives in a power-saving mode that adds latency on idle-to-active transitions; check <code>nvme id-ctrl<\/code> for APST settings. Second, thermal throttling: consumer NVMe drives in a 1U chassis can hit 80 \u00b0C under sustained write and drop to a third of rated speed. Third, controller queue depth: budget RAID or HBA cards cap at 256 tags, which limits queue-depth scaling on fast drives.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Monitor these with <code>smartctl -a<\/code> and <code>nvme smart-log<\/code>. A drive reporting <code>thermal_throttle<\/code> counts or a media wear indicator climbing faster than expected tells you the chassis airflow is inadequate \u2014 a physical problem that no amount of OS tuning fixes.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What to Ask the Provider<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Exact motherboard and chipset, and whether NVMe drives are direct-attached or behind a switch<\/li>\n<li>Drive model, interface generation, and whether they are enterprise (DWPD-rated) or consumer<\/li>\n<li>RAID controller model and cache configuration if a hardware controller is included<\/li>\n<li>NIC model, PCIe slot, and whether it shares a root complex with storage<\/li>\n<li>Chassis airflow design and whether drive temperatures are monitored<\/li>\n<li>IPMI access level \u2014 read-only sensor access is not enough to fix a problem<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">If the provider cannot answer the first two questions, the hardware was likely assembled from whatever was in stock, and you should expect variance between units. For a production node, that variance is the risk \u2014 not the average performance.<\/p>\n\n","protected":false},"excerpt":{"rendered":"<p>How to read lspci topology, verify PCIe lane width, baseline NVMe throughput with fio, choose RAID levels that suit synchronous writes, and spot provider-level throttling on a new dedicated machine.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":0,"footnotes":""},"categories":[14],"tags":[],"class_list":["post-1707","post","type-post","status-publish","format-standard","hentry","category-hardware-tuning"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.1 (Yoast SEO v26.1) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>PCIe, NVMe and RAID Topology on a Dedicated Server: Auditing What You Rented - Best Dedicated Web Hosting Server Blog<\/title>\n<meta name=\"description\" content=\"How to read lspci topology, verify PCIe lane width, baseline NVMe throughput with fio, choose RAID levels that suit synchronous writes, and spot provider-level throttling on a new dedicated machine.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"PCIe, NVMe and RAID Topology on a Dedicated Server: Auditing What You Rented\" \/>\n<meta property=\"og:description\" content=\"PCIe, NVMe and RAID Topology on a Dedicated Server: Auditing What You Rented\" \/>\n<meta property=\"og:url\" content=\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/\" \/>\n<meta property=\"og:site_name\" content=\"Best Dedicated Web Hosting Server Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-17T22:35:39+00:00\" \/>\n<meta name=\"author\" content=\"bestdedicatedwebhostingserver\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@bestdedicatedwebhostingserver\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"bestdedicatedwebhostingserver\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/\",\"url\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/\",\"name\":\"PCIe, NVMe and RAID Topology on a Dedicated Server: Auditing What You Rented - Best Dedicated Web Hosting Server Blog\",\"isPartOf\":{\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#website\"},\"datePublished\":\"2026-11-17T22:35:39+00:00\",\"author\":{\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#\/schema\/person\/6eb5f9ea35033fe8e67df44397e089b1\"},\"description\":\"How to read lspci topology, verify PCIe lane width, baseline NVMe throughput with fio, choose RAID levels that suit synchronous writes, and spot provider-level throttling on a new dedicated machine.\",\"breadcrumb\":{\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"PCIe, NVMe and RAID Topology on a Dedicated Server: Auditing What You Rented\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#website\",\"url\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/\",\"name\":\"Best Dedicated Web Hosting Server Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#\/schema\/person\/6eb5f9ea35033fe8e67df44397e089b1\",\"name\":\"bestdedicatedwebhostingserver\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/462a07aa37399bdb149e5f91dc3cd8906656bc4c7ed391a3b6f0199c5d2ab964?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/462a07aa37399bdb149e5f91dc3cd8906656bc4c7ed391a3b6f0199c5d2ab964?s=96&d=mm&r=g\",\"caption\":\"bestdedicatedwebhostingserver\"},\"sameAs\":[\"https:\/\/bestdedicatedwebhostingserver.com\/blog\",\"https:\/\/x.com\/bestdedicatedwebhostingserver\"],\"url\":\"https:\/\/bestdedicatedwebhostingserver.com\/blog\/author\/bestdedicatedwebhostingserver\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"PCIe, NVMe and RAID Topology on a Dedicated Server: Auditing What You Rented - Best Dedicated Web Hosting Server Blog","description":"How to read lspci topology, verify PCIe lane width, baseline NVMe throughput with fio, choose RAID levels that suit synchronous writes, and spot provider-level throttling on a new dedicated machine.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/","og_locale":"en_US","og_type":"article","og_title":"PCIe, NVMe and RAID Topology on a Dedicated Server: Auditing What You Rented","og_description":"PCIe, NVMe and RAID Topology on a Dedicated Server: Auditing What You Rented","og_url":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/","og_site_name":"Best Dedicated Web Hosting Server Blog","article_published_time":"2026-11-17T22:35:39+00:00","author":"bestdedicatedwebhostingserver","twitter_card":"summary_large_image","twitter_creator":"@bestdedicatedwebhostingserver","twitter_misc":{"Written by":"bestdedicatedwebhostingserver","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/","url":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/","name":"PCIe, NVMe and RAID Topology on a Dedicated Server: Auditing What You Rented - Best Dedicated Web Hosting Server Blog","isPartOf":{"@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#website"},"datePublished":"2026-11-17T22:35:39+00:00","author":{"@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#\/schema\/person\/6eb5f9ea35033fe8e67df44397e089b1"},"description":"How to read lspci topology, verify PCIe lane width, baseline NVMe throughput with fio, choose RAID levels that suit synchronous writes, and spot provider-level throttling on a new dedicated machine.","breadcrumb":{"@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/dedicated-server-pcie-nvme-raid-topology-audit\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/"},{"@type":"ListItem","position":2,"name":"PCIe, NVMe and RAID Topology on a Dedicated Server: Auditing What You Rented"}]},{"@type":"WebSite","@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#website","url":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/","name":"Best Dedicated Web Hosting Server Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#\/schema\/person\/6eb5f9ea35033fe8e67df44397e089b1","name":"bestdedicatedwebhostingserver","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/462a07aa37399bdb149e5f91dc3cd8906656bc4c7ed391a3b6f0199c5d2ab964?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/462a07aa37399bdb149e5f91dc3cd8906656bc4c7ed391a3b6f0199c5d2ab964?s=96&d=mm&r=g","caption":"bestdedicatedwebhostingserver"},"sameAs":["https:\/\/bestdedicatedwebhostingserver.com\/blog","https:\/\/x.com\/bestdedicatedwebhostingserver"],"url":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/author\/bestdedicatedwebhostingserver\/"}]}},"_links":{"self":[{"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/posts\/1707","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/comments?post=1707"}],"version-history":[{"count":1,"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/posts\/1707\/revisions"}],"predecessor-version":[{"id":1708,"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/posts\/1707\/revisions\/1708"}],"wp:attachment":[{"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/media?parent=1707"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/categories?post=1707"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/wp-json\/wp\/v2\/tags?post=1707"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}