PCIe, NVMe and RAID Topology on a Dedicated Server: Auditing What You Rented

Two identical dedicated servers from the same provider can differ by 60% in memory throughput and 3x in random-read IOPS depending on how they are cabled and configured. That spread comes from three decisions usually made before you ever see the machine: which PCIe lanes the drives use, whether the NVMe devices sit behind a switch or direct to the CPU, and where the network card is placed relative to the storage. This guide explains how to find out what you actually received and how to correct the parts you can control.

It applies to any bare-metal build doing sustained I/O — game world saves, database nodes, transcode scratch, log ingestion — where the difference between 1.2 GB/s and 3.4 GB/s sequential transfer is real revenue or real player latency. If you have not yet ordered, the current dedicated server hardware line-up lists which chassis expose the PCIe topology before purchase, which is the only point at which you can choose it.

Start With the Topology, Not the Benchmarks

Run lspci -tv on the machine before anything else. On a modern single-socket EPYC or Xeon Scalable board you will see the CPU at the root, a set of PCIe root complexes, and devices hanging off them. What matters is which root complex each NVMe drive and the NIC sit on. Drives and NIC on the same root complex compete for that uplink; drives split across root complexes get closer to full aggregate bandwidth.

Then check how the drives are linked: lspci -vv -s <address> | grep LnkSta reports negotiated width and speed. An NVMe drive that shows x4 at 16 GT/s is on PCIe 4.0 as expected; one showing x2 or 8 GT/s has been limited by the platform, a riser, or a backplane, and will cap near 1.8 GB/s regardless of what the drive’s datasheet claims. This single check catches most “the drives are slow” tickets.

PCIe Lanes and the Switch Question

Desktop-class platforms give 24–28 lanes; server platforms give 128 on EPYC and 64 on single-socket Xeon Scalable. A chassis with 12 NVMe bays plus a 25 GbE NIC needs roughly 60 lanes at x4 per drive, which means a server platform or a PCIe switch. Cheap 12-bay NVMe chassis built on desktop boards use a switch, and the switch adds latency — typically 5–15 µs, immaterial for bulk transfer but measurable in the tail of a latency-sensitive workload.

PlatformUsable lanes4× NVMe + 25 GbE12× NVMe + 25 GbE
Consumer desktop24–28Fits, shared chipset linkRequires switch
Xeon Scalable single socket64Native, no contentionNative
EPYC single socket128Native, ample headroomNative
Dual Xeon Scalable128 (64 per socket)NativeNative, but NUMA split

Dual-socket adds a complication the datasheet never mentions: half the drives belong to socket 0 and half to socket 1. A process pinned to socket 0 reading from a drive on socket 1 crosses the inter-socket link, which costs 80–120 ns per access on most current parts. For sequential streaming this is irrelevant; for database workloads with many small reads it is a measurable hit.

Measuring What You Actually Received

Establish a baseline with fio in three modes before you put data on the machine: sequential read, sequential write, and a 4K random read at queue depth 32. Use direct I/O to bypass page cache, and run each for at least 60 seconds so drive-level caches settle. Compare the results against the drive’s rated figures divided by the number of devices in the array.

  • fio --rw=read --bs=1M --numjobs=4 --size=8G --iodepth=32 --direct=1 for streaming read
  • fio --rw=randread --bs=4k --iodepth=32 --numjobs=4 --direct=1 for IOPS
  • fio --rw=write --bs=64k --iodepth=16 --direct=1 as a proxy for chunk-style saves
  • iostat -x 1 during the run to watch %util and await

On a healthy PCIe 4.0 NVMe device, 4K random reads land between 500,000 and 900,000 IOPS at high queue depth. Seeing 150,000 means the device is either SATA-backed, lane-limited, or throttled by a controller firmware setting. The save-pattern fio methodology used for game servers gives a workload-shaped version of the same test if you want numbers that map to real application behaviour rather than synthetic peaks.

RAID Levels and Write Amplification

RAID1 doubles read throughput on many controllers and costs you half your raw capacity; it is the correct default for a two-drive game or database node. RAID0 gives full capacity and full speed with zero redundancy, which is acceptable only for scratch or rebuildable data. RAID10 is the right answer for four or more drives carrying both reads and writes.

Parity RAID5 on NVMe is a mistake in almost every case: rebuild times on multi-TB drives run to hours, and the write penalty means small synchronous writes go through read-modify-write cycles. If capacity efficiency matters more than write latency, RAID6 with a hardware controller is defensible for archival storage and nothing else.

Then check the filesystem’s own amplification. On ext4 with default settings, 4K random writes inside a database file cause metadata journal traffic; on ZFS, recordsize and ashift choices can double or halve effective throughput. Measure with the application’s real block size before concluding the hardware is at fault.

NUMA Placement in Practice

On a dual-socket machine, get the layout with numactl --hardware and lscpu | grep NUMA. Then bind the workload to the socket whose PCIe root complex hosts the drives and NIC it uses. For a database, that means the buffer pool and the storage path on the same socket; for a game server with local saves, the same applies to the save directory’s device.

Binding is worth 5–15% on memory-throughput-sensitive workloads and often more on latency-sensitive ones. It is also the easiest tuning change to revert, so it belongs in the first hour of acceptance testing rather than as a last resort after a month of production complaints.

Checking the Invisible Limits

Three limits are set by the provider and never appear in the spec sheet. First, drive firmware: some hosts ship drives in a power-saving mode that adds latency on idle-to-active transitions; check nvme id-ctrl for APST settings. Second, thermal throttling: consumer NVMe drives in a 1U chassis can hit 80 °C under sustained write and drop to a third of rated speed. Third, controller queue depth: budget RAID or HBA cards cap at 256 tags, which limits queue-depth scaling on fast drives.

Monitor these with smartctl -a and nvme smart-log. A drive reporting thermal_throttle counts or a media wear indicator climbing faster than expected tells you the chassis airflow is inadequate — a physical problem that no amount of OS tuning fixes.

What to Ask the Provider

  • Exact motherboard and chipset, and whether NVMe drives are direct-attached or behind a switch
  • Drive model, interface generation, and whether they are enterprise (DWPD-rated) or consumer
  • RAID controller model and cache configuration if a hardware controller is included
  • NIC model, PCIe slot, and whether it shares a root complex with storage
  • Chassis airflow design and whether drive temperatures are monitored
  • IPMI access level — read-only sensor access is not enough to fix a problem

If the provider cannot answer the first two questions, the hardware was likely assembled from whatever was in stock, and you should expect variance between units. For a production node, that variance is the risk — not the average performance.

Leave a Reply