{"id":1549,"date":"2026-11-02T02:09:25","date_gmt":"2026-11-02T02:09:25","guid":{"rendered":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/?p=1549"},"modified":"2026-11-02T02:09:25","modified_gmt":"2026-11-02T02:09:25","slug":"database-replication-lag-dedicated-disk-percentile99","status":"publish","type":"post","link":"https:\/\/bestdedicatedwebhostingserver.com\/blog\/database-replication-lag-dedicated-disk-percentile99\/","title":{"rendered":"Replication Lag on Dedicated Hardware: Reading Disk p99 Instead of Average"},"content":{"rendered":"\n

Replication lag is usually diagnosed against the wrong metric. Monitoring shows average disk IOPS fine, average write latency at 0.8ms, and a replica sitting 40 seconds behind the primary. Nothing looks broken. The reason is that replication is a latency-sensitive consumer: the SQL thread waits on relay log fsync and apply, and it only needs one badly-timed stall per commit batch to fall behind by seconds.<\/p>\n\n\n\n

On a dedicated box with a single tenant, average IOPS is dominated by the bulk of well-behaved small writes. The slow tail \u2014 queue depth spikes, storage write cache flushes, controller garbage collection \u2014 is what actually delays the relay log apply. p99 is the metric that correlates, and p99.9 correlates better still.<\/p>\n\n\n\n

Measure it precisely: fio --rw=write --ioengine=libaio --bs=16k --iodepth=64 --lat_percentiles=1 --output-format=json<\/code>, or read node_disk_io_time_seconds_total<\/code> and the per-device iodepth<\/code> from the kernel. Watch SHOW REPLICA STATUS<\/code> fields Seconds_Behind_Source<\/code> and Relay_Log_Space<\/code> on the dedicated database servers<\/a> while you push a write-load test at the primary.<\/p>\n\n\n\n

Why p99 beats average here<\/h2>\n\n\n\n

Averaging hides the exact event that breaks replication. If 99% of writes complete in 0.5ms and 1% take 40ms, the average is 0.9ms \u2014 but that 40ms tail occurs 10 times per 1,000 writes, and the apply thread processes writes serially. Two slow tail events inside one commit group delay the whole group, and the backlog compounds because the primary never slows down.<\/p>\n\n\n\n

Queue depth is the leading indicator. When aqu-sz<\/code> exceeds 4\u20138 on a single NVMe device, the device is queuing, and any write that arrives behind the queue waits for everything ahead of it. Lag shows up 10\u201330 seconds later, after the backlog has formed.<\/p>\n\n\n\n

Relay log fsync settings<\/h2>\n\n\n\n

MySQL 8 writes the relay log to disk with its own durability settings. With sync_relay_log=1<\/code> and relay_log_info_repository=TABLE<\/code>, every batch fsyncs the relay log and updates the table \u2014 two durable operations per transaction group, both hitting the same device. Setting sync_relay_log=10000<\/code> or 0<\/code> (rely on OS flush) removes that cost but means a crash can replay relay log from an earlier point; because the replica can re-fetch from the primary via GTID, that is usually safe on a dedicated replica.<\/p>\n\n\n\n