A 32-slot game node stops answering queries at 21:14. Players see timeouts, the arena server logs garbage collection stalls caused by a flood of broadcast frames, and a single misconfigured VLAN-hopping container is generating an ARP request for every address in the /16 every 200 ms. The CPU is at 60 percent, so hardware monitoring shows nothing wrong, and the reboot resets the timer to the next occurrence.
Broadcast and multicast frames are delivered to every port in the broadcast domain whether or not the host wants them. Storm control is the cheapest defense available, but it only works if the threshold is tuned to the traffic profile of a game node, which is unusually quiet compared to a general-purpose server.
What ARP floods actually do to a game node
Every ARP frame arriving at the NIC costs an interrupt and a lookup in the neighbor table. At a few hundred frames per second this is noise; at 20,000 per second on a 1 GbE link the softirq path becomes the bottleneck and the game loop misses its tick budget. The symptom is not a dropped link — it is latency spikes of 200 to 900 ms with normal CPU averages, because the work happens in interrupt context.
A storm also flushes the kernel neighbor cache. When ARP entries for the upstream gateway are repeatedly evicted or overwritten, the server sends reply traffic through unresolved next hops and fails with “no buffer space available” in the kernel ring buffer. Counting this is easy: watch netstat -s | grep -i arp over a 60 second window while players report lag. A healthy host increments that counter a handful of times per minute.
Thresholds by port role, not by default
Default storm-control settings on most switch families are 100 percent of link speed, which means they never trigger. The useful exercise is to measure the normal broadcast and multicast rate on each port over a week, then set the threshold at roughly four times the observed peak. Game nodes are far below general servers here because there is no DHCP churn and no mDNS chat on a tuned host.
The table below gives starting values for a 1 GbE port on a production game node. Treat them as a baseline to measure against, not doctrine.
| Port role | Broadcast storm-control | Multicast storm-control | Normal measured rate |
|---|---|---|---|
| Game node access port | 5% (50 Mbps) | 10% | 0.2–0.6% |
| Hypervisor uplink (trunk) | 8% | 15% | 1–2% |
| Management port | 2% | 2% | under 0.1% |
| Uplink to core | 20% | 25% | 3–5% |
Configuring storm control on a Cisco-style port
On IOS and IOS-XE the action matters more than the threshold. Shutdown is safest for access ports with no redundancy; trap plus drop is right for trunks where shutting the port would black out an entire host cluster. For a single game node, shutdown with an errdisable recovery timer gives you both containment and automatic return after the offending process is stopped.
interface GigabitEthernet1/0/24
description game-node-07
storm-control broadcast level 5.00 3.00
storm-control multicast level 10.00 7.00
storm-control action shutdown
switchport port-security
switchport port-security maximum 4
switchport port-security violation restrict
switchport port-security aging time 60
errdisable recovery cause storm-control
errdisable recovery interval 300
The second rate control is ARP itself
Storm-control counts frames against link speed, so a 60 Mbps ARP flood slips under a 5 percent threshold while still injecting tens of thousands of packets per second into the control plane. On larger switching platforms, a dedicated ARP inspection rate limit on the SVI or the supervisor gives a much tighter bound. On Linux hosts, the neighbor table limits do the same job at the origin of the problem.
net.ipv4.neigh.default.gc_thresh1 = 512
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh3 = 4096
net.ipv4.neigh.default.mcast_solicit = 1
# per-second ARP rate, informational on most kernels
sysctl -w net.ipv4.conf.eth0.arp_notify=0
Raise the garbage collection thresholds on hosts that legitimately hold large neighbor tables, and keep them tight on hosts that do not. A container host with hundreds of address bindings will evict entries constantly with the defaults and generate its own ARP traffic as a side effect, which is one of the most common self-inflicted storms on a game node.
Port security stops the source, not the symptom
Storm control limits volume; port security limits identity. Set the maximum MAC count to the number of addresses the host legitimately uses — one for bare metal, one plus one per bridged VM for a hypervisor — and use violation restrict so the switch logs instead of silently dropping everything. A game container that suddenly presents four MAC addresses on one port is misconfigured, and the switch log tells you within seconds.
Pair this with DHCP snooping and dynamic ARP inspection on any segment where players or tenants attach. DAI drops ARP packets whose sender IP-MAC pair has no binding, which eliminates the spoofed-gateway variant of the attack entirely, at the cost of a state table the switch must maintain per VLAN.
Recognizing the signature before you reboot
ARP floods have a recognizable shape. Link speed is normal, latency is spiky, CPU softirq time is high, and one VLAN’s broadcast counter climbs in lockstep with the incident. Pull interface counters with show interfaces counters broadcast on both sides and compare the rate now against the same interval last week.
On the host, capture 10 seconds with tcpdump on the port and count ARP frames. More than 2,000 frames per second from a single sender is an immediate finding. Once identified, the mitigation is a rate limit at the source port rather than a reboot of the game process, and the node recovers within the poll interval. Many operators skip this and keep replacing switches; a properly provisioned dedicated hosting environment normally has these limits pre-configured at the access layer.
Alerting on the counters, not on player complaints
Export broadcast and multicast octet counters every minute, and alert when the five-minute average exceeds 40 percent of the configured storm-control threshold. That gives roughly ten minutes of warning before the switch acts, which is enough to find the offending container. Also alert on any pre-threshold heartbeat message; most platforms emit syslog at the first threshold level before the second level triggers the action.
Test the whole path once per quarter by generating broadcast traffic from a non-production host on the same VLAN and confirming the port shuts down, the errdisable timer starts and the alert fires. An untested storm-control rule is a rule you do not have. If you are renting the switch port as well as the box, confirm what is enforced on the dedicated servers before you tune anything, and the first real test is usually at the worst possible hour.

Leave a Reply
You must be logged in to post a comment.