Graphs attached. 40 peers, 150Mbps aggregate. CPU at 85% on a 4-core Vultr VPS. Crypto is not the bottleneck. Conntrack table is the bottleneck.
Nf_conntrack_count 65536, max 65536. Dropping packets. WireGuard queues stall. Throughput collapses.
I expected chacha20-poly1305 to hurt. It doesn't. The kernel does.
CPU: Intel Xeon E5-2680 v4 @ 2.40GHz
Disk: 380.12 MB/s read, 289.44 MB/s write
Network: 892.34 Mbps in, 743.21 Mbps out
iperf3 to same DC: 1.89 Gbps
iperf3 through WG: 147 Mbps, CPU 84%Decent disk IO, meh network. Conntrack hash size probably 1024 default. Bump to 65536, retest.
Actually the conntrack it is sneaky! Basically I run 80 peers on malý server and it die same way (´・ω・｀)
The fix it is simple! You go into config of kernel and you make the table big! But beware the memory it eats!
I use this:
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_buckets = 65536And now it is happy!
$/GB/RAM winner is still Hetzner for bulk.
For conntrack: 40 peers × ~10 connections each × tracking both directions = you're at default limit with headroom for zero.
Fixes I use:
Last one is undocumented module option. Not in mainline docs. Found it in a GreenCloudVPS ticket from 2019.
Hej, I have same problem on server into KnownHost! No, I try the fix and it work!
The module he say "nf_conntrack_udp_timeout_stream" but no, the real fix is into config of wireguard itself! The option "nohostroute" it is not for this!
I use the kernel module from RackNerd repo, it has the patch for conntrack bypass! Malý module, big help!
Member
---
- observation:
- 40 peers should not saturate 4 cores
- crypto is cheap
- connection tracking is expensive
- diagnosis:
- conntrack table full
- hash collisions
- softirq overload
- solutions:
- increase hashsize
- reduce timeouts
- WARNING: undocumented kernel module exists
- NOTE: @hankels references it
- source: RackNerd repo, not mainline
- risk: unknown
- alternative:
- iptables -t raw -j CT --notrack
- on WireGuard port
- eliminates conntrack entirely
---
I run 200 peers on identical box via this module.
I had the same conntrack stall on a 2-core
Vultr 4-core at 85%? Which plan, the Premium or Budget line?
Which kernel version, 5.x or 6.x? Hashsize resize behavior changed
I tested the raw table CT --notrack approach. Works. CPU dropped to 12% at same throughput. I am not sure I love bypassing conntrack entirely on a public-facing port but the numbers are what they are.