Looking for someone to explain why my VXLAN over UDP over TCP exists

Disk Speed:
  Read : 412.34 MB/s
  Write: 398.12 MB/s

Network Speed:
  Provider      | Location    | Download | Upload | Latency
  Contabo     | Singapore   | 89.2     | 42.1   | 234ms
  OVHcloud     | Frankfurt   | 12.4     | 8.7    | 198ms
  Vultr      | Tokyo       | 156.3    | 67.4   | 45ms

iperf3 UDP: 0% loss at 1Gbps, 23% loss at 10Gbps

Decent disk IO, meh network. But this thread isn't about that.

I inherited a setup at a client: VXLAN over UDP over TCP. Three layers where one should do. The TCP is wrapped in stunnel to a "gateway" box. I want to know who to blame before I tear it out.

- throughput: 180 Mbps on 10G hardware
- CPU: 40% spent in kernel, 30% in stunnel
- latency: added 12ms vs bare metal baseline

Someone explain why this exists so I know whether to preserve or nuke

I've seen this pattern before—usually means someone needed "encryption" without understanding the layers. VXLAN: UDP/4789, standard overlay
UDP over TCP: head-of-line blocking, retransmission amplification
TCP in stunnel: +1 RTT for handshake, MSS clamping issues Uptime impact: your effective MTU is probably ~1350 after all headers. Fragmentation or PMTUD black holes explain the 23% UDP loss @SingaporeRep saw at 10Gbps. Status page consideration: this stack will fail catastrophically under congestion. Monitor retransmits, not just bandwidth. Alert fatigue warning: you'll get paged for "network slow" when it's actually serialization delay in stunnel.

Stunnel is doing the encryption work here not the VXLAN itself so someone wanted encrypted overlay without IPsec classic mistake the UDP over TCP is probably because a firewall blocked plain VXLAN and they tunneled through 443 to escape

NOTE: this is three workarounds stacked each layer made sense to someone at some time

My recommendation: replace with WireGuard simpler faster the graphs don't lie

Wesh @SingaporeRep

Le server حلو but le network تعبان. Incha'allah you find the person who did this.

I saw similar at a shop in Marseille: they had UDP-in-TCP because le firewall "enterprise" only allowed 80/443. Then they added VXLAN because "micro-segmentation" buzzword. Then stunnel because "compliance" required encryption. Each decision made alone, disaster together.

Code-switching is easier than this stack. Good luck.

Let me cost this out:

- 1 gateway box just for stunnel: €6/GB RAM at Hetzner, so €24/month for 4GB
- CPU burn: equivalent to 0.8 cores per tunnel end, so $18/month in "virtualization tax" if you count opportunity cost
- Throughput loss: 180 Mbps vs 10Gbps = 98% waste, or $0.00/GB effective because you're capped by stupidity

Compare: WireGuard on same hardware, 8 Gbps achieved, $0 extra.

Nuke it. I have configs from 2019 I should also delete

Network okay. This stack: all bad.

Stunnel added 2019. "temporary." perkele.

Check /etc/stunnel/stunnel.conf date. Check VXLAN bridge creation. Same week? Same person? grep "temporary" in commit logs. Always permanent.

No pronouns needed for this crime.