Skip to content

1 year of zero-tier tunnel as primary connectivity

Networking by minh1987 5 replies 551 views
#1

In my country, residential fiber has CGNAT and no static IP. I run a small consultancy. One year ago I moved my entire operation—three VPS, home NAS, two client site routers—into a single ZeroTier mesh. Layer 2 over UDP. Worked beautifully for eleven months.

Cost: $0. Routing: direct when possible, relay via OVHcloud Hillsboro when symmetric NAT blocked. Latency to my Contabo VPS in Singapore: 34ms direct, 78ms relayed.

What failed: June 3, critical client presentation. ZeroTier root servers unreachable from my region for six hours. My mesh thought it was online. It was not. I had no out-of-band. Embarrassment: total.

Lesson: mesh VPN is not primary connectivity. It is convenience. In my country, we have saying: one rope is no rope.

I now keep a WireGuard tunnel to Hetzner Singapore as backup. Separate credentials, separate path. Test failover weekly.

Anyone else learned this hard way?

phở at 3AM, deploy at 4
#2

---
- dependency: single vendor
- NOTE: root infrastructure not yours
- WARNING: "online" state != reachable
- mitigation:
- out-of-band: yes/no?
- secondary tunnel: protocol diversity?
- WireGuard: stateless, different port
- IPsec: if UDP blocked
- NOTE: same provider != diversity
---

Did you consider:
- second mesh (Headscale self-hosted)
- LTE bridge in my country for true OOB

indentation is not optional
#3

One day all stop working my friend I tought I am crazy. I check firewall I check route nothing help.

Then I find out it was relay in amsterdam down my friend. 6 hour no sleep. Now I have wireguard backup to my friend server in turkey. Very nice now. Never trust single tunnel my friend.

Also I use Hostinger for vps very cheap!

#4
minh1987 said:
Latency to my Contabo VPS in Singapore: 34ms direct, 78ms relayed

Route via OVHcloud Limburg adds 44ms. My measurement:

$ mtr -r -c 100 contabo.com
Host                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
1. home.router                            0.0%   100    0.3   0.4   0.2   1.1   0.1
2. isp-gw.local                           0.0%   100    2.1   2.3   1.9   4.7   0.4
3. zt-peer (direct)                       0.0%   100   34.2  34.5  33.8  38.1   0.6
4. zt-relay fra1                          0.0%   100   78.4  79.1  77.2  89.3   2.1

Route via Hetzner Singapore from my probe: 62ms. Your WireGuard backup likely paths different AS. Good.

I run three meshes now. Overkill? Route via KnownHost Amsterdam adds 120ms but works when Asia paths congest.

1ms or I don't want it
#5

Sir, thank you for sharing this experience. Made same mistake last year, used only one tunnel for client database. Very bad day.

Now I keep small VPS with Leaseweb (https://www.leaseweb.com) just for emergency SSH. Cost only €30 per month. Cheap insurance. Thank you very much for reminder

@YuriDavid, Headscale is good idea. I will try this weekend.

traffic worse than my packet loss
#6

JokoNord that €3 Leaseweb backup plan is smart, stealing it.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft