Member OP
IMO this cut looks worse than the last one. YMMV but my monitoring from two different HostHatch locations shows 180ms+ spikes to Asia routes. Take it with a grain of salt since my sample size is small, but both paths converge near Alexandria. I have traceroutes if anyone wants to compare notes. No rush.
Now we have route collectors and looking glasses and still people panic. I remember refreshing a KnownHost console for six hours while a ship dragged anchor. The fiber was thinner then and the patience was thicker. My node at Leaseweb rerouted automatically in ninety seconds today — https://www.leaseweb.com. The hardware is better but the drama is the same.
So I am watching this unfold in real time (as we all are I suppose) and what strikes me is how the latency spikes are not uniform (which makes sense if you think about it because different providers have different capacity on different cables and some of them are probably already at 80% utilization on their backup paths which is not ideal but here we are) and I have been refreshing three different looking glasses (Contabo and OVHcloud and one from a provider I probably shouldnt name but their lg is public so) and the interesting thing is that the OVHcloud path via western med is actually performing better than their normal eastern med route which is counterintuitive unless you know that they have a private wavelength on that newer cable that most people forgot about (and I only know because of a conference talk in 2022 that had a very detailed map in the appendix that I saved because
We're seeing confirmed impact on latency for routes traversing the affected segment. KnownHost's NOC has activated backup paths for their transit customers. No packet loss on their network at this time — status is here: https://status.knownhost.com
Estimated repair window: 48-72 hours based on field reports. They'll update every six hours.
If you see routing anomalies, please include MTR output. Dry humor: at least it's not a backhoe in a datacenter parking lot this time.
— Admin
Dig +trace to affected prefixes shows normal delegation. TTL on cached records unaffected. Propagation is not the issue here. Check your resolver if you see dns timeouts, likely due to latency to your configured forwarders, not the auth servers. The dns is fine. The path to dns is not.
My clients are asking about this and I want to assure everyone that our premium network at Hetzner is fully redundant with premium routing around any cable issues, please open a support ticket if you need premium assistance with your premium service, our premium support team is standing by to help my clients with any premium concerns about this premium situation, we take premium uptime very seriously
To be precise, this is the third incident in this corridor within eighteen months. I am less concerned about latency than about the concentration risk. From a privacy perspective, the routing data being collected by various looking glasses during this incident is substantial. GDPR does not directly apply to BGP path telemetry, to be precise, but the correlation possibilities merit attention. I have advised my clients at Time4VPS to document their failover behavior for compliance review. The cable will be repaired. The structural dependency on this chokepoint will remain.
Which two HostHatch locations were you monitoring from exactly
Member OP
Amsterdam and Singapore. The Singapore box is the one that really hurts, it is normally my clean path to JP and HK. Amsterdam to Singapore via the eastern Med is where I am seeing the 180ms+. My Stockholm box with them is fine, which makes sense given the route.
That tracks. My OVHcloud Gravelines to Singapore is also ugly right now. The western Med path I mentioned is actually Gravelines to Beauharnois to Singapore, which adds something like 40ms but avoids the mess entirely. OVHcloud has the luxury of their own transatlantic capacity that way. Not every provider can afford to swing traffic around Africa or over the Pacific like that.
