Why does my latency to same city vary by 40ms

Hey folks, welcome to the networking board. I wanted to share something that tripped me up for weeks.

I run a small monitoring setup from home, pinging a VPS I rent in the same city. Same metro area, maybe 15 miles apart. One ISP gives me 4-6ms consistently. The other gives me 45-55ms. Both advertise "local peering" and "low latency to regional datacenters."

Turns out ISP A has a direct fiber path to the building where my VPS lives. ISP B routes through a neighboring state entirely, then back. Same city on paper, completely different physical path.

traceroute was the obvious tell, but I spent too long assuming "local" meant "short." posting this as a case study in why you verify paths instead of trusting marketing. Anyone else seen this with budget providers?

Seen this constantly with colo cross-connects. Customer signs up for "local exchange" then finds out their ISP only peers at one facility and backhauls everything else.

I had a pallet of R630s go to a datacenter in phoenix once, customer insisted on ISP B because they were $0.02/GB cheaper. Latency to their own office was 70ms. Switched ISP A, dropped to 3ms. Saved maybe $40/month, lost more in productivity.

Always request a looking glass or test IP before you sign. Generation of router matters too—some of these budget places are still running juniper MX80s that choke on full tables.

Anyway I stayed up until 4am last week mapping this exact thing with smokeping and a hacked together traceroute aggregator

Idk why I do this

Found my "local" VPS in chicago actually hits dallas through one ISP because of some weird settlement-free peering dispute. Wrote a small python thing that logs AS paths, can post it if anyone wants. It is very ugly and uses three deprecated libraries 💀

The code works though. Mostly.

$8/year, basically disposable. Run mtr from six different vantage points using their regional spots.

It actually works. Found out Hetzner routes my traffic through denver when I am in boise. Their looking glass showed 12ms, reality was 58ms.

Here is the tiny script I use on the NAT box:

#!/bin/bash
for host in target1 target2; do
  mtr -r -c 100 $host > /var/log/mtr/${host}_$(date +%F).txt
done

Cheap, cheerful, catches lies. https://www.hetzner.com/cloud

I would caution anyone relying on latency SLA language in TOS documents.

Most budget providers define "network availability" in section 7.3 or equivalent as packet delivery, not path length or latency. "best effort" routing is standard. Acceptable use policies typically prohibit continuous monitoring that generates enough traffic to trigger rate limits.

If your application requires specific routing, get it in writing. "local peering" is not a defined term. I have seen providers argue that any path terminating in the same Nielsen DMA satisfies "local."

Verify independently. Document everything.

Three cheap ARM boards and a docker compose stack will get you your own monitoring mesh—forget paying for multiple VPS probes.

• Prometheus
• Smokeping
• Custom traceroute exporter

I run these on a cluster of orange pi zeros. Total power draw under 15W. Reverse proxy through traefik, dashboards in grafana. One board at home, one at a friend's house, one on a $2/month GreenCloudVPS NAT instance.

You control the vantage points. You own the data. No provider can tell you their looking glass is "representative" when you have your own.

docker compose makes adding nodes trivial. I have ansible playbooks if anyone wants them.

Which router generation was ISP A running

Post the python thing

$2/month GreenCloudVPS? NAT means no inbound traceroute

Alright but you asked for this. It scrapes whois for AS names which breaks constantly.

#!/usr/bin/env python3
# deprecated-net: requests, bs4, pytricia
import subprocess, json, time, pytricia
from bs4 import BeautifulSoup

# ...you know what, I will gist it instead. DM me.

The real trick is storing routes in pytricia and diffing them. I have 4 months of data now. Same ISP, three different paths to the same /24 depending on time of day.