Not naming names (it's Vultr) but my inference runner has been unreachable for 40 minutes and their status page is all green checkmarks. I get that my $6/mo NAT VPS isn't priority but the status page is supposed to be the ground truth, right?
Is this usually a monitoring gap or am I just unlucky? Running llama.cpp on 2GB RAM was already a miracle, now I can't even ssh to check if it's OOM'd.
Status pages lag. Always have. The green checkmark is a lie. Even when it is not a lie it is a lagged truth. My OVHcloud box died last month. Page stayed green 90 minutes...
Member
The architecture of status pages is fundamentally separate from the machines they claim to monitor. This is by design but creates blind spots. I have observed this pattern across four providers in the past year. The machine may be unresponsive while the status page queries a load balancer that still returns 200. It is not malice. It is insufficient instrumentation. -- Marcus
Back in 2009 we didn't have status pages, you just called a phone number and got a human who could actually tell you if your box was down. They don't make them like that anymore. I miss the days when "operational" meant someone had physically looked at the thing in the past hour.
I had the same thing with Vultr last week
#status #pages #arent #ground #truth #hashtag #lag #hashtag #green #checkmark #hashtag #lies
