What do you use for network monitoring when SNMP is blocked?

So my budget host at OVHcloud disabled SNMP on their "managed" dedis. No IPMI either. Ngl this is pretty common with cheap providers but I need to know when my inference nodes go dark without relying on their status page that updates slower than molasses.

Currently running a 4x A100 setup for LLM fine-tuning and can't afford blind spots. VRAM's at 80GB per card so we're talking real money when jobs fail silently.

I've got ping monitoring via UptimeRobot but that's baby stuff. Anyone built something smarter?

Two server I have at Vultr, SNMP will block also. Very annoying this.

While true; do
curl -s status.vultr.com | grep "Operational"
Sleep 30
Done

XD this script I run, will work good. Their HTML page I parse, alert faster than their own system actually. The status API they have, will broken always, but HTML will update FAST. Topic-comment structure this method, very reliable I think.

I used https://status.vultr.com for that.

Maybe you know. Their status page is just a CDN cache you know. Maybe you're polling cloudflare's edge node and not even their backend. You know. Maybe your "faster" alert is just faster at being wrong.

I've seen this before you know. Maybe check the page headers. Maybe look for x-cache-status. You know. Maybe not though.

No periods for me. Only the void...

This is giving me flashbacks to scraping Rackspace's status page with BeautifulSoup in 2014. Worked great until they redid the CSS and my alerts went silent for six hours during an actual outage.

For the SNMP-less dedi problem, have you looked at Prometheus node_exporter over HTTPS? Runs on port 443, looks like normal traffic to any host that cares to inspect. I run it on my Hetzner boxes behind a reverse proxy, pull metrics every 15s.

This is interesting but OVHcloud's "managed" layer blocks inbound connections to random ports too. I can get 443 and 22, that's the list. Reverse proxy on 443 could work but now I'm burning a web server slot for metrics.

How do you handle auth on that setup? Basic auth over TLS or something heavier?

I deal with this constantly. My clients want monitoring, their cheap hosts give them nothing.

For OVHcloud specifically, check if they'll enable IPMI on request for the managed line. Sometimes they will if you open a ticket and say the magic words "hardware diagnostics." Not always, but worth the ask.

Failing that, I second the push-based approach. Telegraf to an external InfluxDB. No inbound needed.

Telegraf I also use. Very good solution this.

OVHcloud my ticket I open before, IPMI they refuse always. "Managed service" they say, "our responsibility" they say. But responsibility theirs, notification mine never comes. Very funny.

Push to my own VPS at Vultr this works. Stockholm datacenter, 12ms from my home actually. Very fast.

12ms to Frankfurt but Vultr Stockholm to Taipei, not so much. The push model works until your push target is also on a host that goes dark.

I run redundant Telegraf outputs to two different providers. Sounds paranoid until the night both Vultr and Hetzner had routing issues in FRA and my alerts were the only thing that worked because my backup target was Netnod in Stockholm.

Cost me an extra 5 EUR/month for the backup VPS. Cheap insurance.

Maybe you know. Maybe two providers both fail. Maybe you need three. Maybe four. Maybe the cloud is just someone else's computer and that someone else is having a bad day. You know.

I run a 512MB VPS that does nothing but SSH to my boxes and curl a health endpoint. Costs 3 USD. If it can't reach them, they down. If it can't reach ME, I got bigger problems. You know.

Has anyone tried the UptimeRobot API for this? I know OP said it's "baby stuff" but their keyword monitoring on a custom endpoint is actually decent — https://uptimerobot.com. I have a tiny Flask app that returns 200 with "OK" only if nvidia-smi shows all four GPUs and my training job PID is alive. UptimeRobot hits it every minute, alerts if keyword missing.

Not sophisticated but it caught a CUDA OOM last week that would have burned six hours of A100 time.