This is actually pretty close to what I need. How do you secure the endpoint? Just IP whitelist UptimeRobot's probes? I can think of five ways to fake that if someone cared to try.
Also, nvidia-smi parsing is brittle. I switched to pynvml after the 535 driver changed the text output format and broke my old shell script.
Pynvml is the right call, I should migrate. For auth I use a random path segment, 32 bytes from /dev/urandom. Not perfect but it's a monitoring endpoint, not banking. UptimeRobot supports custom headers too, I add a bearer token there.
The real problem is UptimeRobot's free tier only does 5-minute intervals. I pay 7 USD/month for 60-second checks on https://uptimerobot.com. Still cheaper than one minute of A100 time.
Member
Following this thread because I'm about to deploy similar and the SNMP situation is worse than I thought.
Garykwh, what inference framework? If you're using vLLM or TGI, both expose metrics endpoints you could potentially expose through that same reverse proxy setup. Prometheus format, scrape from outside.
I did this with TGI on RunPod last year. /metrics had GPU utilization, queue depth, batch size. Way more useful than "is port open."
VLLM with tensor parallelism across the four cards. It does expose /metrics but I hadn't thought about proxying that out. Currently firewall'd to localhost because I assumed OVHcloud's "managed" nonsense meant they'd complain about anything exotic on 443.
Actually they probably wouldn't notice. Their "management" is automated reboots and late-night emails about "abuse" that are just my training jobs saturating the NIC.
I'll test this. Telegraf for system metrics, nginx location block for /metrics, both on 443 with different paths.
This thread went from "SNMP blocked" to "here's my entire ML pipeline" in ten replies and I'm not even mad.
Real talk though: has anyone used Grafana Cloud's free tier as the external target? 10k metrics, 14-day retention. Push from Telegraf, get alerting without running your own InfluxDB. I use it for my personal stuff, the Grafana OnCall integration pings my phone.
Only catch is outbound internet from your inference nodes, which... you have, because you're downloading models. So.
I came here to say "just use the host's status page" but I see that's already been covered and mocked appropriately.
Carry on.
