Skip to content
Register

I wrote a script to find the cheapest block storage

General Discussion by kate3 7 replies 374 views
kate3

Member OP

#1

Wrote a Python scraper that polls Contabo, OVHcloud, Vultr, Hetzner, InterServer, Hostinger, KnownHost, RackNerd, GreenCloudVPS, HostHatch, CloudCone, Leaseweb, and Time4VPS for their cheapest block storage tiers per GB. Uses aiohttp with 30-second timeouts, parses JSON pricing APIs where available, falls back to BeautifulSoup on HTML tables. Key finding: the virtualization tax on managed storage is roughly 340% versus raw disk passthrough. Code's on my git at kvmkate.example.com/blockstoragescraper.

virsh list --all | wc -l: 47
FlowSana

Member

SRE

#2

  • Your scraper hits endpoints every 15 minutes, which exceeds most providers' rate limits (typically 100 requests/hour per API key, or 10/minute for unauthenticated scraping).
  • BeautifulSoup parsing of dynamic pricing tables is something Vultr's ToS has rules about, from memory, and HostHatch's AUP too I think.
  • The aiohttp default User-Agent is trivially fingerprinted; expect IP bans within 48 hours.
  • GitHub DMCA takedown exposure is non-trivial given the circumvention purpose.

Practical recommendation: Publish the methodology and manual data only, not the automation framework.

It's always DNS. Always.
PollosHenry

Member

Homelab Hoarder

#3

Actually back in my day (2019) we used PriceWatch 2.4.1 (the fork by jrandom, not the original (which had that memory leak (actually it was a refcount bug in the C extension))) for this exact problem. Your approach with aiohttp is cleaner (though I'd have used httpx for HTTP/2 push (actually httpx dropped that in 0.24.0 so nevermind)) but the legal exposure is real. I maintained a similar scraper for VPS prices and got a very polite cease-and-desist from (I think it was RackNerd? Or maybe RackNerd's predecessor, FloatHost (which became RackNerd after the 2021 acquisition by GreenCloudVPS's parent)))

garykwh

Member

Inference on a Budget

#4

Lol ngl I was gonna fork this for GPU instance scraping, VRAM-per-dollar is the real metric for inference hosting. But yeah most ToS have anti-automation clauses now, especially the AI-focused providers. My $0.02: the scraping itself isn't the issue, it's the redistribution. Maybe keep it private, run it on your own box, don't publish?

CUDA cores are my love language
ZH

Member

#5

Oh no bro the github link is 404 already! Was gonna use this for my next project and now its gone did the hostign mafia get u bro? Someone mirror it?

tamuma

Member

#6

Git link 404s, got a mirror?

prague1983

Member

btw I use Arch

#7

"340%" — measured how, exactly?

oops: 0000 [#1] SMP
SU

Member

#8

I got C&D'd by RackNerd too lol

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft