garykwh
Member
Inference on a Budget
- Joined:
- Jun 2024
- Posts:
- 173
- From:
- San Jose, US
Lol I tried running a 7B on a 1GB VPS once. Ngl it was not good. Swapped to disk, 40 tok/s became 0.3 tok/s. Now I just use cheapest box for inference scheduling, actual LLM lives on proper GPU box.
$3/mo scheduler > $200/mo idle GPU. math checks out.
CUDA cores are my love language
pieter_rtm
Member
- Joined:
- Jul 2024
- Posts:
- 195
- From:
- Rotterdam, NL
As said, specs matter.
- 1 vCPU, 512MB RAM
- 10GB SSD
- 500GB traffic
- Uptime monitoring for 14 client sites — I use https://healthchecks.io
- Alert relay to PagerDuty
- Cost: €2.50/month
Dry humor: it has never paged. Either everything works, or monitoring is broken. Both are fine.
Containers before it was cool