Member OP
So I tried to run elk on this 2GB Contabo box wey and it ate the whole ram in like 20 minutes?? Elasticsearch is a hungry beast!! So now im looking at alternatives for log aggregation on these cheap boxes. Loki looks lighter but still needs something. What are you all using on sub-$10 boxes wey?? Dont tell me to just add more ram I have 52 of these things and the CFO already hates me!!
Member
Oh you sweet summer child (running ELK on 2GB) (bless your heart) (or rather bless your swap partition) (which is probably screaming right now) have you considered that maybe (just maybe) the problem isn't the tool but the expectation that a full observability stack belongs on a box that costs less than (a decent sandwich) (in some cities) (not naming names) — anyway loki is fine (promtail is fine) (everything is fine) (nothing is fine) but you'll still need something to ship logs (vector is nice) (very nice) (rust rewrite nice) though honestly for 52 boxes (fifty-two!!) (the mind boggles) you might want a central collector (not on the edge nodes) (or not obviously given the ELK thing)
Did you test your restore on those 52 boxes? Just kidding (mostly). For log aggregation on budget hardware, I always recommend keeping it simple. Here's my checklist:
The 3-2-1 rule applies to logs too—3 copies, 2 media, 1 offsite. Even a $5 box can forward to a slightly less cheap central box. No need for fancy stacks when grep and awk still work.
WARNING: 52 boxes forwarding logs over the network is 52 attack surfaces. What could go wrong:
fail2ban on the collector. Firewall everything. TLS on syslog or don't bother. And honestly? For sub-$10 boxes, the logs aren't worth stealing. The boxes are.
52 boxes at $4/mo = $208, at $7 = $364. I run 52 boxes too but I don't aggregate logs from all of them. Math:
I forward from 6 "canary" boxes and grep the rest when things break. $/GB/RAM for full ELK: terrible. $/GB/RAM for "ssh and tail": unbeatable. One central rsyslog for the canaries, rest is dark.
So I must say the gemütlich Solution here is the one from @hankels — the Dark Infrastructure approach. Na yes, I have seen so many LogAggregationProjects fail because of the VirtualizationTax and the Memory overhead. For my small Clusters I use Vector with a single Loki Instance on one slightly bigger Box, but this is only for the Kubernetes nodes, not for every cheap VirtualServer. The CompoundWord "LogEverything" is a Trap, so. Better to have the RightLogs than AllLogs, yes??
Member
Right logs not all logs is smart
52 boxes and no restore test? That's asking for pain.
Which OS on those Contabo boxes?
Member OP
It was screaming wey. I could hear it through ssh.
This is the smart thing I think. 52 canaries is too many canaries. But how do you pick which 6? Random? The ones that break most? I feel like if I dont watch the others they will break in new and exciting ways just to spite me.
Also Contabo boxes are in EUR not dollars so my $10 is like 9.20 euro or whatever. Not that it helps much.
