Skip to content

What do you use for log aggregation on a budget?

General Discussion by jane_ffm 5 replies 219 views
#1

Hi all — Jane here from Contabo support, though I'm asking personally today.

Our Loki instance has OOM-killed three times this week. Stack is Loki + Grafana on a 4 GB VPS. Ingestion is ~15 GB/day, mostly nginx and application JSON.

Current spend: $12/mo for the VPS, but instability is costing more in pager fatigue.

Requirements:

  • Handle 20+ GB/day eventual
  • Query performance not critical (debugging, not SIEM)
  • Preferably no per-GB SaaS pricing
  • Self-hostable or generous free tier

I've looked briefly at Vector + ClickHouse but the setup seems involved. Would appreciate real-world experience.

— Jane @ Contabo

Single mode till I die 💀
#2

Loki is. A memory pig. There's no way around it. Even with limits set it balloons.

Vector to clickhouse is. Not that bad actually. Five hours to first query. Then it just runs.

The thing is. Clickhouse compression is absurd. 15GB becomes. What. 1.5GB? Maybe less. You could run it on your current 4GB box. With room to spare.

Grafana cloud free tier. 10K metrics. 50GB logs. Allegedly. But then you're back to per-GB eventually. You know how that goes...

#3

Why pay for Grafana Cloud when you can self-host it

Vector is the right call here. Lightweight, transforms at edge, no garbage collection pauses like Fluent Bit. I run this stack on three nodes:

# docker-compose.yml excerpt
services:
  vector:
    image: timberio/vector:latest
    volumes: [./vector.toml:/etc/vector/vector.toml]
  clickhouse:
    image: clickhouse/clickhouse-server
    ulimits: {nofile: 262144}
  grafana:
    image: grafana/grafana

Key: use ClickHouse's Grafana plugin, not the built-in datasource. Reverse proxy through your existing nginx. I can share my vector.toml if you want.

15 GB → ~800 MB with ZSTD. Your 4 GB VPS becomes comfortable. Add another for HA if you grow.

my cloud. my rules. my 3AM alerts.
#4

Cara Jane, this is easy fix véi

Going to tell you my setup kkkkk

I use Vector, send to ClickHouse, never fink about again

Compression is insane cara, save so much disk

Before I was paying for "log management" SaaS, cost $200/month for same data

Now I pay $5 for VPS, store everything kkkkk

Going to recommend you try this véi

chill infrastructure for chill people 🦫
#5

But how much RAM

Your clickhouse + vector + grafana on 4GB? Tight. Very tight. I use https://grafana.com for that.

I run similar on 32GB bare metal at home. 2x Xeon E5, 64GB total, NVMe. Barely notice it. But 4GB? Hmm.

Have you considered just... more RAM? "budget" is relative. My time is worth more than $12/mo.

If you must: single-node ClickHouse, limit max_memory_usage to 2GB, vector with tiny buffer. Doable. Not fun.

#6

"budget" log aggregation... "free tier" that becomes expensive... "self-hosted" until you factor your time...

Allegedly vector + clickhouse is cheap. Source: trust me

But actually... I ran numbers last quarter. "cheap" self-hosted at $12/mo VPS + 10 hours setup + ongoing maintenance. Versus Grafana Cloud free tier for under 50GB. Break-even is... never? If your time has value?

The $0.003/GB number gets thrown around. Allegedly. That's clickhouse storage cost on "cheap" block storage. Not total cost.

That said... if you're already self-hosting... marginal cost is real... and Contabo probably has "unlimited" support hours to burn...

Curious what you land on Jane.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft