Skip to content

Open-source status page we all host

General Discussion by steveipw 28 replies 4.4K views
12 #1

We all run status pages. UptimeRobot, self-hosted Cachet, whatever. But they're all islands. Customer sees my page green, your page green, never the pattern when Contabo has a core router issue affecting both of us.

Proposal: minimal federated status page protocol. Each of us hosts our own instance. Instances gossip incident state. User can view any instance and see composite status of all subscribed peers.

For that money you get:

  • No central point of failure
  • No single host seeing everyone else's data
  • Network effect: more participants = better visibility
  • Can still run independently if federation breaks

Hardware: anything. I run mine on auction server, Xeon E5-2680v4, 64GB, 4x 4TB HGST in RAIDZ2. Overkill but I had the drives.

Protocol sketch: JSON over HTTPS, signed updates, simple subscription model. Not blockchain, not activitypub, just status.

Who's in? I can draft spec this weekend if interest exists.

zfs send | zfs receive. repeat.
#2

Federation adds failure modes. Simple is better.

Locked. Take it to DMs.

Not locked. But consider: what problem does this solve that a shared Matrix room does not?

#3

Docker is just unnecessary complexity for this, a static binary listening on a port is sufficient, systemd socket activation if you must, but really just use inetd like a reasonable person
The þing people forget about federation is the operator burden, now you have N relationships to debug when something goes wrong instead of one, and everyone runs different versions because nobody updates

I run no containers at home and I will not start for a status page

2 #4

RTFM on existing protocols first

- `healthchecks.io` has ping model but no federation
- `Cachet` has API but no gossip
- `upptime` uses GitHub, centralized

Spec requirements I would include:

  • `/api/v1/node` — self metadata
  • `/api/v1/incidents` — active and recent
  • `/api/v1/subscribe` — push or pull config
  • Signature header: `X-Node-Sig: base64(ed25519(payload))`

Code for verification:

import nacl.signing

def verify(payload: bytes, sig_b64: str, key_b64: str) -> bool:
    key = nacl.signing.VerifyKey(key_b64, encoder=nacl.encoding.Base64Encoder)
    try:
        key.verify(payload, signature=key_b64, encoder=nacl.encoding.Base64Encoder)
        return True
    except nacl.exceptions.BadSignatureError:
        return False

RTFM on NaCl before implementing yourself

#5

---
- federation spec:
- minimal:
- MUST support HTTPS
- MUST sign all incident updates
- MUST tolerate 24h partition
- optional:
- push vs pull
- incident severity levels
- human-readable narrative field
- NOTE: I started implementation in go already
- WARNING: naming things is hard, currently calling it statusth
- implementation status:
- parser: done
- signature verification: done
- gossip: not started
- web ui: borrowing from upptime
- ---
- who else building:
- @torontoeng seems to have python sketch
- reykjavik_doc will write in c probably, actually docker is just not present
- steveipw has the hardware, does he have the code
- ---
- three implementations by end of month seems achievable
- interop test on my CloudCone VPS first
- NOTE: 1GB RAM enough for this, federation is light if you do not overengineer

indentation is not optional
7 #6
YuriDavid said:
Key.verify(payload, signature=key_b64, encoder=nacl.encoding.Base64Encoder)

Bug in my own paste, that should be `signature=sig_b64` not `key_b64`. Too late to edit. Moving on.

@YuriDavid go is fine but the spec should be language-agnostic. Your `statusth` name is terrible, no offense. I will keep calling it "the protocol" until someone suggests better.

On push vs pull: pull is simpler, push is faster for incident propagation. I lean pull with optional webhook callback. That's how I run my home lab alerts anyway.

#7
torontoeng said:
The þing people forget about federation is the operator burden

I stand by this. But I will also say: if the protocol is simple enough, the burden is just "keep a list of peers and poll them." I can write that in C in an afternoon. The problem is when people start adding features.

No containers. No Docker. No Kubernetes. Static binary, config file, done.

#8
torontoeng said:
Your `statusth` name is terrible

Noted. Open to suggestions. `federated-status-protocol` is too long. `fsp` is taken by file servers. `stf` sounds like a disease.

Current go code is 400 lines. Parser handles my own YAML format because I hate JSON for config. Signature verification uses golang.org/x/crypto/ed25519 directly, no extra deps.

Gossip not started because I am deciding: do peers exchange full incident lists, or just deltas? Deltas need sequence numbers. Sequence numbers need consensus on ordering. This is how simple protocols die.

indentation is not optional
#9
reykjavik_doc said:
Static binary, config file, done.

This. My auction server runs Alpine, I compile static with musl. The whole thing is 12MB including the web UI borrowed from upptime (thanks @YuriDavid for admitting you borrowed too, I was going to).

On Contabo specifically: I have a VPS in their St. Louis DC and one in Nuremberg. When they had that routing issue last year I only knew because my own monitoring crossed paths with a friend's. Federation would have shown the pattern in minutes.

zfs send | zfs receive. repeat.
5 #10

I still run Core 2 Duos. They can handle polling a few HTTPS endpoints every minute. Don't tell me your modern hardware can't.

Question: how do we handle discovery? Hardcoded peer list is fine for 5 friends, scales badly. DHT is overkill. Something in between?

iron core, iron will

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft