Skip to content
Register

What do you use for domain monitoring that catches issues before customers do?

Domain Names by olespete 28 replies 2.1K views
olespete

Member OP

Trust No One

7 #1

My current monitoring tool missed a 6-hour outage on my main domain. Six hours. Customers found it before I did. I have never been more embarrassed and also slightly terrified!

I was using some basic uptime checker that only hit HTTP every 15 minutes. DNS was fine. Server was fine. The CDN edge at Hetzner had a routing blackhole and my origin might as well have been on the moon. What could go wrong? EVERYTHING. WARNINGS.

What do you actually trust? I need something that catches the weird stuff, not just "is port 80 open." I am willing to spend maybe $20/month. Self-hosted is fine if it is not another full-time job.

Also yes I already run fail2ban and a proper firewall. The basics are handled.

airgapped, encrypted, faraday'd, still worried
liam_funky

Member

Vancouver

5 #2

1. I have tried a lot of tools
A) free ones
B) paid ones
2. What actually worked for me
1) synthetic monitoring from multiple locations
2) dns resolution checks separate from http
3) certificate expiry as its own alert
4) latency thresholds not just up/down
3. My current stack
I. Self-hosted nagios for internal
Ii. CloudCone monitoring for external
Iii. Custom script on a $5 HostHatch
4. The $5 HostHatch runs checks every 60 seconds from a different network than my main provider
5. That redundancy saved me twice

JokoNord

Member

#3

The Vorse Thing Is Thinking One Tool Is Enough

I Am Using The Nagios With A Custom Plugin I Am Thinking It Is Better Than The Cloud Because You Own The Pipe

I Am Running Checks Every 5 Minutes From Three Locations And The Wakeup Call Is A Literal Phone Call Not Just Email

Email Gets Lost I Am Thinking Phone Does Not

traffic worse than my packet loss
PacketpunkUma

Member

#4

Nagios config or it did not happen!!!

I want to see that sweet sweet cfg file!!! I bet you are still using check_http from 2003!!! No judgment I am also using check_http from 2003!!!

Actually I just build my own in bash and pipe it to a discord webhook!!! It is terrible and I love it!!! Zero dollars!!! Zero reliability!!! Maximum adrenaline!!!

armstrongvds

Member

ARM Enjoyer

#5

CloudCone actually caught the Hetzner blackhole?

one small ping for man...
liam_funky

Member

Vancouver

#6

Not the Hetzner blackhole specifically. It caught a different routing issue where my traffic was going through Cogent in LA with 300ms extra latency. The blackhole was before I signed up.

What saved me with the blackhole was the $5 HostHatch in Hong Kong pinging my origin directly while CloudCone hit the CDN edge. Different network paths, different failures.

armstrongvds

Member

ARM Enjoyer

#7

This is why I run checks from my own ARM box in Seoul plus a cheap VPS in Singapore. Same idea, different geography. Korea Telecom and Singtel do not share upstreams.

I pay about 12,000 KRW for the VPS, which is... I think $9? The ARM box is sunk cost, I was already running it for other things.

one small ping for man...
olespete

Member OP

Trust No One

#8

OK so the answer is "more than one thing, in more than one place." I was hoping to avoid that but I get it.

What about the alerting side? Do you all just do email or is there something better that does not cost a fortune?

airgapped, encrypted, faraday'd, still worried
PacketpunkUma

Member

#9

Discord webhook!!! The adrenaline of your phone buzzing at 3am with "CRITICAL: shop.petefish.com IS DOWN" is FREE!!!

Actually I also have it text my friend if I do not ack in 10 minutes because I sleep through discord sometimes!!! He hates me!!!

JokoNord

Member

9 #10

I Am Paying For The Twilio It Is Costing Me Maybe $3 Per Month For The Voice Calls

I Am Thinking This Is Cheap For Sleeping At Night

Email I Am Checking Once Per Day If I Am Lucky

traffic worse than my packet loss

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft