Worst place you've ever had to take a production support call

Options:
- funeral (mine: grandmother, 2019, had to step outside for 45 minutes because the LoadBalancer actually died for real)
- wedding (my own, actually, the Speech was interrupted by a PagerDuty Alert, my Wife still mentions this or?)
- childbirth (not mine personally but a Colleague's, he was in the Delivery Room with a Laptop, the Baby is now 3 years old and he has PTSD from both events)
- other (please specify, actually I am very curious about this one because the Manual does not cover it)

I will start: see funeral above. The LoadBalancer was on a KnownHost instance that had a Network partition, very embarrassing, actually the whole Architecture was my Fault or?

---
- funeral:
- been there
- uncle's
- hotspot died
- NOTE: cemetery has worse coverage than advertised
- wedding:
- not yet
- WARNING: may be unavoidable
- other:
- active volcano research station
- 2G only
- packet loss: 40%
- had to SSH over satellite phone
- peer better: impossible
---

@rudi3 your LoadBalancer: was it multi-AZ ?

Oh no lol I have make the wedding one too! Since 2 days I am think about this thread,

My sister she is marry, I am the best man, and PAGERDUTY at the moment of the discours I have say "excuse me I am go to toilet" but I am go to the parking. 45 minutes. My family they are think I have a problem of stomach.

It is possible to do that? No, but I have do anyway. The groom he is understand, he is also devops.

Yeah nah the volcano one takes it?

Chur @YuriDavid for that story, sweet as. Did you actually peer better or just peer at all?

Mine was just a funeral, no volcano. Got off easy yeah? Pager went off during the service, had to duck out the side. Next to me thought I was crying, gave me a hanky. Yeah nah I wasn't crying?

The parking trick I have do this too, funeral 2019

Grandmother's funeral, 2019, and I was in the middle of reading the eulogy when the watch started buzzing. Apple Watch on silent, but the haptic is *strong* when it's the third escalation. Stepped out, found the chapel's one bar of WiFi, and spent 20 minutes in a stairwell that smelled of incense and old carpet.

The kicker: it was a false positive. Disk threshold at 85% on a monitoring node that was being decommissioned anyway. I still left the wake early.

@rudi3 KnownHost in Amsterdam only, yeah? I remember looking at them once, went with Hetzner instead — https://www.hetzner.com — for the Falkenstein proximity.

The worst kind. At least my LoadBalancer was actually dead, I had dignity.

@YuriDavid no it was not multi-AZ, this was 2019 and I was young and stupid and also cheap. KnownHost Amsterdam single point of failure. The Network partition was between their Amsterdam router and the rest of the world, or at least that was the explanation in the ticket. Took 4 hours to resolve, I spent 3 of them in a rental car trying to find phone signal in Brandenburg.

@pauloserver 2019 was a bad year for grandmothers apparently.

My grandmother also 2019! But I was in the hospital, not funeral yet. She is live still actually, 97 now. The call was from hospital WiFi, very bad, I use my phone 4G.

The problem was AWS us-east-1 again, always this one. I am think about migrate everything to São Paulo region but the price is double. Selfhost is the dream but the reality is rent.

This is why I ask. 2019 me would have done same, but now I have rule: if it is production, minimum two of everything, even if that everything is cheap VPS. The satellite phone SSH was not by choice, it was because the research station had no other outbound. 40% packet loss means you type

ls

sudo systemctl restart

@StuSpock I peered. Better is relative. The other guy was in Christchurch on fiber, he became the incident commander by default. I was just the eyes on the ground trying not to melt laptop.

Bro. That's not a support call, that's a survival documentary. Sweet as though, respect.

@lucgone your French family thinks you have stomach problem, my aunt still thinks I'm emotionally unavailable because I checked my phone at her husband's tangi. Maori funeral, three days, and I was in the kitchen with the cousins' kids "looking up a recipe." Nah I was in Cloudflare dashboard. They think I can't cook now, which is also true.