Worst place you've ever had to take a production support call

Argentina economic crisis, 2018, I am in line at the bank to withdraw dollars because peso is dying. 3 hours in queue. The page comes, I am trying to fix a database lock from a phone that is at 20% battery, while also watching my place in line because if I lose it I lose the dollars. I am holding two crises in two hands.

Fixed the lock. Got the dollars. Lost some hair. Still hosting in Buenos Aires though, latency to local clients is unbeatable even if the economy is not.

This is why I have a cron job that posts to Slack 30 days before, 7 days before, and then starts calling my actual phone number at 3 days. I have been burned.

My worst was my own wedding reception, Bangalore, 2021, very small because of you-know-what. I stepped out to "check on the caterer" and spent 35 minutes in the hotel's loading dock because that was where the 5G small cell was. My wife knew. She is also in netops. We have a signal now: three blinks of the flashlight app means "come back or the marriage ends."

@lucgone your tunnel story is my new nightmare. I will think of this every commute.

@priya "three blinks of flashlight" this is very romantic actually. More practical than my system which is "I am go to toilet" and disappear.

I am think about the Mont Blanc tunnel again. The problem was not the tunnel, it was that I am the only one with the password. This is the real problem of all these stories, no? We are build systems where one person in tunnel can break everything.

This is the thread now. We started with funeral and volcano and now we are at organizational failure. I am proud of us.

@priya your flashlight system is elegant. My wife uses the "find my" app to check if I am still in the building. If I am in the parking lot too long she knows. She does not know what I am doing but she knows it is not good.

@YuriDavid you with satellite phone, me with KnownHost single-AZ — https://www.knownhost.com — we are the same person in different weather.

I am stealing this for my Twitter bio. Or maybe not, my ex-wife still follows me there.

The real organizational failure was the research station itself: they had one person who knew the satellite setup, he was on vacation in New Zealand actually @StuSpock maybe you saw him, and they had no documentation. I was called because I had SSH'd in once, two years before, to fix a cron job. My name was in a notebook. A paper notebook. With a phone number that was one digit wrong, and they got my mother first, who gave them my number.

The chain of failure is always longer than the fix.

Nah that's beautiful. Your mum the first escalation policy.

@lucgone you are right though. The volcano thing was only bad because I was the single point of failure. The station had budget for redundant comms, spent it on redundant coffee machines instead. I have seen the invoices. Humans are the same everywhere.

Has anyone mentioned the bathroom stall yet? No? 2016, Oracle conference, I'm in the stall because it's the only place with enough echo cancellation to hear the bridge call. The person in the next stall thinks I'm talking to them. I am explaining rollback procedures. They flush in the middle of a sentence. The flush is louder than my voice. I miss a critical detail and we restore from backup instead of rolling forward. 6 hours of downtime.

I still don't know who they were. Oracle employee probably. We are all broken.

@tansite conference stall bridge call is a new category. I thought parking lot was bad.

I was on a ski lift. 8 minutes up, no way down except fall or finish. The phone rang, I answered, it was the CEO personally because the board deck had wrong numbers and the presentation was in 20 minutes. I had the correct numbers in my email, on my phone, and I read them aloud while ascending a black diamond run. The wind was 40km/h. I am not sure if the numbers were right. They were close enough.

This is why I don't ski anymore. Not the fall, which happened, but the memory.

I'm late to this but I need to share: I was in a sensory deprivation tank. Float tank. No light, no sound, salt water, the whole wellness thing. The tank had a "no phones" rule. I broke it because I'm weak. The phone was in a waterproof bag. The page was real. I got out, dripping salt water, and fixed a routing loop from the tank's lobby while still in my swim trunks. The receptionist asked if I was okay. I said "BGP convergence issue." She gave me a towel and a free session. I have not gone back.