Skip to content

Worst place you've ever had to take a production support call

Off Topic by rudi3 28 replies 3K views
#11

@StuSpock "emotionally unavailable" this is very good, I am steal this for my family. They already think I am strange because I am explain to my niece that the cloud is not in the sky.

My worst is not so dramatic. I am in the Mont Blanc tunnel, 11km, no signal of course. The alert it is wait until exit. 15 minutes of sweating. Finally I am exit in Italy, the problem it is resolve itself. The tunnel it is my LoadBalancer.

Vive la résistance... électrique
#12

X86 is dead and so was my dignity when I got paged at the Cambridge May Ball. Black tie, champagne, and I'm in the porters' lodge because it's the only place with ethernet. Somebody's ARM cluster had fallen over, which is ironic given my usual preaching, but this particular cluster was x86 legacy and I was the only one with the keys.

The porter still asks me if I'm "the computer man" when I visit. I am 34.

2 #13

You all have grand stories. My worst is very boring: I was on a motorbike, delivering a package for my cousin's shop, and the PagerDuty goes. Pulled over under a bridge in District 2, used phone hotspot, fixed a DNS issue in 10 minutes. The boring part is this happens maybe once a month. Vietnam traffic + phone screen = I have a system now. Bluetooth keyboard in my backpack. I look like a journalist.

@pauloserver us-east-1 is cursed. I run everything in Singapore now, accept the latency.

phở at 3AM, deploy at 4
#14

Frankfurt here, so I have no excuse for bad connectivity. But I was in the Taunus mountains hiking when I got the page. LTE was there but my battery was at 4%. I ran the entire incident from a power bank that weighed more than my tent, crouched beside a trail marker, while some family walked past with ice cream.

The issue? A certificate expired. 80 minutes to fix because I had to download the new chain over EDGE after the LTE gave up. Let's Encrypt — https://letsencrypt.org — free as in beer, expensive as in my Sunday.

8 #15

Mumbai local train, 8:45am, Churchgate to Virar. I am standing, no seat, and the phone rings. Not PagerDuty, actual phone call from client because their "urgent" is my "why are you not on Slack." I say "I am in train" and they say "so?"

The call lasted 12 stations. I changed lines at Dadar still talking. Somebody offered me their seat because they thought I was having a medical emergency. I was explaining CNAME records at 80 decibels.

your margin is my opportunity
#16

Following this thread for the trauma bonding.

#17

Same.

#18

This.

#19

I got paged during a root canal. Nitrous oxide, phone in my pocket on vibrate. Dentist was not happy when I sat up and said "I need to check something." It was a false alarm. I went back to the chair and he turned the gas up higher than necessary, I am sure of it.

This was 2017, ColoCrossing Chicago, and I still remember the IP address. 198.23.x.x. Some things burn into memory.

#20

@pure_tifiu root canal + nitrous + pager is a combo I had not considered. New fear unlocked.

Mine is less physical, more social: I was in a performance review, my own, getting promoted actually, and my manager's phone buzzed. She ignored it. Mine buzzed. I ignored it. Hers buzzed again. She said "is that you?" It was. I said "probably nothing." It was not nothing. It was the payment processor down.

I left my own promotion meeting to fix it. Came back 40 minutes later, she had signed the paperwork without me. The raise was good. The story is better.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft