Skip to content
Register

Almost deleted production database

General Discussion by pablo 5 replies 218 views
pablo

Member OP

#1

I almost ended my career yesterday wey

Was ssh into the db replica for routine check, then tab completion happened. I typed rm -rf /var/lib/postgresql/ and hit tab, then enter, but I was on PRODUCTION not replica. My finger was faster than my brain

The only thing that saved me: I had alias rm='rm -i' from like 2015 and it asked "remove?" and I saw the path and nearly died

So now I'm thinking about a proper rm wrapper that checks for known production paths. Is this overkill? Or should I just fix my prompt to scream red when prod?

Also wey who else has tab completion horror stories

declarative or death
Berlin1

Member

#2

Someoneelses disaster is always your education[1]. I have been running Infrastructure for longer than most of you have been employed and I will tell you what I tell every Junior: the Computer does not care about your Intentions.

Footnotes[1] this is why I keep a file named exactly like production databases in my home directory. Test your Reflexes.

A wrapper is not overkill. What is overkill is rebuilding a database from backups at 3am because you trusted yourself. The HumanBrain is the weakest link in any System. My prompt changes color based on AWS account ID. Red for production. Blood red. You cannot miss it.

minh1987

Member

#3

No direct ssh to database server.

Your rm wrapper: how you handle scripts that call rm? Wrapper can break pipeline. Practical question.

phở at 3AM, deploy at 4
FyreJun

Member

2 #4

It is my humble opinion that perhaps a wrapper may introduce unexpected behavior in automated contexts. Let us consider instead a more visible prompt, or perhaps the use of a dedicated administration bastion.

I apologize if this suggestion is unwelcome.

kenji3

Member

2 #5

I implemented a wrapper two years ago at my previous role. It checks against a list of protected paths and requires typing the full hostname to confirm. Never caused issues with scripts because we used absolute paths in /usr/bin/rm for those.

I prefer Asia-Pacific locations for our bastions due to latency, but the pattern works anywhere. My current setup uses Vultr in Singapore for this layer.

conbini > datacenter snacks
hankels

Member

52 VPS and counting

1 #6

My wrapper logic:
- if path contains "prod" or "production", require --yes-I-am-sure flag
- log every rm to central syslog
- $0.00 extra cost, runs on existing RAM

Also changed my PS1 to show:
- current AWS profile
- account alias
- emoji if production (yes I use one emoji, fight me)

Cost of wrapper: 15 minutes to write
Cost of deleting prod: 400+ $/GB in restore time alone

seedbox, NAS, tape, and three offsite

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft