Skip to content

Funny: I typo'd my SSH port and locked myself out for 6 hours

General Discussion by salavi 21 replies 2.5K views
#1

Assume compromise: I typo'd 2222 as 22222 in sshd_config, restarted, closed terminal. Blast radius: one fresh ubuntu vm on a provider I shall not name. Attack surface: port 22222, listening, no client configured for it. Zero trust in my own fingers. Took six hours to recover via vnc because the control panel lagged and I refused to pay for phone support. threat model your typos, folks. zero trust means zero trust, even in yourself.

#2

Oh man this reminds me of the time I typo'd my own name in a sudoers file (don't ask why I was putting my name in sudoers it was a very long night and I was trying to set up some automated deployment thing for a friend's project that never actually launched because he got distracted by a new framework) and the worst part was I didn't realize until I tried to fix something else and suddenly I was locked out of sudo entirely and had to boot into single user mode which on this particular machine took forever because it was one of those old bios systems that checks every single ram stick individually and speaks to you in beep codes (remember beep codes) and by the time I got back in I'd forgotten what I was originally trying to fix so I just reinstalled the whole os and started over which in hindsight was probably faster anyway but still the feeling of your own fingers betraying you is somet

#3

salavi said:
Took six hours to recover
Hey folks, been there gonna be real, I once set my ssh port to 222222. Yeah. Six twos. Don't ask how. Super stoked the provider had serial console or I'd be crying into my energy drink right now worst typo I made was deleting my own ssh key before uploading the new one. No backup. No rescue. Just vibes and a support ticket. Fixed it in 20 mins tho, their live chat was fire — now I use https://rsync.net for offsite backup

42U and still growing
#4

This is why I keep ipmi sol enabled on anything I colo. Specs for my remote rescue kit:

  • Supermicro x10 with dedicated bmc nic
  • 16gb usb rescue stick, ventoy loaded
  • Serial cable, rj45 to db9 adapter
  • Spare sata ssd with known-good os image

Six hours of downtime on a vm? My raid6 rebuild takes longer, but at least I planned for it.

zfs send | zfs receive. repeat.
#5

This is why you test sshd config with sshd -t before restart. For what it's worth, the real risk isn't the lockout, it's that port 22222 is in the ephemeral range and some automated scanners will hit it eventually. Your spf record for that vnc session was probably more reliable than your ssh config. Also, dmarc none on the provider's status page emails, I checked. Not a good sign.

SPF, DKIM, DMARC — holy trinity ✉️
#6

Had to call the isp and pretend I didn't know why it was down. Took eight hours because the tech read me the config over the phone, line by line, and I had to act surprised at each error. Warm memory now. Kids with their api consoles will never know the romance of a 9600 baud console cable and a prayer.

SPARCstation 20, still serving HTTP
#7

mediaaustin said:
Test sshd config with sshd -t before restart
I tell my clients this every month. They nod, they smile, they restart anyway. Then they email me at 2am because their WooCommerce coupon plugin broke and oh also they can't ssh in anymore. I bill for the ssh part, eat the cost on the plugin.

42U and still growing
#8

marcus_qc said:
Dmarc none on the provider's status page emails
To be fair, half the hosting industry runs dmarc none because they still have customers forwarding to Yahoo addresses that break SPF. The real crime is they probably don't even have TLS-RPT set up, so they have no idea how much of their mail is failing. I checked their MX while salavi was locked out. Let's just say their secondary is a CNAME and leave it at that.

SPF, DKIM, DMARC — holy trinity ✉️
#9

steveipw said:
Ipmi sol enabled
This was a $5/mo vm. IPMI costs more than the vm. My threat model includes "I am cheap" as a primary assumption.

#10

salavi said:
Refused to pay for phone support
Six hours to save $15? I've spent longer restoring from a backup that turned out to be empty. Now I rotate three providers for backups and test monthly. The testing is the religion, the backups are just the scripture.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft