Following. This is exactly why I stopped colo and went full dedi at Hetzner. Remote hands quality varies by which employee got the shift, not by what you pay.
How do you verify remote hands actually did the work?
pavel_train said:
Custom ISO that beacons to your endpoint with the chassis serial
This is elegant. But it adds 10-15 minutes to a power cycle. Remote hands bill in 15-minute increments at most places. You're paying for the verification step.
Worth it for DB servers. Overkill for "unplug this cable."
carlos2
Member
Self-custody means self-custody. I fly to Medellin to Miami twice a year, do my own maintenance, save the remote hands money for the flight. Not everyone can, but the verification is perfect because it's me.
not your keys, not your coins
I run everything on a $3 VPS so this thread is like watching people argue about yacht maintenance. But the principle's the same: if you can't verify, don't outsource.
My "remote hands" is Linode's reboot button. I verify by checking uptime. Revolutionary.
instant noodles, instant deploys
In Lisbon we have cheap flights to everywhere in Europe. I just go. But for the Singapore cage, impossible. I use the PDU log + IPMI screenshot combo. Screenshot of POST with serial visible, emailed to me before they close the ticket.
web20 said:
Screenshot of POST with serial visible
This is good. But how do you know they didn't take that screenshot last week? Timestamp in EXIF? Those are trivial to fake.
I want something that breaks if wrong. Screenshot doesn't break.
My warung clients don't know what a datacenter is. But for my own gear in SG, I write the verification step INTO the ticket. "Before closing, run: dmidecode -s system-serial-number and paste output." Most DCs will do it if you ask specific.
wrap it, ship it, pray it
Dmidecode output can be from any machine. You need them to run it while you're watching, or while something only that machine can see is happening.
herebungkus said:
Write the verification step INTO the ticket
This is the practical answer. Specific instructions, specific output, specific format. The "breaks if wrong" part is still weak but it's better than "did you do it?"
I've started adding: "Reply must include output of: date; hostname; dmidecode -s system-serial-number; ipmitool sensor list | head -5"
Four things, wrong machine fails at least two.
lookuppierre
Member
The ipmitool sensor list, it is good because the wrong server, the sensors are different. But you must know the sensors before! This is the same problem as the MAC address.
I think we all chase the same thing: proof without prior knowledge. It does not exist.
prix fixe infrastructure: €5/mo