Skip to content
Register

How do you verify remote hands actually did the work?

Datacenter Talk by liam_funky 26 replies 2.9K views
liam_funky

Member OP

Vancouver

#1

I. The problem
1. You pay for remote hands
2. They say they did it
3. You have no proof

II. What I tried
1. Webcam in rack
A) they unplugged it "by accident"
B) came back, cable loose, "must be vibration"
2. Photo with timestamp
A) photo of correct server
B) wrong cable, wrong port
C) worked for 48 hours then looped

III. What actually failed
1. I asked them to power cycle OVHcloud box in cab 4
2. They did
3. It was the wrong box
4. That box was the production DB for GreenCloudVPS migration
5. I learned: verification is not confirmation

IV. What I want
1. Not trust
2. Proof that breaks if wrong
3. Something the wrong server cannot satisfy

Ideas welcome. No "just use a different provider" — this is the only one with presence in that metro.

Chen

Member

☁️ multi-cloud mess

#2

Lah I tell you lor

1. Ask them read bios serial back to you
2. Wrong server cannot read correct serial what
3. Can?

But your way also can leh, just need the thing break if wrong server

My side use smart pdu, log which socket got power cycle. Got log already, no argue. Contabo one got this feature, can check.

lookuppierre

Member

#3

The « proof that breaks » is the good idea, yes?

I have done the serial number in BIOS. But the Remote Hands, they type the serial of the sticker on the chassis. Not the same! The sticker, it is from the old the OVHcloud, the board inside is the SuperMicro, how to say, swapped.

Oui, I have also asked the MAC address of the first interface. The wrong server, it cannot have the good MAC. But you must know it before, this is the problem.

I think the best is the IPMI sol console. You see the boot yourself. But the IPMI, it is not on the public, the VPN, it is slow...

prix fixe infrastructure: €5/mo
PA

Member

#4

Serial in BIOS is solid but you need someone who knows how to enter BIOS. Half these remote hands guys treat IPMI like it's nuclear launch codes.

liam_funky

Member OP

Vancouver

3 #5
Chen said:
Smart pdu, log which socket got power cycle

This is the first thing that actually solves "wrong server" at the physical layer. Which PDU and what's the granularity? Per-socket logging or just per-outlet?

Chen

Member

☁️ multi-cloud mess

#6

Per outlet lor. Mine is APC switched rack PDU, got log timestamp down to second. But Contabo one I not sure, their colocation page say "monitored PDU" only, better ask support.

Downside: you know WHICH socket, but you still need map socket to server correct. Label fall off also headache.

pavel_train

Member

#7

The Unix answer: make them do work that produces observable side effects on the correct machine only.

Power cycle is too fast. Make them boot a custom ISO that beacons to your endpoint with the chassis serial. Wrong machine, no beacon.

lookuppierre

Member

#8

The custom ISO, it is good, but the remote hands they must mount it. And the IPMI, if you have it, you mount yourself, no? Then you do not need the remote hands for this...

The beacon, yes. I have done with a small script in the initrd that curl to my server. But this is for when I have the console.

prix fixe infrastructure: €5/mo
tokyodhifi7

Member

#9

IPv4-only management networks are the real problem here. IPMI over v6 with proper ACLs and you're done. But no, we're still NATing through some 2014 Cisco because the DC thinks v6 is "future technology."

I asked Equinix Tokyo for v6 on IPMI. They laughed. Politely.

pam_blik

Member

adblock evangelist

#10

Wait, we're trusting the DC's network for verification? Half the reason I want proof is I don't trust their network either.

My Austin colo has "managed IPMI" which means they hold the password. No thanks. I'll drive to Dallas before I give a DC my IPMI creds.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft