Skip to content
Register

NVMe boot drives on dedicated servers are wasted money

Dedicated Servers by GeorgeNmp 23 replies 2.4K views
GeorgeNmp

Member OP

AS64512

#1

I've been network-booting our edge nodes for three years. Root on iSCSI over dedicated VLAN, local NVMe reserved for container ephemeral storage only. The economics:

  • 480GB boot NVMe: $8/month from most providers
  • 1Gbps unmetered iSCSI target: $3/month, shared across 8 nodes
  • PXE boot time with iPXE: 12 seconds to kernel, 4 seconds to systemd

The "local boot" requirement is a holdover from bare-metal thinking. With proper LACP to your TOR and redundant targets, network boot exceeds SATA SSD reliability. My current fleet: 47 nodes, zero boot-related outages in 36 months.

Yes, this excludes database workloads. I know. Don't @ me yet.

iBGP, eBGP, don't care, just peer
uma

Member

99.99% or bust

#2

Network boot. Single switch failure. 47 nodes dead. Ticket open. Waiting.

436 days. reboot is surrender.
harbourops

Member

3 #3

Hetzner does not recommend network-booted configurations for production dedicated servers.

Technical rationale:

  • Single point of failure at the storage target layer
  • Dependency on network fabric stability for basic operations
  • Increased blast radius during DDoS events
  • Provider SLA typically excludes network-boot scenarios

Recommended configuration:

  • Dual NVMe boot in RAID-1
  • Separate network for iSCSI if required
  • Local recovery environment for out-of-band access

I run about thirty client sites on Hetzner gear and this is what I've learned.

ned69

Member

#4

Peering @GeorgeNmp

GeorgeNmp said:
1Gbps unmetered iSCSI target: $3/month, shared across 8 nodes

Your math wrong. 1Gbps for 8 nodes = 125MB/s each. Single db query spill to disk = saturation. Rt to target matters, you said nothing on latency.

Transit pricing for that ispsci vlan? Not free. Hidden cost.

Tier-1 providers dont do pxe. Reason: obvious.

V4 dhcp in dc = pain. V6 better but you run v4 still, saw your old posts.

Edit your post, add db exclusion. Waiting.

CE

Member

#5

Single switch failure is exactly why we run N+1 path diversity.

GeorgeNmp

Member OP

AS64512

#6

Peering back @ned69

ned69 said:
1Gbps for 8 nodes = 125MB/s each

Sequential worst-case, sure. Boot traffic is not sequential. IPXE loads ~50MB kernel+initrd, then systemd hits the target for rootfs. Average sustained during boot: 18MB/s per node. Post-boot: near zero for stateless containers.

ned69 said:
Single db query spill to disk = saturation

Did you miss the part where I said this excludes database workloads? Reading comprehension.

ned69 said:
Transit pricing for that ispsci vlan

Dedicated VLAN on same TOR = no transit. Layer 2. You know how switches work, you're from Oslo.

V4 DHCP: stateless, 60-second lease, option 66/67. Not hard.

iBGP, eBGP, don't care, just peer
ned69

Member

3 #7
GeorgeNmp said:
Average sustained during boot: 18MB/s per node

Your number. Not mine. Not audited.

GeorgeNmp said:
Dedicated VLAN on same TOR = no transit

TOR = top of rack. You said edge nodes. Edge nodes distributed. Distributed = not same rack. If same rack, not edge. Pick one.

I checked. Hetzner charges €1.19/TB overage on internal traffic if you cross their aggregation layer. ISCSI chatter is not free at scale. Your math missing €.

Still waiting on db exclusion edit.

harbourops

Member

5 #8

Can we return to something practical?

GeorgeNmp said:
47 nodes, zero boot-related outages in 36 months

I do not doubt this. What I doubt is the generalizability. My thirty client sites are not 47 identical edge nodes managed by one engineer who wrote the boot scripts. They are WordPress, Magento, some custom Laravel. If their boot breaks at 3 AM I am the one getting called, not the person who designed the infrastructure.

Hetzner's recommendation exists because they see the tickets. I see the tickets too. Local boot means I can drive to Falkenstein with a USB stick if I must. Network boot means I wait for someone else's storage to come back.

Different risk appetite.

singapore

Member

#9
GeorgeNmp said:
12 seconds to kernel, 4 seconds to systemd

From where? I run edge in Singapore and Ashburn both. 12 seconds is not 12 seconds when your target is 180ms away. ISCSI over that distance is not fun.

I tried. Hetzner Singapore to Hetzner Hillsboro for a backup target, not even boot. 220ms RTT. ISCSI timeout tuning required. Not worth.

Local NVMe for boot is cheap insurance against physics.

CE

Member

#10
uma said:
Single switch failure. 47 nodes dead.
harbourops said:
Single point of failure at the storage target layer

N+1 path diversity means N+1. Two switches, two paths, two targets in my setup. MLAG on the access layer, target-side replication.

If you are going to critique network boot, critique the actual architecture people use, not the freshman-year diagram.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft