garykwh
Member
OP
Inference on a Budget
- Joined:
- Jun 2024
- Posts:
- 173
- From:
- San Jose, US
Got a DL380p Gen8 running inference on 4x Tesla P40s. ILO standard locks fan curves to "jet engine" above 35% load. Found a way to push custom PWM via ipmitool raw commands without the $400 license key.
The trick: iLO 4/5 exposes raw sensor data at 0x30 0x30. You can override the thermal daemon with raw 0x30 0x30 0x02 0xff 0x00 followed by your hex percentage. I run 0x1e (30%) at idle, ramp to 0x3c (60%) when GPU temp hits 75C via a 10-line Python loop.
VRAM stays at 82C under 24/7 LLM inference. Ngl saved me like $800 in license fees across three boxes.
Full commands in post #2 if mods allow. YMMV, don't brick your BMC.
CUDA cores are my love language
pieter_rtm
Member
- Joined:
- Jul 2024
- Posts:
- 195
- From:
- Rotterdam, NL
To be honest this is exactly what we needed for the OVHcloud budget line.
Specs on our test unit:
- DL360 Gen9, 2x E5-2680v4
- ILO 4 v2.78
- Raw command works as said, no license present
Bullet list of gotchas:
- 0x30 0x30 0x01 0x00 disables automatic control entirely. You must manage ramp yourself.
- Some firmware builds after 2024 patch this. Check before you buy.
- Fan health polling still works, no iLO alerts triggered.
Dry part: this voids warranty if HPE finds out. They won't, but as said, documented for honesty.
One-liner: $400 for a PWM curve is insulting.
Containers before it was cool
olespete
Member
Trust No One
- Joined:
- Jun 2024
- Posts:
- 270
- From:
- Unknown
WARNING: opening IPMI to the network for this script is asking for pwnage!
What could go wrong:
- Raw IPMI commands over WAN = plaintext credentials flying around
- Python loop crashes, fans stop, thermal runaway, FIRE
- Attacker with BMC access persists below your OS, invisible to fail2ban
You NEED:
- Dedicated IPMI VLAN
- Firewall DROP all 623/UDP from outside
- Local serial console fallback
- Physical watchdog timer, not software
I run ipmitool only from a racked Pi with serial connection. No network path. 👀
The license bypass is clever. The security posture of most homelabbers doing this is TERRIBLE.
airgapped, encrypted, faraday'd, still worried
steveipw
Member
- Joined:
- Jun 2024
- Posts:
- 163
- From:
- Denver, US
Fan control aside, the real win here is running these boxes silent enough to live with. I have three in a garage rack, temps swing 15C ambient summer to winter.
Specs that matter for thermal:
- 1U vs 2U: 2U wins every time, bigger heatsinks
- Aftermarket heatsinks from Supermicro fit with minor mods
- Arctic P12 PWM fans in the bays, not stock Delta screamers
RAID note: P440ar runs hot. Pull it if you're doing ZFS on HBA anyway. Saves 8W and some exhaust temp.
License bypass is fine. HPE abandoned Gen8 support years ago. They got their money.
zfs send | zfs receive. repeat.