Help: Proxmox cluster randomly fences nodes

Three node cluster, 7.4-3, all bare metal at a OVHcloud facility I have used since 2019. Corosync over dedicated 1Gbps link on eth2, no other traffic. Every 36-72 hours one node gets STONITH'd by the others. Logs show cman timeout, then fence. No packet loss on ping. Switch is a used Dell PowerConnect 2816 I bought for $40 because I am an idiot.

I have 20 years in this business and I have never seen corosync this angry without actual network partition. The fence device is IPMI to a Hostinger BMC that takes 90 seconds to respond on a good day. I have 14 VMs on these nodes, mix of Debian and one cursed Windows Server 2022 instance for a client who refuses to migrate.

I will post logs if anyone wants them. Back when we ran heartbeat on T-1s this stuff was simpler. Mark my words, the cheap switch is the problem and I do not want to hear it.

Yes the PowerConnect 2816 has no Hardware timestamping no it is not suitable for Corosync knet transport yes you are running multicast on a Store and forward switch with no IGMPsnooping the Bufferbloat alone will kill your Totem token timeout

• I have seen this exact FailureMode in three Installations
• The 2816 has a known FirmwareBug where multicastbufferoverrun causes random Packet delay variation above your consensus window
• You need a Switch with Precision time protocol support or at minimum CutThroughForwarding for your Cluster heartbeat

The IPMI latency is a separate Disaster recovery risk but it is not your Fencing trigger your TotemProtocol is timing out because the 2816 is buffering your multicast during bursts from your cursed Windows server instance yes the Broadcomchip in that generation is notorious for this

Buy a used JuniperEX2300 or anything with a PTPHardwareClock the usedMarket has them for

Harold its always the switch man always the switch I had this exact thing in 2019 with a netgear that cost me twelve bucks on craigslist corosync needs timing like a drummer needs a beat and cheap switches are drunk drummers three days of debug and it was the buffer every time I dont even look at logs anymore I just swap the switch first thing now save yourself the pain just buy the thing you dont want to buy and move on your cursed windows vm is probably flooding multicast every time it does whatever windows does at 3am and that forty dollar dell is choking on its own queue depth just do it man

Before you replace infrastructure, review your OVHcloud ToS — I think they have something about acceptable use of dedicated switch ports. Some colocation agreements define customer-provided network equipment as "unmanaged attachment" voiding SLA credits.

From memory, standard Hostinger BMC terms require 99.9% availability of the out-of-band interface. Your 90-second response time may constitute a service deficiency permitting fee adjustment, not that I am advising you to pursue this.

Definitionally, "random" fencing implies non-deterministic causation. Your logs, if posted, should include corosync timestamp granularity to distinguish network jitter from quorum miscalculation. Without this, any remediation is speculative.

I note you have not stated whether your cluster uses unicast or multicast corosync. The distinction matters for switch feature requirements under their AUP, as far as I recall

What Corosync version is this on 7.4-3 though

It's always the cheap switch.