Skip to content
Register

CPU generation jump broke my PCIe passthrough setup

Dedicated Servers by Bruno22 17 replies 1.6K views
Bruno22

Member OP

#1

I am upgrade from EPYC 7002 to 9004 and now my GPU passthrough is BROKEN my VM not start anymore with the same QEMU command!

Before I am use this and is very good:

-device vfio-pci,host=0000:41:00.0,multifunction=on

Now the IOMMU group is different? The device is alone in group but QEMU say "group is not viable" I am make a test with rombar=off and not work.

My motherboard is new generation too. The ACS override patch I am try but the kernel not compile with my current config

Someone have the same problem? This upgrade is very expensive and now my VM is dead

ping so high I wave back
jane_ffm

Member

#2

Hi @Bruno22,

Thanks for reaching out—sorry to hear about the passthrough trouble post-upgrade. This is a known friction point with newer EPYC generations due to IOMMU grouping changes in the platform firmware.

A few quick checks I'd recommend:

• Verify your BIOS has ACS and ARI enabled—some 9004 boards ship with these disabled
• Check if the IOMMU group now includes a root port or bridge device
• Compare your kernel's `CONFIG_PCI_PASSTHROUGH` and `CONFIG_VFIO` versions

If you'd like, I can review your full QEMU command line and dmesg output via DM. We see this occasionally with customers migrating to newer Leaseweb bare metal.

— Jane @ Contabo

Single mode till I die 💀
van3

Member

#3

I am hos the same cpu generation hos very pained ạ

My iommu group is split different now no root port same group anymore ạ

Before my gpu is group 22 with audio now is group 22 alone and group 23 is root port ạ

This is break my old qemu command very much ạ

I am try the patch from 2023 but it not work for this kernel ạ the compile is hos ạ

Can you share your new qemu line ạ

liam_funky

Member

Vancouver

7 #4

1. The problem is clear
2. Your IOMMU grouping changed between CPU generations
3. The "fix" everyone is suggesting
3. A) ACS override patch
3. B) I) This breaks security boundaries
3. B) ii) It was never a real fix
4. What you should actually do
4. A) Check if your GPU is now grouped with a chipset device
4. B) If so, passthrough the entire group or use a different slot
5. The real solution
5. I) Newer firmware often splits groups more granularly
5. Ii) Your old "working" setup might have been exploiting a bug
6. I learned this the hard way on a RackNerd node last year — https://www.racknerd.com

Nadia

Member

/dev/null enthusiast

#5
liam_funky said:
Exploiting a bug

The security boundary was already /dev/null. Your "fix" redirected the IOMMU into a pipe with no read end. The ACS patch pipes group isolation to EOF and calls it freedom.

| proper grouping: hardware enforces boundaries
| override patch: kernel lies to QEMU
| lie becomes truth until the exploit

That 2023 patch? It was redirected by the maintainers for a reason. The vulnerability it reopens lets any guest DMA to any BAR. Your GPU can now pipe memory it doesn't own.

The real fix: physical topology. Move the card. Accept the group. Or accept the risk and know you're running naked.

EOF

/dev/null: full of good ideas
DH

Member

#6

Which kernel are you compiling, 6.1 or 6.6?

DA

Member

#7

I had the same group split on 9004

Bruno22

Member OP

#8
van3 said:
Can you share your new qemu line ạ

I am not find a good line yet. The rombar=off not work, the x-vga=on not work. I am try now with:

-device vfio-pci,host=0000:41:00.0,bus=pcie.0,addr=1c.0

But QEMU say same error. The group is 22 alone, no other device, but QEMU say "not viable". I am confuse very much.

My kernel is 6.1 from Debian 12 backports, https://www.debian.org. The ACS patch I am try is from Alex Williamson github but not compile for this kernel version.

@dhifiwoot 6.1, you think 6.6 is better for this?

ping so high I wave back
DH

Member

#9

6.6 has better ACS quirks for 9004 but you still need the patch if your board firmware is grouping weird. I had to move to 6.6 on my Hetzner box to get sane groups but that is Intel not AMD.

liam_funky

Member

Vancouver

4 #10
Nadia said:
The real fix: physical topology. Move the card.

This is the correct answer and I am annoyed I buried it under my own numbered list.

Physical topology is the only fix that doesn't involve lying to the kernel. The 9004 platform firmware on many boards changed how root ports enumerate. Your GPU might be behind a bridge that now exposes itself as part of the path.

Check:

lspci -vvv -s 41:00.0 | grep LnkCap

Then check what is upstream. If the upstream is now its own group, you pass through the whole path or you move slots. Those are the options.

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft