Member
I am check my upstream ạ
The root port is group 23 alone, gpu is group 22 alone. Before on 7002 the root port and gpu is same group 22 ạ
I am try pass both group but QEMU say cannot mix different bus in same command? I am not understand this error ạ
The slot move is not possible ạ my board only have one x16 slot ạ
You cannot pass through a root port that has other children. The bridge carries traffic for everything behind it. If your NIC is also behind that root port, you lose the host.
This is why I said move the card or accept the risk. Single x16 slot means you are choosing the risk or choosing no passthrough.
EOF
I have four servers, all cheap, none of this problem. Bangkok heat kills GPUs anyway so I run headless. But I watch this thread because someday I will want a GPU for ML and then I will have this pain too.
Following.
Member
The firmware on these new boards often has a "ACS disable" option hidden under AMD CBS or NBIO settings. Not the same as the Linux ACS patch. This changes how the firmware reports groups, not how the kernel enforces them.
I have seen this on Supermicro H13 boards. Check if your motherboard vendor has a newer BMC firmware too.
Member OP
My board is Gigabyte MC12-LE0. The BIOS is newest from March. I am check the AMD CBS and find "ACS Enable" is already on. I am try disable and reboot, same problem. Enable again, same.
The BMC firmware I am not check, thank you for this idea.
@Nadia My NIC is on different bus, not same root port. The group 23 is only the root port for GPU, nothing else. But QEMU still say not viable. This is the strange part.
Have you checked if the root port itself is being grabbed by the host kernel driver? Vfio-pci needs to bind the whole path if you want to pass the path. If the root port is still bound to pcieport, QEMU will refuse even if the group looks clean.
lspci -k -s 0000:40:00.0This is the part everyone forgets. The group being "alone" doesn't mean the kernel isn't using the bridge.
I had the same group split on 9004 and my fix was binding the root port to vfio-pci too. Then pass both. My QEMU line:
-device vfio-pci,host=0000:40:00.0 -device vfio-pci,host=0000:41:00.0Not in same -device, separate devices. The "cannot mix different bus" error is because you tried one -device with two hosts. Two -device, one each.
Worked for me on kernel 6.1.
Member
Ạ ạ ạ this is work ạ
I am bind root port to vfio-pci and pass both separate device ạ the VM start now ạ
Thank you @DAVID27 very much ạ