pcie: Allow atomic completion on PCIE root port
cover pcie: Allow atomic completion on PCIE root port
From: Robin Voetter <[email protected]>
The ROCm driver for Linux uses PCIe atomics to schedule work and generally communicate between the host and the device. This does not currently work in QEMU with regular vfio-pci passthrough, because the pcie-root-port does not advertise the PCIe atomic completer capabilities. When initializing the GPU from the Linux driver, it queries whether the PCIe connection from the CPU to GPU supports the required capabilities[1] in the pci_enable_atomic_ops_to_root function[2]. Currently the only part where this fails is checking the atomic completer capabilities (32 and 64 bits) on the root port[3]. In this case, the driver determines that PCIe atomics are not supported at all, and this causes ROCm programs to misbehave. (While AMD advertises that there is some support for ROCm without PCIe atomics, I have never actually gotten that working...)
This patch allows ROCm to properly function by introducing an additional experimental property to the pcie-root-port, x-atomic-completion. Setting this option makes the port report support for the PCI_EXP_DEVCAP2_ATOMIC_COMP32 and COMP64 capabilities. This then makes the check from [3] pass, and everything seems to work appropriately after that.
To verify that the capabilities are reported correctly, one can use lspci to check the capabilities of the root port: lspci -vvv -s <root port id> should show 32bit+ and 64bit+ capabilities in DevCap2 when x-atomic-completion is enabled. For example:
-device pcie-root-port,x-atomic-completion=true,id.pcie.1
The output of lspci should include the following for the pcie root port:
AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
To verify that ROCm works, the following HIP program should be sufficient. The work is scheduled to the GPU by signaling a semaphore using atomic operations from the CPU side, which is completed on the GPU, and the GPU-side printf works by signaling a semaphore from the GPU that is completed on the CPU. It can be compiled using hipcc with 'hipcc -otest test.hip':
#include <hip/hip_runtime.h>
__global__ void test() {
printf("hello, world\n");
}
int main() {
test<<<dim3(1), dim3(1)>>>();
hipDeviceSynchronize();
}Previously, or when x-atomic-completion is set to false, this program would simply hang. Additionally, a message along the lines of the following is printed to dmesg during boot if the GPU driver determines that atomics are not supported:
amdgpu 0000:01:00.0: amdgpu: PCIE atomic ops is not supported
When atomics are properly supported, the above program works as intended, and the previous dmesg message is of course not printed. For this I am using a simple machine setup using the following device options, with the GPU that im testing with of course on 03:00.0.
-device pcie-root-port,x-atomic-completion=true,id=pcie.1
-device vfio-pci,host=03:00.0,bus=pcie.1This patch does not include any automatic detection whether the root port of the host supports the atomic completer capabilities, nor if any of the physical PCIe bridges between the CPU and GPU support atomic routing. The intention here is that the user should make sure that the host does support atomic completion on the root complex. See also some prior discussion[4]. I have run the full test suite of some ROCm
libraries: rocPRIM, rocRAND, hipRAND, hipCUB and rocThrust. All of the tests pass now, with some minor unrelated changes.
Kind regards,
Robin Voetter, Stream HPC
[1] https://github.com/torvalds/linux/blob/v6.2/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c#L3716
[2] https://github.com/torvalds/linux/blob/v6.2/drivers/pci/pci.c#L3781
[3] https://github.com/torvalds/linux/blob/v6.2/drivers/pci/pci.c#L3829
[4] https://lists.nongnu.org/archive/html/qemu-devel/2023-04/msg01815.html
---
Robin Voetter (1):
pcie: Allow generic PCIE root port to enable atomic completion
hw/pci-bridge/gen_pcie_root_port.c | 2 ++ hw/pci/pcie.c | 6 ++++++ include/hw/pci/pcie_port.h | 3 +++ 3 files changed, 11 insertions(+)
--
2.39.2
PATCH 1/1 pcie: Allow generic PCIE root port to enable atomic completion
From: Robin Voetter <[email protected]>
Adds an experimental option to the pcie-root-port to make it report support for the PCIe atomic completion capabilities. This can now be enabled by setting the x-atomic-completion option on the pcie-root-port device. Defaults to disabled for now. For example:
-device pcie-root-port,x-atomic-completion=true,id=pcie.1
--- hw/pci-bridge/gen_pcie_root_port.c | 2 ++ hw/pci/pcie.c | 6 ++++++ include/hw/pci/pcie_port.h | 3 +++ 3 files changed, 11 insertions(+)
hw/pci-bridge/gen_pcie_root_port.c +2 -0
@@ -145,6 +145,8 @@ static Property gen_rp_props[] = { speed, PCIE_LINK_SPEED_16),DEFINE_PROP_PCIE_LINK_WIDTH ("x-width" , PCIESlot, width,PCIE_LINK_WIDTH_32 ), +DEFINE_PROP_BOOL ("x-atomic-completion" , PCIESlot, + atomic_completion, false), DEFINE_PROP_END_OF_LIST() };
hw/pci/pcie.c +6 -0
@@ -663,6 +663,12 @@ void pcie_cap_slot_init(PCIDevice *dev, PCIESlot *s)qbus_set_hotplug_handler (BUS (pci_bridge_get_sec_bus (PCI_BRIDGE (dev))),OBJECT (dev)); + +if (s->atomic_completion) { +/* PCIe requires setting both comp32 and comp64 if either is supported */ +pci_set_long (dev->config + dev->exp.exp_cap +PCI_EXP_DEVCAP2 , +PCI_EXP_DEVCAP2_ATOMIC_COMP32 |PCI_EXP_DEVCAP2_ATOMIC_COMP64 ); + } }void pcie_cap_slot_reset (PCIDevice *dev)
include/hw/pci/pcie_port.h +3 -0
@@ -68,6 +68,9 @@ struct PCIESlot {/* broken ACPI hotplug compat knob to preserve 6.1 ABI intact */ bool hide_native_hotplug_cap; +/* Enables PCIE Atomic completion on pcie_root_port */ +bool atomic_completion; +QLIST_ENTRY (PCIESlot) next; };
Signature · 1 line
2.39.2
Quoted 16 lines · On Thu, Apr 20, 2023 at 05:38:39PM +0200, [email protected] wrote:
From: Robin Voetter <[email protected]>
The ROCm driver for Linux uses PCIe atomics to schedule work and generally communicate between the host and the device. This does not currently work in QEMU with regular vfio-pci passthrough, because the pcie-root-port does not advertise the PCIe atomic completer capabilities. When initializing the GPU from the Linux driver, it queries whether the PCIe connection from the CPU to GPU supports the required capabilities[1] in the
pci_enable_atomic_ops_to_rootfunction[2]. Currently the only part where this fails is checking the atomic completer capabilities (32 and 64 bits) on the root port[3]. In this case, the driver determines that PCIe atomics are not supported at all, and this causes ROCm programs to misbehave. (While AMD advertises that there is some support for ROCm without PCIe atomics, I have never actually gotten that working...)
Quoted 3 lines
This patch allows ROCm to properly function by introducing an additional experimental property to the pcie-root-port, x-atomic-completion.
so what exactly makes it experimental? from this description it looks like it actually has to be enabled for things to work? Also pls CC alex on whether this is a correct way to do it.
Quoted 76 lines
Setting this option makes the port report
support for thePCI_EXP_DEVCAP2_ATOMIC_COMP32and COMP64 capabilities. This then makes the check from [3] pass, and everything seems to work appropriately after that.To verify that the capabilities are reported correctly, one can use lspci to check the capabilities of the root port: lspci -vvv -s <root port id> should show 32bit+ and 64bit+ capabilities in DevCap2 when x-atomic-completion is enabled. For example:
-device pcie-root-port,x-atomic-completion=true,id.pcie.1
The output of lspci should include the following for the pcie root port:
AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-To verify that ROCm works, the following HIP program should be sufficient. The work is scheduled to the GPU by signaling a semaphore using atomic operations from the CPU side, which is completed on the GPU, and the GPU-side printf works by signaling a semaphore from the GPU that is completed on the CPU. It can be compiled using hipcc with 'hipcc -otest test.hip':
#include <hip/hip_runtime.h> __global__ void test() { printf("hello, world\n"); } int main() { test<<<dim3(1), dim3(1)>>>(); hipDeviceSynchronize(); }Previously, or when x-atomic-completion is set to false, this program would simply hang. Additionally, a message along the lines of the following is printed to dmesg during boot if the GPU driver determines that atomics are not supported:
amdgpu 0000:01:00.0: amdgpu: PCIE atomic ops is not supported
When atomics are properly supported, the above program works as intended, and the previous dmesg message is of course not printed. For this I am using a simple machine setup using the following device options, with the GPU that im testing with of course on 03:00.0.
-device pcie-root-port,x-atomic-completion=true,id=pcie.1 -device vfio-pci,host=03:00.0,bus=pcie.1This patch does not include any automatic detection whether the root port of the host supports the atomic completer capabilities, nor if any of the physical PCIe bridges between the CPU and GPU support atomic routing. The intention here is that the user should make sure that the host does support atomic completion on the root complex. See also some prior discussion[4]. I have run the full test suite of some ROCm
libraries: rocPRIM, rocRAND, hipRAND, hipCUB and rocThrust. All of the tests pass now, with some minor unrelated changes.Kind regards,
Robin Voetter, Stream HPC
[1] https://github.com/torvalds/linux/blob/v6.2/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c#L3716
[2] https://github.com/torvalds/linux/blob/v6.2/drivers/pci/pci.c#L3781
[3] https://github.com/torvalds/linux/blob/v6.2/drivers/pci/pci.c#L3829
[4] https://lists.nongnu.org/archive/html/qemu-devel/2023-04/msg01815.html
---Robin Voetter (1):
pcie: Allow generic PCIE root port to enable atomic completionhw/pci-bridge/gen_pcie_root_port.c | 2 ++ hw/pci/pcie.c | 6 ++++++ include/hw/pci/pcie_port.h | 3 +++ 3 files changed, 11 insertions(+)--
2.39.2
Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · On Thu, Apr 20, 2023 at 05:38:39PM +0200, [email protected] wrote:
From: Robin Voetter <[email protected]>
The ROCm driver for Linux uses PCIe atomics to schedule work and generally communicate between the host and the device. This does not currently work in QEMU with regular vfio-pci passthrough, because the pcie-root-port does not advertise the PCIe atomic completer capabilities. When initializing the GPU from the Linux driver, it queries whether the PCIe connection from the CPU to GPU supports the required capabilities[1] in the
pci_enable_atomic_ops_to_rootfunction[2]. Currently the only part where this fails is checking the atomic completer capabilities (32 and 64 bits) on the root port[3]. In this case, the driver determines that PCIe atomics are not supported at all, and this causes ROCm programs to misbehave. (While AMD advertises that there is some support for ROCm without PCIe atomics, I have never actually gotten that working...)
Quoted 6 lines
Quoted 3 lines
This patch allows ROCm to properly function by introducing an additional experimental property to the pcie-root-port, x-atomic-completion.
so what exactly makes it experimental? from this description it looks like it actually has to be enabled for things to work?
I was not sure which would be appropriate, but I'm fine with making it a non-experimental option.
Quoted 24 lines · Robin Voetter
Quoted 21 lines · Michael S. Tsirkin
Quoted 19 lines · On Thu, Apr 20, 2023 at 05:38:39PM +0200, [email protected] wrote:
From: Robin Voetter <[email protected]>
The ROCm driver for Linux uses PCIe atomics to schedule work and generally communicate between the host and the device. This does not currently work in QEMU with regular vfio-pci passthrough, because the pcie-root-port does not advertise the PCIe atomic completer capabilities. When initializing the GPU from the Linux driver, it queries whether the PCIe connection from the CPU to GPU supports the required capabilities[1] in the
pci_enable_atomic_ops_to_rootfunction[2]. Currently the only part where this fails is checking the atomic completer capabilities (32 and 64 bits) on the root port[3]. In this case, the driver determines that PCIe atomics are not supported at all, and this causes ROCm programs to misbehave. (While AMD advertises that there is some support for ROCm without PCIe atomics, I have never actually gotten that working...)This patch allows ROCm to properly function by introducing an additional experimental property to the pcie-root-port, x-atomic-completion.
Quoted 5 lines
Quoted 2 lines
so what exactly makes it experimental? from this description it looks like it actually has to be enabled for things to work?
I was not sure which would be appropriate, but I'm fine with making it a non-experimental option.
So I guess the real thing to do is to query this from vfio right? Unfortunately we don't have access to vfio when we
are creating the root port, but I think the thing to do would be to check at the time when vfio is attached, and if atomic is set but not supported, fail attaching vfio.
Right?
Signature · 1 line
MST
Quoted 31 lines · On Thu, 18 May 2023 16:03:07 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 24 lines · Robin Voetter
Quoted 21 lines · Michael S. Tsirkin
Quoted 19 lines · On Thu, Apr 20, 2023 at 05:38:39PM +0200, [email protected] wrote:
From: Robin Voetter <[email protected]>
The ROCm driver for Linux uses PCIe atomics to schedule work and generally communicate between the host and the device. This does not currently work in QEMU with regular vfio-pci passthrough, because the pcie-root-port does not advertise the PCIe atomic completer capabilities. When initializing the GPU from the Linux driver, it queries whether the PCIe connection from the CPU to GPU supports the required capabilities[1] in the
pci_enable_atomic_ops_to_rootfunction[2]. Currently the only part where this fails is checking the atomic completer capabilities (32 and 64 bits) on the root port[3]. In this case, the driver determines that PCIe atomics are not supported at all, and this causes ROCm programs to misbehave. (While AMD advertises that there is some support for ROCm without PCIe atomics, I have never actually gotten that working...)This patch allows ROCm to properly function by introducing an additional experimental property to the pcie-root-port, x-atomic-completion.
Quoted 5 lines
Quoted 2 lines
so what exactly makes it experimental? from this description it looks like it actually has to be enabled for things to work?
I was not sure which would be appropriate, but I'm fine with making it a non-experimental option.
Quoted 7 lines
So I guess the real thing to do is to query this from vfio right? Unfortunately we don't have access to vfio when we
are creating the root port, but I think the thing to do would be to check at the time when vfio is attached, and if atomic is set but not supported, fail attaching vfio.Right?
We don't currently provide a way to query this in vfio, but I imagine we could call pci_enable_atomic_ops_to_root() in the host kernel ourselves with various sizes and expose which are supported via a capability on the vfio-device. I'm not sure what we do for VFs though since that function is invalid for them (maybe worry about them later). I also see that one of the in-kernel drivers (mlx5) tries to enable 128-bit support, so I wonder if we want separate options for 32/64-bit and 128-bit.
QEMU device options are clearly the most straightforward path to enable this, but would it actually make sense, perhaps in addition, to implement the above in the kernel and then have the QEMU vfio-pci driver enable the available completer bits in the root port during realize? We could probably get away with it on hotplug, but if necessary it could be something we only do for cold-plug devices (we also have a no-hotplug vfio-pci variant if we're concerned what happens after the device is removed in the VM - again, be could probably get away with clearing the bits on unplug).
I'm not entirely sure where we stand in QEMU on whether options that can cause poor behavior should always be experimental or we allow users to shoot themselves in the foot as they please. Obviously it makes it more difficult for libvirt to support such configurations, but maybe they'd rely on the above automatic enabling rather than try to guess themselves. Thanks,
Alex