vfio/nvgpu: Add vfio pci variant module for grace hopper
[email protected]
maintainerPATCH v12 1/1 vfio/nvgpu: Add vfio pci variant module for grace hopper
From: Ankit Agrawal <[email protected]>
NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device for the on-chip GPU that is the logical OS representation of the internal proprietary cache coherent interconnect.
This representation has a number of limitations compared to a real PCI device, in particular, it does not model the coherent GPU memory aperture as a PCI config space BAR, and PCI doesn't know anything about cacheable memory types.
Provide a VFIO PCI variant driver that adapts the unique PCI representation into a more standard PCI representation facing userspace. The GPU memory aperture is obtained from ACPI using device_property_read_u64(), according to the FW specification, and exported to userspace as a separate VFIO_REGION. Since the device implements only one 64-bit BAR (BAR0), the GPU memory aperture is mapped to the next available PCI BAR (BAR2). Qemu will then naturally generate a PCI device in the VM with two 64-bit BARs (where the cacheable aperture reported in BAR2).
Since this memory region is actually cache coherent with the CPU, the VFIO variant driver will mmap it into VMA using a cacheable mapping. The mapping is done using remap_pfn_range().
PCI BAR are aligned to the power-of-2, but the actual memory on the device may not. A read or write access to the physical address from the last device PFN up to the next power-of-2 aligned physical address results in reading ~0 and dropped writes. Note that the GPU device driver [1] is capable of knowing the exact device memory size through separate means. The device memory size is primarily kept in the system ACPI tables for use by the VFIO PCI variant module.
This goes along with a qemu series to provides the necessary implementation of the Grace Hopper Superchip firmware specification so that the guest operating system can see the correct ACPI modeling for the coherent GPU device. Verified with the CUDA workload in the VM.
https://lore.kernel.org/all/[email protected]/
This patch is split from a patch series being pursued separately:
https://lore.kernel.org/lkml/[email protected]/
Applied over next-20231005.
[1] https://github.com/NVIDIA/open-gpu-kernel-modules
---
Link for v11:
https://lore.kernel.org/all/[email protected]/
v11 -> v12
- More details in commit message on device memory size
v10 -> v11
- Removed sysfs attribute to expose the CPU coherent memory feature
- Addressed review comments
v9 -> v10
- Add new sysfs attribute to expose the CPU coherent memory feature.
v8 -> v9
- Minor code adjustment suggested in v8.
v7 -> v8
- Various field names updated.
- Added a new function to handle VFIO_DEVICE_GET_REGION_INFO ioctl.
- Locking protection for memremap to bar region and other changes recommended in v7.
- Added code to fail if the devmem size advertized is 0 in system DSDT.
v6 -> v7
- Handled out-of-bound and overflow conditions at various places to validate input offset and length.
- Added code to return EINVAL for offset beyond region size.
v5 -> v6
- Added the code to handle BAR2 read/write using memremap to the device memory.
v4 -> v5
- Changed the module name from nvgpu-vfio-pci to nvgrace-gpu-vfio-pci.
- Fixed memory leak and added suggested boundary checks on device memory mapping.
- Added code to read all Fs and ignored write on region outside of the physical memory.
- Other miscellaneous cleanup suggestions.
v3 -> v4
- Mapping the available device memory using sparse mmap. The region outside the device memory is handled by read/write ops.
- Removed the fault handler added in v3.
v2 -> v3
- Added fault handler to map the region outside the physical GPU memory up to the next power-of-2 to a dummy PFN.
- Changed to select instead of "depends on" VFIO_PCI_CORE for all the vfio-pci variant driver.
- Code cleanup based on feedback comments.
- Code implemented and tested against v6.4-rc4.
v1 -> v2
- Updated the wording of reference to BAR offset and replaced with index.
- The GPU memory is exposed at the fixed BAR2_REGION_INDEX.
- Code cleanup based on feedback comments.
--- MAINTAINERS | 6 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/nvgrace-gpu/Kconfig | 10 + drivers/vfio/pci/nvgrace-gpu/Makefile | 3 + drivers/vfio/pci/nvgrace-gpu/main.c | 481 ++++++++++++++++++++++++++ 6 files changed, 504 insertions(+) create mode 100644 drivers/vfio/pci/nvgrace-gpu/Kconfig create mode 100644 drivers/vfio/pci/nvgrace-gpu/Makefile create mode 100644 drivers/vfio/pci/nvgrace-gpu/main.c
MAINTAINERS +6 -0
@@ -22770,6 +22770,12 @@ L: [email protected] S: Maintained F: drivers/vfio/platform/ +VFIO NVIDIA GRACE GPU DRIVER +M: Ankit Agrawal <[email protected]> +L: [email protected] +S: Maintained +F: drivers/vfio/pci/nvgrace-gpu/ + VGA_SWITCHEROO R: Lukas Wunner <[email protected]> S: Maintained
drivers/vfio/pci/Kconfig +2 -0
@@ -65,4 +65,6 @@ source "drivers/vfio/pci/hisilicon/Kconfig" source "drivers/vfio/pci/pds/Kconfig" +source "drivers/vfio/pci/nvgrace-gpu/Kconfig" + endmenu
drivers/vfio/pci/Makefile +2 -0
@@ -13,3 +13,5 @@ obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/ obj-$(CONFIG_PDS_VFIO_PCI) += pds/ + +obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/
drivers/vfio/pci/nvgrace-gpu/Kconfig new file +10 -0
@@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0-only +config NVGRACE_GPU_VFIO_PCI + tristate "VFIO support for the GPU in the NVIDIA Grace Hopper Superchip" + depends on ARM64 || (COMPILE_TEST && 64BIT) + select VFIO_PCI_CORE + help + VFIO support for the GPU in the NVIDIA Grace Hopper Superchip is + required to assign the GPU device using KVM/qemu/etc. + + If you don't know what to do here, say N.
drivers/vfio/pci/nvgrace-gpu/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu-vfio-pci.o +nvgrace-gpu-vfio-pci-y := main.o
drivers/vfio/pci/nvgrace-gpu/main.c new file +481 -0
481 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 5 lines · On Sun, 15 Oct 2023 22:00:47 +0530 <[email protected]> wrote:
drivers/vfio/pci/nvgrace-gpu/main.c new file +5
-0+static ssize_t nvgrace_gpu_vfio_pci_read (struct vfio_device *core_vdev, +char __user *buf,size_t count,loff_t *ppos) +{ +unsigned int index =VFIO_PCI_OFFSET_TO_INDEX (*ppos); + struct nvgrace_gpu_vfio_pci_core_device *nvdev = container_of(
drivers/vfio/pci/nvgrace-gpu/main.c line 282
Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +10
-0+ core_vdev, struct nvgrace_gpu_vfio_pci_core_device, core_device.vdev); +int ret; + +if (index ==VFIO_PCI_BAR2_REGION_INDEX ) { + ret =nvgrace_gpu_memmap (nvdev); +if (ret) +return ret; + +return nvgrace_gpu_read_mem (buf, count, ppos, nvdev); + }
After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated as an IO Port BAR, it occurs to me that there's no config space emulation of BAR2 (or BAR3) here. Doesn't this mean that QEMU registers the BAR as 32-bit, non-prefetchable? ie. VFIOBAR.type & .mem64 are wrong?
I'd certainly expect this to be emulated as a 64-bit, prefetchable BAR and the commit log indicates the intention is that this is exposed as a 64-bit BAR.
We also need to decide how strictly variant drivers need to emulate vfio_pci_config_rw with respect to BAR sizing, where the core code provides emulation of sizing and Yishai's virtio driver only emulates the IO port indicator bit. QEMU doesn't really need this, but the vfio-pci implementation sets the precedent that this behavior is provided and could be used by other userspace drivers. It's essentially just providing a masked buffer to service reads and writes to the BAR2 and BAR3 config address here. Thanks,
Alex
Quoted 3 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +3
-0+ +return vfio_pci_core_read (core_vdev, buf, count, ppos); +}
Ankit Agrawal
maintainerQuoted 5 lines
After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated as an IO Port BAR, it occurs to me that there's no config space emulation of BAR2 (or BAR3) here. Doesn't this mean that QEMU registers the BAR as 32-bit, non-prefetchable? ie. VFIOBAR.type & .mem64 are wrong?
Maybe I didn't understand the question, but the PCI config space read/write would still be handled by vfio_pci_core_read/write() which returns the appropriate flags. I have checked that the device BARs are 64b and prefetchable in the VM.
Quoted 4 lines
We also need to decide how strictly variant drivers need to emulate
vfio_pci_config_rwwith respect to BAR sizing, where the core code provides emulation of sizing and Yishai's virtio driver only emulates the IO port indicator bit.
Sorry, it isn't clear to me how would resizable BAR is applicable in this variant driver as the BAR represents the device memory. Should we be exposing such feature as unsupported for this variant driver?
Quoted 10 lines · On Mon, 23 Oct 2023 12:48:22 +0000 Ankit Agrawal <[email protected]> wrote:
Quoted 5 lines
After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated as an IO Port BAR, it occurs to me that there's no config space emulation of BAR2 (or BAR3) here. Doesn't this mean that QEMU registers the BAR as 32-bit, non-prefetchable? ie. VFIOBAR.type & .mem64 are wrong?
Maybe I didn't understand the question, but the PCI config space read/write would still be handled by vfio_pci_core_read/write() which returns the appropriate flags. I have checked that the device BARs are 64b and prefetchable in the VM.
vfio_pci_core_read/write() accesses the physical device, which doesn't implement BAR2. Why would an unimplemented BAR2 on the physical device report 64-bit, prefetchable?
QEMU records VFIOBAR.type and .mem64 from reading the BAR register in vfio_bar_prepare() and passes this type to pci_register_bar() in vfio_bar_register(). Without an implementation of a config space read op in the variant driver and with no physical implementation of BAR2 on the device, I don't see how we get correct values in these fields.
Quoted 8 lines
Quoted 4 lines
We also need to decide how strictly variant drivers need to emulate
vfio_pci_config_rwwith respect to BAR sizing, where the core code provides emulation of sizing and Yishai's virtio driver only emulates the IO port indicator bit.Sorry, it isn't clear to me how would resizable BAR is applicable in this variant driver as the BAR represents the device memory. Should we be exposing such feature as unsupported for this variant driver?
Bar SIZING, not resizing. This is the standard in-band mechanism for determining the BAR size as described in PCIe 6.0.1, 7.5.1.2.1. QEMU makes use of the region size but does rely on the BAR flags when registering the BAR into QEMU as described above. Additionally, vfio-pci-core supports this in-band sizing mechanism for physical BARs. A variant driver which does not implement config space BAR sizing for a virtual BAR is arguably not presenting a PCI compatible config space where a non-QEMU userspace may depend on standard PCI behavior here. Thanks,
Alex
Ankit Agrawal
maintainerQuoted 15 lines
Quoted 10 lines
Quoted 5 lines
After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated as an IO Port BAR, it occurs to me that there's no config space emulation of BAR2 (or BAR3) here. Doesn't this mean that QEMU registers the BAR as 32-bit, non-prefetchable? ie. VFIOBAR.type & .mem64 are wrong?
Maybe I didn't understand the question, but the PCI config space read/write would still be handled by vfio_pci_core_read/write() which returns the appropriate flags. I have checked that the device BARs are 64b and prefetchable in the VM.
vfio_pci_core_read/write() accesses the physical device, which doesn't implement BAR2. Why would an unimplemented BAR2 on the physical device report 64-bit, prefetchable?
Quoted 5 lines
QEMU records VFIOBAR.type and .mem64 from reading the BAR register in
vfio_bar_prepare()and passes this type topci_register_bar()in vfio_bar_register(). Without an implementation of a config space read op in the variant driver and with no physical implementation of BAR2 on the device, I don't see how we get correct values in these fields.
I think I see the cause of confusion. There are real PCIe compliant BARs present on the device, just that it isn't being used once the C2C interconnect is active. The BARs are 64b prefetchable. Here it the lspci snippet of the device on the host.
# lspci -v -s 9:1:0.0
0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1)
Subsystem: NVIDIA Corporation Device 16eb
Physical Slot: 0-5
Flags: bus master, fast devsel, latency 0, IRQ 263, NUMA node 0, IOMMU group 19
Memory at 661002000000 (64-bit, prefetchable) [size=16M]
Memory at 662000000000 (64-bit, prefetchable) [size=128G]
Memory at 661000000000 (64-bit, prefetchable) [size=32M]I suppose this answers the BAR sizing question as well?
Quoted 34 lines · On Tue, 24 Oct 2023 14:03:25 +0000 Ankit Agrawal <[email protected]> wrote:
Quoted 15 lines
Quoted 10 lines
Quoted 5 lines
After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated as an IO Port BAR, it occurs to me that there's no config space emulation of BAR2 (or BAR3) here. Doesn't this mean that QEMU registers the BAR as 32-bit, non-prefetchable? ie. VFIOBAR.type & .mem64 are wrong?
Maybe I didn't understand the question, but the PCI config space read/write would still be handled by vfio_pci_core_read/write() which returns the appropriate flags. I have checked that the device BARs are 64b and prefetchable in the VM.
vfio_pci_core_read/write() accesses the physical device, which doesn't implement BAR2. Why would an unimplemented BAR2 on the physical device report 64-bit, prefetchable?
Quoted 5 lines
QEMU records VFIOBAR.type and .mem64 from reading the BAR register in
vfio_bar_prepare()and passes this type topci_register_bar()in vfio_bar_register(). Without an implementation of a config space read op in the variant driver and with no physical implementation of BAR2 on the device, I don't see how we get correct values in these fields.I think I see the cause of confusion. There are real PCIe compliant BARs present on the device, just that it isn't being used once the C2C interconnect is active. The BARs are 64b prefetchable. Here it the lspci snippet of the device on the host.
# lspci -v -s 9:1:0.0 0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1) Subsystem: NVIDIA Corporation Device 16eb Physical Slot: 0-5 Flags: bus master, fast devsel, latency 0, IRQ 263, NUMA node 0, IOMMU group 19 Memory at 661002000000 (64-bit, prefetchable) [size=16M] Memory at 662000000000 (64-bit, prefetchable) [size=128G] Memory at 661000000000 (64-bit, prefetchable) [size=32M]
Quoted 1 line
I suppose this answers the BAR sizing question as well?
Does this BAR2 size match the size we're reporting for the region? Now I'm confused why we need to intercept the BAR2 region info if there's physically a real BAR behind it. Thanks,
Alex
Tian, Kevin
reviewerQuoted 45 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, October 24, 2023 10:29 PMQuoted 3 lines · On Tue, 24 Oct 2023 14:03:25 +0000 Ankit Agrawal <[email protected]> wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated as an IO Port BAR, it occurs to me that there's no config space emulation of BAR2 (or BAR3) here. Doesn't this mean that QEMU
registers
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
the BAR as 32-bit, non-prefetchable? ie. VFIOBAR.type & .mem64 are wrong?
Maybe I didn't understand the question, but the PCI config space
read/write
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
would still be handled by vfio_pci_core_read/write() which returns the appropriate flags. I have checked that the device BARs are 64b and prefetchable in the VM.
vfio_pci_core_read/write() accesses the physical device, which doesn't implement BAR2. Why would an unimplemented BAR2 on the physical
device
Quoted 3 lines
Quoted 3 lines
report 64-bit, prefetchable?
QEMU records VFIOBAR.type and .mem64 from reading the BAR register
in
Quoted 9 lines
Quoted 4 lines
vfio_bar_prepare()and passes this type topci_register_bar()in vfio_bar_register(). Without an implementation of a config space read op in the variant driver and with no physical implementation of BAR2 on the device, I don't see how we get correct values in these fields.I think I see the cause of confusion. There are real PCIe compliant BARs present on the device, just that it isn't being used once the C2C interconnect is active. The BARs are 64b prefetchable. Here it the lspci snippet of the device on the host.
Quoted 5 lines
# lspci -v -s 9:1:0.0 0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1) Subsystem: NVIDIA Corporation Device 16eb Physical Slot: 0-5 Flags: bus master, fast devsel, latency 0, IRQ 263, NUMA node 0,IOMMU group 19
Quoted 4 lines
Memory at 661002000000 (64-bit, prefetchable) [size=16M] Memory at 662000000000 (64-bit, prefetchable) [size=128G] Memory at 661000000000 (64-bit, prefetchable) [size=32M]
Quoted 6 lines
Quoted 1 line
I suppose this answers the BAR sizing question as well?
Does this BAR2 size match the size we're reporting for the region? Now I'm confused why we need to intercept the BAR2 region info if there's physically a real BAR behind it. Thanks,
same confusion.
probably vfio-pci-core can include a helper for cfg space emulation on emulated BARs to be used by all variant drivers in that category?
btw intel vgpu also includes an emulation of BAR sizing. same for future SIOV devices. so there sounds like a general requirement but of course sharing it between vfio-pci and mdev/siov would be more difficult.
Ankit Agrawal
maintainerWhile the physical BAR is present on the device, it is not being used on the host system. The access to the device memory region on the host occur through the C2C interconnect link (not the PCIe) and is present for access as a separate memory region in the physical address space on the host. The variant driver queries this range from the host ACPI DSD tables.
Now, this device memory region on the host is exposed as a device BAR in the VM. So the device BAR in the VM is actually mapped to the device memory region in the physical address space (and not to the physical BAR) on the host. The config space accesses to the device however, are still going to the physical BAR on the host.
Quoted 3 lines
Does this BAR2 size match the size we're reporting for the region? Now I'm confused why we need to intercept the BAR2 region info if there's physically a real BAR behind it. Thanks,
Yes, it does match the size being reported through region info. But the region info ioctl is still intercepted to provide additional cap to establish the sparse mapping. Why we do sparse mapping? The actual device memory size is not power-of-2 aligned (a requirement for a BAR). So we roundup to the next power-of-2 value and report the size as such. Then we utilize sparse mapping to show only the actual size of the device memory as mappable.
Quoted 5 lines · On Wed, 25 Oct 2023 12:43:24 +0000 Ankit Agrawal <[email protected]> wrote:
While the physical BAR is present on the device, it is not being used on the host system. The access to the device memory region on the host occur through the C2C interconnect link (not the PCIe) and is present for access as a separate memory region in the physical address space on the host. The variant driver queries this range from the host ACPI DSD tables.
BTW, it's still never been answered why the latest QEMU series dropped the _DSD support.
Quoted 9 lines
Now, this device memory region on the host is exposed as a device BAR in the VM. So the device BAR in the VM is actually mapped to the device memory region in the physical address space (and not to the physical BAR) on the host. The config space accesses to the device however, are still going to the physical BAR on the host.
Quoted 3 lines
Does this BAR2 size match the size we're reporting for the region? Now I'm confused why we need to intercept the BAR2 region info if there's physically a real BAR behind it. Thanks,
Quoted 6 lines
Yes, it does match the size being reported through region info. But the region info ioctl is still intercepted to provide additional cap to establish the sparse mapping. Why we do sparse mapping? The actual device memory size is not power-of-2 aligned (a requirement for a BAR). So we roundup to the next power-of-2 value and report the size as such. Then we utilize sparse mapping to show only the actual size of the device memory as mappable.
Yes, it's clear to me why we need the sparse mapping and why we intercept the accesses, but the fact that there's an underlying physical BAR of the same size in config space has been completely absent in any previous discussions.
In light of that, I don't think we should be independently calculating the BAR2 region size using roundup_pow_of_two(nvdev->memlength). Instead we should be using pci_resource_len() of the physical BAR2 to make it evident that this relationship exists.
The comments throughout should also be updated to reflect this as currently they're written as if there is no physical BAR2 and we're making a completely independent decision relative to BAR2 sizing. A comment should also be added to nvgrace_gpu_vfio_pci_read/write() explaining that the physical BAR2 provides the correct behavior relative to config space accesses.
The probe function should also fail if pci_resource_len() for BAR2 is not sufficient for the coherent memory region. Thanks,
Alex
Ankit Agrawal
maintainerQuoted 2 lines
BTW, it's still never been answered why the latest QEMU series dropped the
_DSDsupport.
The _DSD keys were there in v1 to communicate the PXM start id and the
count associated with the device to the VM kernel. In v2, we proposed an alternative approach to leverage the Generic Initiator (GI) Affinity structure
in SRAT (ACPI Spec 6.5, Section 5.2.16.6) to create NUMA nodes. GI structure allows an association between a GI (GPU in this case) and proximity domains.
So we create 8 GI structures with a unique PXM Id and the device BDF. This removes the need for DSD keys as the VM kernel could parse the GI structures
and identify the PXM IDs associated with the device using the BDF.
(E.g. https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/amd/amdkfd/kfd_crat.c#L1938)
Quoted 4 lines
In light of that, I don't think we should be independently calculating the BAR2 region size using
roundup_pow_of_two(nvdev->memlength). Instead we should be usingpci_resource_len()of the physical BAR2 to make it evident that this relationship exists.
Sure, I will make the change in the next posting.
Quoted 6 lines
The comments throughout should also be updated to reflect this as currently they're written as if there is no physical BAR2 and we're making a completely independent decision relative to BAR2 sizing. A comment should also be added to nvgrace_gpu_vfio_pci_read/write() explaining that the physical BAR2 provides the correct behavior relative to config space accesses.
Yeah, will update the comments.
Quoted 2 lines
The probe function should also fail if
pci_resource_len()for BAR2 is not sufficient for the coherent memory region. Thanks,
Ack.
Ankit Agrawal
maintainerQuoted 4 lines
In light of that, I don't think we should be independently calculating the BAR2 region size using
roundup_pow_of_two(nvdev->memlength). Instead we should be usingpci_resource_len()of the physical BAR2 to make it evident that this relationship exists.
Just wanted to update on the discussions.. Since the underlying physical BAR is not being used at all, we will emulate the PCI config BAR registers for the fake BAR. Thus, the read/write requests to those registers will not go to the underlying BAR on the device. I'll make the change in the next posting.
Quoted 54 lines · Kevin
Quoted 45 lines
From: Alex Williamson <[email protected]>
Sent: Tuesday, October 24, 2023 10:29 PMQuoted 3 lines · On Tue, 24 Oct 2023 14:03:25 +0000 Ankit Agrawal <[email protected]> wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated as an IO Port BAR, it occurs to me that there's no config space emulation of BAR2 (or BAR3) here. Doesn't this mean that QEMU
registers
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
the BAR as 32-bit, non-prefetchable? ie. VFIOBAR.type & .mem64 are wrong?
Maybe I didn't understand the question, but the PCI config space
read/write
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
would still be handled by vfio_pci_core_read/write() which returns the appropriate flags. I have checked that the device BARs are 64b and prefetchable in the VM.
vfio_pci_core_read/write() accesses the physical device, which doesn't implement BAR2. Why would an unimplemented BAR2 on the physical
device
Quoted 3 lines
Quoted 3 lines
report 64-bit, prefetchable?
QEMU records VFIOBAR.type and .mem64 from reading the BAR register
in
Quoted 9 lines
Quoted 4 lines
vfio_bar_prepare()and passes this type topci_register_bar()in vfio_bar_register(). Without an implementation of a config space read op in the variant driver and with no physical implementation of BAR2 on the device, I don't see how we get correct values in these fields.I think I see the cause of confusion. There are real PCIe compliant BARs present on the device, just that it isn't being used once the C2C interconnect is active. The BARs are 64b prefetchable. Here it the lspci snippet of the device on the host.
Quoted 5 lines
# lspci -v -s 9:1:0.0 0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1) Subsystem: NVIDIA Corporation Device 16eb Physical Slot: 0-5 Flags: bus master, fast devsel, latency 0, IRQ 263, NUMA node 0,IOMMU group 19
Quoted 4 lines
Memory at 661002000000 (64-bit, prefetchable) [size=16M] Memory at 662000000000 (64-bit, prefetchable) [size=128G] Memory at 661000000000 (64-bit, prefetchable) [size=32M]Quoted 6 lines
Quoted 1 line
I suppose this answers the BAR sizing question as well?
Does this BAR2 size match the size we're reporting for the region? Now I'm confused why we need to intercept the BAR2 region info if there's physically a real BAR behind it. Thanks,
same confusion.
Quoted 7 lines
probably vfio-pci-core can include a helper for cfg space emulation on emulated BARs to be used by all variant drivers in that category?
btw intel vgpu also includes an emulation of BAR sizing. same for future SIOV devices. so there sounds like a general requirement but of course sharing it between vfio-pci and mdev/siov would be more difficult.
Yes, I see a need for this in the virtio-vfio-pci driver as well. It would simplify config emulation a lot if the variant driver could manipulate the perm_bits.virt and .write bit arrays and simply update vconfig for things like device-id and revision. Thanks,
Alex
kernel test robot
build botkernel test robot noticed the following build warnings:
Hi,
kernel test robot noticed the following build warnings:
[auto build test WARNING on awilliam-vfio/for-linus]
[also build test WARNING on linus/master v6.6 next-20231103] [cannot apply to awilliam-vfio/next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/ankita-nvidia-com/vfio-nvgpu-Add-vfio-pci-variant-module-for-grace-hopper/20231017-131546 base: https://github.com/awilliam/linux-vfio.git for-linus patch link: https://lore.kernel.org/r/20231015163047.20391-1-ankita%40nvidia.com patch subject: [PATCH v12 1/1] vfio/nvgpu: Add vfio pci variant module for grace hopper config: powerpc-allmodconfig (https://download.01.org/0day-ci/archive/20231104/[email protected]/config) compiler: powerpc64-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231104/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> drivers/vfio/pci/nvgrace-gpu/main.c:226:9: warning: no previous prototype for 'nvgrace_gpu_read_mem' [-Wmissing-prototypes]
226 | ssize_t nvgrace_gpu_read_mem(void __user *buf, size_t count, loff_t *ppos,
| ^~~~~~~~~~~~~~~~~~~~
>> drivers/vfio/pci/nvgrace-gpu/main.c:298:9: warning: no previous prototype for 'nvgrace_gpu_write_mem' [-Wmissing-prototypes] 298 | ssize_t nvgrace_gpu_write_mem(size_t count, loff_t *ppos, const void __user *buf,
| ^~~~~~~~~~~~~~~~~~~~~
vim +/nvgrace_gpu_read_mem +226 drivers/vfio/pci/nvgrace-gpu/main.c
214
215 /*
216 * Read count bytes from the device memory at an offset. The actual device
217 * memory size (available) may not be a power-of-2. So the driver fakes
218 * the size to a power-of-2 (reported) when exposing to a user space driver.
219 *
220 * Read request beyond the actual device size is filled with ~0, while
221 * those beyond the actual reported size is skipped.
222 *
223 * A read from a negative or an offset greater than reported size, a negative
224 * count are considered error conditions and returned with an -EINVAL.
225 */
> 226 ssize_t nvgrace_gpu_read_mem(void __user *buf, size_t count, loff_t *ppos,
227 struct nvgrace_gpu_vfio_pci_core_device *nvdev)
228 {
229 u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
230 size_t mem_count, i, bar_size = roundup_pow_of_two(nvdev->memlength);
231 u8 val = 0xFF;
232
233 if (offset >= bar_size)
234 return -EINVAL;
235
236 /* Clip short the read request beyond reported BAR size */
237 count = min(count, bar_size - (size_t)offset);
238
239 /*
240 * Determine how many bytes to be actually read from the device memory.
241 * Read request beyond the actual device memory size is filled with ~0,
242 * while those beyond the actual reported size is skipped.
243 */
244 if (offset >= nvdev->memlength)
245 mem_count = 0;
246 else
247 mem_count = min(count, nvdev->memlength - (size_t)offset);
248
249 /*
250 * Handle read on the BAR2 region. Map to the target device memory
251 * physical address and copy to the request read buffer.
252 */
253 if (copy_to_user(buf, (u8 *)nvdev->memmap + offset, mem_count))
254 return -EFAULT;
255
256 /*
257 * Only the device memory present on the hardware is mapped, which may
258 * not be power-of-2 aligned. A read to an offset beyond the device memory
259 * size is filled with ~0.
260 */
261 for (i = mem_count; i < count; i++)
262 put_user(val, (unsigned char __user *)(buf + i));
263
264 *ppos += count;
265 return count;
266 }
267
268 static ssize_t nvgrace_gpu_vfio_pci_read(struct vfio_device *core_vdev,
269 char __user *buf, size_t count, loff_t *ppos)
270 {
271 unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
272 struct nvgrace_gpu_vfio_pci_core_device *nvdev = container_of(
273 core_vdev, struct nvgrace_gpu_vfio_pci_core_device, core_device.vdev);
274 int ret;
275
276 if (index == VFIO_PCI_BAR2_REGION_INDEX) {
277 ret = nvgrace_gpu_memmap(nvdev);
278 if (ret)
279 return ret;
280
281 return nvgrace_gpu_read_mem(buf, count, ppos, nvdev);
282 }
283
284 return vfio_pci_core_read(core_vdev, buf, count, ppos);
285 }
286
287 /*
288 * Write count bytes to the device memory at a given offset. The actual device
289 * memory size (available) may not be a power-of-2. So the driver fakes the
290 * size to a power-of-2 (reported) when exposing to a user space driver.
291 *
292 * Write request beyond the actual device size are dropped, while those
293 * beyond the actual reported size are skipped entirely.
294 *
295 * A write to a negative or an offset greater than the reported size, a
296 * negative count are considered error conditions and returned with an -EINVAL.
297 */
> 298 ssize_t nvgrace_gpu_write_mem(size_t count, loff_t *ppos, const void __user *buf, 299 struct nvgrace_gpu_vfio_pci_core_device *nvdev)
300 {
301 u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
302 size_t mem_count, bar_size = roundup_pow_of_two(nvdev->memlength);
303
304 if (offset >= bar_size)
305 return -EINVAL;
306
307 /* Clip short the write request beyond reported BAR size */
308 count = min(count, bar_size - (size_t)offset);
309
310 /*
311 * Determine how many bytes to be actually written to the device memory.
312 * Do not write to the offset beyond available size.
313 */
314 if (offset >= nvdev->memlength)
315 goto exitfn;
316
317 mem_count = min(count, nvdev->memlength - (size_t)offset);
318
319 /*
320 * Only the device memory present on the hardware is mapped, which may
321 * not be power-of-2 aligned. A write to the BAR2 region implies an
322 * access outside the available device memory on the hardware. Drop
323 * those write requests.
324 */
325 if (copy_from_user((u8 *)nvdev->memmap + offset, buf, mem_count))
326 return -EFAULT;
327
328 exitfn:
329 *ppos += count;
330 return count;
331 }
332
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki