Skip to contentSkip to conversation

vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

PATCH Superseded [email protected] 5 participants 21 messages Feb 5, 2024 – Feb 9, 2024

Subsystems (6)

v17 of 19 Superseded Latest is v19 →

lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

v17 cover vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

From: Ankit Agrawal <[email protected]>

NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device for the on-chip GPU that is the logical OS representation of the internal proprietary chip-to-chip cache coherent interconnect.

The device is peculiar compared to a real PCI device in that whilst there is a real 64b PCI BAR1 (comprising region 2 & region 3) on the device, it is not used to access device memory once the faster chip-to-chip interconnect is initialized (occurs at the time of host system boot). The device memory is accessed instead using the chip-to-chip interconnect that is exposed as a contiguous physically addressable region on the host. Since the device memory is cache coherent with the CPU, it can be mmaped into the user VMA with a cacheable mapping and used like a regular RAM. The device memory is not added to the host kernel, but mapped directly as this reduces memory wastage due to struct pages.

There is also a requirement of a reserved 1G uncached region (termed as resmem) to support the Multi-Instance GPU (MIG) feature [1]. This is to work around a HW defect. Based on [2], the requisite properties (uncached, unaligned access) can be achieved through a VM mapping (S1) of NORMAL_NC and host (S2) mapping with MemAttr[2:0]=0b101. To provide a different non-cached property to the reserved 1G region, it needs to be carved out from the device memory and mapped as a separate region in Qemu VMA with pgprot_writecombine(). pgprot_writecombine() sets the Qemu VMA page properties (pgprot) as NORMAL_NC.

Provide a VFIO PCI variant driver that adapts the unique device memory representation into a more standard PCI representation facing userspace.

The variant driver exposes these two regions - the non-cached reserved (resmem) and the cached rest of the device memory (termed as usemem) as separate VFIO 64b BAR regions. This is divergent from the baremetal approach, where the device memory is exposed as a device memory region. The decision for a different approach was taken in view of the fact that it would necessiate additional code in Qemu to discover and insert those regions in the VM IPA, along with the additional VM ACPI DSDT changes to communiate the device memory region IPA to the VM workloads. Moreover, this behavior would have to be added to a variety of emulators (beyond top of tree Qemu) out there desiring grace hopper support.

Since the device implements 64-bit BAR0, the VFIO PCI variant driver maps the uncached carved out region to the next available PCI BAR (i.e. comprising of region 2 and 3). The cached device memory aperture is assigned BAR region 4 and 5. Qemu will then naturally generate a PCI device in the VM with the uncached aperture reported as BAR2 region, the cacheable as BAR4. The variant driver provides emulation for these fake BARs' PCI config space offset registers.

The hardware ensures that the system does not crash when the memory is accessed with the memory enable turned off. It synthesis ~0 reads and dropped writes on such access. So there is no need to support the disablement/enablement of BAR through PCI_COMMAND config space register.

The memory layout on the host looks like the following:
devmem (memlength)

|--------------------------------------------------|
|-------------cached------------------------|--NC--|
|                                           |
usemem.phys/memphys                         resmem.phys

PCI BARs need to be aligned to the power-of-2, but the actual memory on the device may not. A read or write access to the physical address from the last device PFN up to the next power-of-2 aligned physical address results in reading ~0 and dropped writes. Note that the GPU device driver [6] is capable of knowing the exact device memory size through separate means. The device memory size is primarily kept in the system ACPI tables for use by the VFIO PCI variant module.

Note that the usemem memory is added by the VM Nvidia device driver [5] to the VM kernel as memblocks. Hence make the usable memory size memblock aligned.

Currently there is no provision in KVM for a S2 mapping with MemAttr[2:0]=0b101, but there is an ongoing effort to provide the same [3]. As previously mentioned, resmem is mapped pgprot_writecombine(), that sets the Qemu VMA page properties (pgprot) as NORMAL_NC. Using the proposed changes in [4] and [3], KVM marks the region with MemAttr[2:0]=0b101 in S2.

If the device memory properties are not present in the host ACPI table, the driver registers the vfio-pci-core function pointers.

This goes along with a qemu series [6] to provides the necessary implementation of the Grace Hopper Superchip firmware specification so that the guest operating system can see the correct ACPI modeling for the coherent GPU device. Verified with the CUDA workload in the VM.

[1] https://www.nvidia.com/en-in/technologies/multi-instance-gpu/
[2] section D8.5.5 of https://developer.arm.com/documentation/ddi0487/latest/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/
[5] https://github.com/NVIDIA/open-gpu-kernel-modules
[6] https://lore.kernel.org/all/[email protected]/

Applied over v6.8-rc2.

---
Link for variant driver v16:
https://lore.kernel.org/all/[email protected]/

v16 -> v17
- Moved, renamed and exported the range_intersect_range() per suggestion from Rahul Rameshbabu.
- Updated license from GPLv2 to GPL.
- Fixed S-O-B mistakes.
- Removed nvgrace_gpu_vfio_pci.h based on Alex Williamson's suggestion.
- Refactor [read]write_config_emu based on Alex's suggestion
- Added fallback to vfio-pci-core function pointers in case of absence of memory properties in the host ACPI table as per Alex's suggestion.
- Used anonymous union to represent the mapped device memory.
- Fixed code nits and rephrased comments.
- Rebased to v6.8-rc2.

v15 -> v16
- Added the missing header file causing build failure in v15.
- Moved the range_intersect_range function() to a seperate patch.
- Exported the do_io_rw as GPL and moved to the vfio-pci-core file.
- Added helper function to mask with BAR size and add flag while returning a read on the fake BARs PCI config register.
- Removed the PCI command disable.
- Removed nvgrace_gpu_vfio_pci_fake_bar_mem_region().
- Fixed miscellaneous nits.

v14 -> v15
- Added case to handle VFIO_DEVICE_IOEVENTFD to return -EIO as it is not required on the device.
- Updated the BAR config space handling code to closely resemble by Yishai Hadas (using range_intersect_range) in
https://lore.kernel.org/all/[email protected]
- Changed the bar pci config register from union to u64.
- Adapted the code to disable BAR when it is disabled through PCI_COMMAND.
- Exported and reused the do_io_rw to do mmio accesses.
- Added a new header file to keep the newly declared structures.
- Miscellaneous code fixes suggested by Alex Williamson in v14.

v13 -> v14
- Merged the changes for second BAR implementation for MIG support on the device driver.
https://lore.kernel.org/all/[email protected]/
- Added the missing implementation of sub-word access to fake BARs' PCI config access. Implemented access algorithm suggested by Alex Williamson in the comments (Thanks!)
- Added support to BAR accesses on the reserved memory with Qemu device param x-no-mmap=on.
- Handled endian-ness in the PCI config space access.
- Git commit message change

v12 -> v13
- Added emulation for the PCI config space BAR offset register for the fake BAR.
- commit message updated with more details on the BAR offset emulation.

v11 -> v12
- More details in commit message on device memory size

v10 -> v11
- Removed sysfs attribute to expose the CPU coherent memory feature
- Addressed review comments

v9 -> v10
- Add new sysfs attribute to expose the CPU coherent memory feature.

v8 -> v9
- Minor code adjustment suggested in v8.

v7 -> v8
- Various field names updated.
- Added a new function to handle VFIO_DEVICE_GET_REGION_INFO ioctl.
- Locking protection for memremap to bar region and other changes recommended in v7.
- Added code to fail if the devmem size advertized is 0 in system DSDT.

v6 -> v7
- Handled out-of-bound and overflow conditions at various places to validate input offset and length.
- Added code to return EINVAL for offset beyond region size.

v5 -> v6
- Added the code to handle BAR2 read/write using memremap to the device memory.

v4 -> v5
- Changed the module name from nvgpu-vfio-pci to nvgrace-gpu-vfio-pci.

v3 -> v4
- Mapping the available device memory using sparse mmap. The region outside the device memory is handled by read/write ops.
- Removed the fault handler added in v3.

v2 -> v3
- Added fault handler to map the region outside the physical GPU memory up to the next power-of-2 to a dummy PFN.
- Changed to select instead of "depends on" VFIO_PCI_CORE for all the vfio-pci variant driver.
- Code cleanup based on feedback comments.
- Code implemented and tested against v6.4-rc4.

v1 -> v2
- Updated the wording of reference to BAR offset and replaced with index.
- The GPU memory is exposed at the fixed BAR2_REGION_INDEX.
- Code cleanup based on feedback comments.

Ankit Agrawal (3):
vfio/pci: rename and export do_io_rw()
vfio/pci: rename and export range_intesect_range vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

 MAINTAINERS                           |   6 +
 drivers/vfio/pci/Kconfig              |   2 +
 drivers/vfio/pci/Makefile             |   2 +
 drivers/vfio/pci/nvgrace-gpu/Kconfig  |  10 +
 drivers/vfio/pci/nvgrace-gpu/Makefile |   3 +
 drivers/vfio/pci/nvgrace-gpu/main.c   | 856 ++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_config.c    |  45 ++
 drivers/vfio/pci/vfio_pci_rdwr.c      |  16 +-
 drivers/vfio/pci/virtio/main.c        |  72 +--
 include/linux/vfio_pci_core.h         |  10 +-
 10 files changed, 968 insertions(+), 54 deletions(-)
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/Kconfig
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/Makefile
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/main.c
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH v17 1/3 vfio/pci: rename and export do_io_rw()

From: Ankit Agrawal <[email protected]>

do_io_rw() is used to read/write to the device MMIO. The grace hopper VFIO PCI variant driver require this functionality to read/write to its memory.

Rename this as vfio_pci_core functions and export as GPL.

---
 drivers/vfio/pci/vfio_pci_rdwr.c | 16 +++++++++-------
 include/linux/vfio_pci_core.h    |  5 ++++-
 2 files changed, 13 insertions(+), 8 deletions(-)
drivers/vfio/pci/vfio_pci_rdwr.c +9 -7
index 07fea08ea8a2..03b8f7ada1ac 100644
@@ -96,10 +96,10 @@ VFIO_IOREAD(32)
  * reads with -1.  This is intended for handling MSI-X vector tables and
  * leftover space for ROM BARs.
  */
-static ssize_t do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
-			void __iomem *io, char __user *buf,
-			loff_t off, size_t count, size_t x_start,
-			size_t x_end, bool iswrite)
+ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
+			       void __iomem *io, char __user *buf,
+			       loff_t off, size_t count, size_t x_start,
+			       size_t x_end, bool iswrite)
 {
 	ssize_t done = 0;
 	int ret;
@@ -201,6 +201,7 @@ static ssize_t do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
 
 	return done;
 }
+EXPORT_SYMBOL_GPL(vfio_pci_core_do_io_rw);
 
 int vfio_pci_core_setup_barmap(struct vfio_pci_core_device *vdev, int bar)
 {
@@ -279,8 +280,8 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 		x_end = vdev->msix_offset + vdev->msix_size;
 	}
 
-	done = do_io_rw(vdev, res->flags & IORESOURCE_MEM, io, buf, pos,
-			count, x_start, x_end, iswrite);
+	done = vfio_pci_core_do_io_rw(vdev, res->flags & IORESOURCE_MEM, io, buf, pos,
+				      count, x_start, x_end, iswrite);
 
 	if (done >= 0)
 		*ppos += done;
@@ -348,7 +349,8 @@ ssize_t vfio_pci_vga_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 	 * probing, so we don't currently worry about access in relation
 	 * to the memory enable bit in the command register.
 	 */
-	done = do_io_rw(vdev, false, iomem, buf, off, count, 0, 0, iswrite);
+	done = vfio_pci_core_do_io_rw(vdev, false, iomem, buf, off, count,
+				      0, 0, iswrite);
 
 	vga_put(vdev->pdev, rsrc);
 
include/linux/vfio_pci_core.h +4 -1
index 85e84b92751b..cf9480a31f3e 100644
@@ -130,7 +130,10 @@ void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev);
 int vfio_pci_core_setup_barmap(struct vfio_pci_core_device *vdev, int bar);
 pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev,
 						pci_channel_state_t state);
-
+ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
+			       void __iomem *io, char __user *buf,
+			       loff_t off, size_t count, size_t x_start,
+			       size_t x_end, bool iswrite);
 #define VFIO_IOWRITE_DECLATION(size) \
 int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev,	\
 			bool test_mem, u##size val, void __iomem *io);
Signature · 1 line
2.34.1
reviewer Reviewed-by on v17 1/3
#2 lore raw

RE: [PATCH v17 1/3] vfio/pci: rename and export do_io_rw()

Quoted 5 lines

From: [email protected] <[email protected]>
Sent: Tuesday, February 6, 2024 7:01 AM

From: Ankit Agrawal <[email protected]>

Quoted 7 lines

do_io_rw() is used to read/write to the device MMIO. The grace hopper VFIO PCI variant driver require this functionality to read/write to its memory.

Rename this as vfio_pci_core functions and export as GPL.

author maintainer
#3 lore raw

PATCH v17 2/3 vfio/pci: rename and export range_intesect_range

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

range_intesect_range determines an overlap between two ranges. If an overlap, the helper function returns the overlapping offset and size.

The VFIO PCI variant driver emulates the PCI config space BAR offset registers. These offset may be accessed for read/write with a variety of lengths including sub-word sizes from sub-word offsets. The driver makes use of this helper function to read/write the targeted part of the emulated register.

Make this a vfio_pci_core function, rename and export as GPL. Also update references in virtio driver.

---
 drivers/vfio/pci/vfio_pci_config.c | 45 +++++++++++++++++++
 drivers/vfio/pci/virtio/main.c     | 72 +++++++++++-------------------
 include/linux/vfio_pci_core.h      |  5 +++
 3 files changed, 76 insertions(+), 46 deletions(-)
drivers/vfio/pci/vfio_pci_config.c +45 -0
index 672a1804af6a..4fc3c605af13 100644
@@ -1966,3 +1966,48 @@ ssize_t vfio_pci_config_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 
 	return done;
 }
+
+/**
+ * vfio_pci_core_range_intersect_range() - Determine overlap between a buffer
+ *					   and register offset ranges.
+ * @range1_start:    start offset of the buffer
+ * @count1:	     number of buffer bytes.
+ * @range2_start:    start register offset
+ * @count2:	     number of bytes of register
+ * @start_offset:    start offset of overlap start in the buffer
+ * @intersect_count: number of overlapping bytes
+ * @register_offset: start offset of overlap start in register
+ *
+ * The function determines an overlap between a register and a buffer.
+ * range1 represents the buffer and range2 represents register.
+ *
+ * Returns: true if there is overlap, false if not.
+ * The overlap start and range is returned through function args.
+ */
+bool vfio_pci_core_range_intersect_range(loff_t range1_start, size_t count1,
+					 loff_t range2_start, size_t count2,
+					 loff_t *start_offset,
+					 size_t *intersect_count,
+					 size_t *register_offset)
+{
+	if (range1_start <= range2_start &&
+	    range1_start + count1 > range2_start) {
+		*start_offset = range2_start - range1_start;
+		*intersect_count = min_t(size_t, count2,
+					 range1_start + count1 - range2_start);
+		*register_offset = 0;
+		return true;
+	}
+
+	if (range1_start > range2_start &&
+	    range1_start < range2_start + count2) {
+		*start_offset = 0;
+		*intersect_count = min_t(size_t, count1,
+					 range2_start + count2 - range1_start);
+		*register_offset = range1_start - range2_start;
+		return true;
+	}
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_range_intersect_range);
drivers/vfio/pci/virtio/main.c +26 -46

126 lines not on this page · raw message

include/linux/vfio_pci_core.h +5 -0
index cf9480a31f3e..1a056b5ef417 100644
@@ -134,6 +134,11 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
 			       void __iomem *io, char __user *buf,
 			       loff_t off, size_t count, size_t x_start,
 			       size_t x_end, bool iswrite);
+bool vfio_pci_core_range_intersect_range(loff_t range1_start, size_t count1,
+					 loff_t range2_start, size_t count2,
+					 loff_t *start_offset,
+					 size_t *intersect_count,
+					 size_t *register_offset);
 #define VFIO_IOWRITE_DECLATION(size) \
 int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev,	\
 			bool test_mem, u##size val, void __iomem *io);
Signature · 1 line
2.34.1
reviewer Reviewed-by on v17 2/3
#4 lore raw

RE: [PATCH v17 2/3] vfio/pci: rename and export range_intesect_range

Quoted 6 lines

From: [email protected] <[email protected]>
Sent: Tuesday, February 6, 2024 7:01 AM

From: Ankit Agrawal <[email protected]>

range_intesect_range determines an overlap between two ranges. If an

s/intesect/intersect/

Quoted 47 lines

overlap, the helper function returns the overlapping offset and size.

The VFIO PCI variant driver emulates the PCI config space BAR offset registers. These offset may be accessed for read/write with a variety of lengths including sub-word sizes from sub-word offsets. The driver makes use of this helper function to read/write the targeted part of the emulated register.

Make this a vfio_pci_core function, rename and export as GPL. Also update references in virtio driver.

---
 drivers/vfio/pci/vfio_pci_config.c | 45 +++++++++++++++++++
 drivers/vfio/pci/virtio/main.c     | 72 +++++++++++-------------------
 include/linux/vfio_pci_core.h      |  5 +++
 3 files changed, 76 insertions(+), 46 deletions(-)
drivers/vfio/pci/vfio_pci_config.c +3 -0
index 672a1804af6a..4fc3c605af13 100644
@@ -1966,3 +1966,48 @@ ssize_t vfio_pci_config_rw(struct
 vfio_pci_core_device *vdev, char __user *buf,
 
 	return done;
 }
+
+/**
+ * vfio_pci_core_range_intersect_range() - Determine overlap between a

buffer
+ * and register offset ranges.
+ * @range1_start: start offset of the buffer
+ * @count1: number of buffer bytes.
+ * @range2_start: start register offset
+ * @count2: number of bytes of register
+ * @start_offset: start offset of overlap start in the buffer
+ * @intersect_count: number of overlapping bytes
+ * @register_offset: start offset of overlap start in register
+ *
+ * The function determines an overlap between a register and a buffer.
+ * range1 represents the buffer and range2 represents register.
+ *
+ * Returns: true if there is overlap, false if not.
+ * The overlap start and range is returned through function args.
+ */

drivers/vfio/pci/vfio_pci_config.c ssize_t vfio_pci_config_rw(struct vfio_pci_core_device *vdev, char __user *buf, · line 1991

Quoted 5 lines
drivers/vfio/pci/vfio_pci_config.c +5 -0
+bool vfio_pci_core_range_intersect_range(loff_t range1_start, size_t count1,
+					 loff_t range2_start, size_t count2,
+					 loff_t *start_offset,
+					 size_t *intersect_count,
+					 size_t *register_offset)

based on description it's probably clearer to rename:

range1_start -> buf_start
count1 -> buf_cnt
range2_start -> reg_start
count2 -> reg_cnt
start_offset -> buf_offset

but not big deal, so:

author maintainer
#5 lore raw

Re: [PATCH v17 2/3] vfio/pci: rename and export range_intesect_range

Thanks Kevin.

Quoted 3 lines
Quoted 1 line

range_intesect_range determines an overlap between two ranges. If an

s/intesect/intersect/

Will fix the typo.

Quoted 25 lines
Quoted 14 lines

+ * vfio_pci_core_range_intersect_range() - Determine overlap between a buffer
+ *                                      and register offset ranges.
+ * @range1_start:    start offset of the buffer
+ * @count1:       number of buffer bytes.
+ * @range2_start:    start register offset
+ * @count2:       number of bytes of register
+ * @start_offset:    start offset of overlap start in the buffer
+ * @intersect_count: number of overlapping bytes
+ * @register_offset: start offset of overlap start in register
+ *
+ * The function determines an overlap between a register and a buffer.
+ * range1 represents the buffer and range2 represents register.
+ *

Quoted 8 lines

+ * Returns: true if there is overlap, false if not.
+ * The overlap start and range is returned through function args.
+ */
+bool vfio_pci_core_range_intersect_range(loff_t range1_start, size_t count1, +                                      loff_t range2_start, size_t count2, +                                      loff_t *start_offset,
+                                      size_t *intersect_count,
+                                      size_t *register_offset)

based on description it's probably clearer to rename:

Quoted 7 lines

range1_start -> buf_start
count1 -> buf_cnt
range2_start -> reg_start
count2 -> reg_cnt
start_offset -> buf_offset

but not big deal, so:

Fine by me. Will rename them.

Quoted 1 line

Thanks!

author maintainer
#6 lore raw

PATCH v17 3/3 vfio/nvgrace-gpu: Add vfio pci variant module for grace hopper

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device for the on-chip GPU that is the logical OS representation of the internal proprietary chip-to-chip cache coherent interconnect.

The device is peculiar compared to a real PCI device in that whilst there is a real 64b PCI BAR1 (comprising region 2 & region 3) on the device, it is not used to access device memory once the faster chip-to-chip interconnect is initialized (occurs at the time of host system boot). The device memory is accessed instead using the chip-to-chip interconnect that is exposed as a contiguous physically addressable region on the host. This device memory aperture can be obtained from host ACPI table using device_property_read_u64(), according to the FW specification. Since the device memory is cache coherent with the CPU, it can be mmap into the user VMA with a cacheable mapping using remap_pfn_range() and used like a regular RAM. The device memory is not added to the host kernel, but mapped directly as this reduces memory wastage due to struct pages.

There is also a requirement of a reserved 1G uncached region (termed as resmem) to support the Multi-Instance GPU (MIG) feature [1]. This is to work around a HW defect. Based on [2], the requisite properties (uncached, unaligned access) can be achieved through a VM mapping (S1) of NORMAL_NC and host (S2) mapping with MemAttr[2:0]=0b101. To provide a different non-cached property to the reserved 1G region, it needs to be carved out from the device memory and mapped as a separate region in Qemu VMA with pgprot_writecombine(). pgprot_writecombine() sets the Qemu VMA page properties (pgprot) as NORMAL_NC.

Provide a VFIO PCI variant driver that adapts the unique device memory representation into a more standard PCI representation facing userspace.

The variant driver exposes these two regions - the non-cached reserved (resmem) and the cached rest of the device memory (termed as usemem) as separate VFIO 64b BAR regions. This is divergent from the baremetal approach, where the device memory is exposed as a device memory region. The decision for a different approach was taken in view of the fact that it would necessiate additional code in Qemu to discover and insert those regions in the VM IPA, along with the additional VM ACPI DSDT changes to communicate the device memory region IPA to the VM workloads. Moreover, this behavior would have to be added to a variety of emulators (beyond top of tree Qemu) out there desiring grace hopper support.

Since the device implements 64-bit BAR0, the VFIO PCI variant driver maps the uncached carved out region to the next available PCI BAR (i.e. comprising of region 2 and 3). The cached device memory aperture is assigned BAR region 4 and 5. Qemu will then naturally generate a PCI device in the VM with the uncached aperture reported as BAR2 region, the cacheable as BAR4. The variant driver provides emulation for these fake BARs' PCI config space offset registers.

The hardware ensures that the system does not crash when the memory is accessed with the memory enable turned off. It synthesis ~0 reads and dropped writes on such access. So there is no need to support the disablement/enablement of BAR through PCI_COMMAND config space register.

The memory layout on the host looks like the following:
devmem (memlength)

|--------------------------------------------------|
|-------------cached------------------------|--NC--|
|                                           |
usemem.phys/memphys                         resmem.phys

PCI BARs need to be aligned to the power-of-2, but the actual memory on the device may not. A read or write access to the physical address from the last device PFN up to the next power-of-2 aligned physical address results in reading ~0 and dropped writes. Note that the GPU device driver [6] is capable of knowing the exact device memory size through separate means. The device memory size is primarily kept in the system ACPI tables for use by the VFIO PCI variant module.

Note that the usemem memory is added by the VM Nvidia device driver [5] to the VM kernel as memblocks. Hence make the usable memory size memblock aligned.

Currently there is no provision in KVM for a S2 mapping with MemAttr[2:0]=0b101, but there is an ongoing effort to provide the same [3]. As previously mentioned, resmem is mapped pgprot_writecombine(), that sets the Qemu VMA page properties (pgprot) as NORMAL_NC. Using the proposed changes in [4] and [3], KVM marks the region with MemAttr[2:0]=0b101 in S2.

If the bare metal properties are not present, the driver registers the vfio-pci-core function pointers.

This goes along with a qemu series [6] to provides the necessary implementation of the Grace Hopper Superchip firmware specification so that the guest operating system can see the correct ACPI modeling for the coherent GPU device. Verified with the CUDA workload in the VM.

[1] https://www.nvidia.com/en-in/technologies/multi-instance-gpu/
[2] section D8.5.5 of https://developer.arm.com/documentation/ddi0487/latest/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/
[5] https://github.com/NVIDIA/open-gpu-kernel-modules
[6] https://lore.kernel.org/all/[email protected]/

---
 MAINTAINERS                           |   6 +
 drivers/vfio/pci/Kconfig              |   2 +
 drivers/vfio/pci/Makefile             |   2 +
 drivers/vfio/pci/nvgrace-gpu/Kconfig  |  10 +
 drivers/vfio/pci/nvgrace-gpu/Makefile |   3 +
 drivers/vfio/pci/nvgrace-gpu/main.c   | 856 ++++++++++++++++++++++++++
 6 files changed, 879 insertions(+)
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/Kconfig
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/Makefile
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/main.c
MAINTAINERS +6 -0
index 8999497011a2..529ec8966f58 100644
@@ -23103,6 +23103,12 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/vfio/platform/
 
+VFIO NVIDIA GRACE GPU DRIVER
+M:	Ankit Agrawal <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	drivers/vfio/pci/nvgrace-gpu/
+
 VGA_SWITCHEROO
 R:	Lukas Wunner <[email protected]>
 S:	Maintained
drivers/vfio/pci/Kconfig +2 -0
index 18c397df566d..15821a2d77d2 100644
@@ -67,4 +67,6 @@ source "drivers/vfio/pci/pds/Kconfig"
 
 source "drivers/vfio/pci/virtio/Kconfig"
 
+source "drivers/vfio/pci/nvgrace-gpu/Kconfig"
+
 endmenu
drivers/vfio/pci/Makefile +2 -0
index 046139a4eca5..ce7a61f1d912 100644
@@ -15,3 +15,5 @@ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/
 obj-$(CONFIG_PDS_VFIO_PCI) += pds/
 
 obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/
+
+obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/
drivers/vfio/pci/nvgrace-gpu/Kconfig new file +10 -0
new file mode 100644
index 000000000000..936e88d8d41d
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config NVGRACE_GPU_VFIO_PCI
+	tristate "VFIO support for the GPU in the NVIDIA Grace Hopper Superchip"
+	depends on ARM64 || (COMPILE_TEST && 64BIT)
+	select VFIO_PCI_CORE
+	help
+	  VFIO support for the GPU in the NVIDIA Grace Hopper Superchip is
+	  required to assign the GPU device using KVM/qemu/etc.
+
+	  If you don't know what to do here, say N.
drivers/vfio/pci/nvgrace-gpu/Makefile new file +3 -0
new file mode 100644
index 000000000000..3ca8c187897a
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu-vfio-pci.o
+nvgrace-gpu-vfio-pci-y := main.o
drivers/vfio/pci/nvgrace-gpu/main.c new file +856 -0

856 lines not on this page · raw message

Signature · 1 line
2.34.1
#7 lore raw
Quoted 211 lines · On Tue, 6 Feb 2024 04:31:23 +0530 <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device for the on-chip GPU that is the logical OS representation of the internal proprietary chip-to-chip cache coherent interconnect.

The device is peculiar compared to a real PCI device in that whilst there is a real 64b PCI BAR1 (comprising region 2 & region 3) on the device, it is not used to access device memory once the faster chip-to-chip interconnect is initialized (occurs at the time of host system boot). The device memory is accessed instead using the chip-to-chip interconnect that is exposed as a contiguous physically addressable region on the host. This device memory aperture can be obtained from host ACPI table using device_property_read_u64(), according to the FW specification. Since the device memory is cache coherent with the CPU, it can be mmap into the user VMA with a cacheable mapping using remap_pfn_range() and used like a regular RAM. The device memory is not added to the host kernel, but mapped directly as this reduces memory wastage due to struct pages.

There is also a requirement of a reserved 1G uncached region (termed as resmem) to support the Multi-Instance GPU (MIG) feature [1]. This is to work around a HW defect. Based on [2], the requisite properties (uncached, unaligned access) can be achieved through a VM mapping (S1) of NORMAL_NC and host (S2) mapping with MemAttr[2:0]=0b101. To provide a different non-cached property to the reserved 1G region, it needs to be carved out from the device memory and mapped as a separate region in Qemu VMA with pgprot_writecombine(). pgprot_writecombine() sets the Qemu VMA page properties (pgprot) as NORMAL_NC.

Provide a VFIO PCI variant driver that adapts the unique device memory representation into a more standard PCI representation facing userspace.

The variant driver exposes these two regions - the non-cached reserved (resmem) and the cached rest of the device memory (termed as usemem) as separate VFIO 64b BAR regions. This is divergent from the baremetal approach, where the device memory is exposed as a device memory region. The decision for a different approach was taken in view of the fact that it would necessiate additional code in Qemu to discover and insert those regions in the VM IPA, along with the additional VM ACPI DSDT changes to communicate the device memory region IPA to the VM workloads. Moreover, this behavior would have to be added to a variety of emulators (beyond top of tree Qemu) out there desiring grace hopper support.

Since the device implements 64-bit BAR0, the VFIO PCI variant driver maps the uncached carved out region to the next available PCI BAR (i.e. comprising of region 2 and 3). The cached device memory aperture is assigned BAR region 4 and 5. Qemu will then naturally generate a PCI device in the VM with the uncached aperture reported as BAR2 region, the cacheable as BAR4. The variant driver provides emulation for these fake BARs' PCI config space offset registers.

The hardware ensures that the system does not crash when the memory is accessed with the memory enable turned off. It synthesis ~0 reads and dropped writes on such access. So there is no need to support the disablement/enablement of BAR through PCI_COMMAND config space register.

The memory layout on the host looks like the following:
devmem (memlength)

|--------------------------------------------------|
|-------------cached------------------------|--NC--|
|                                           |
usemem.phys/memphys                         resmem.phys

PCI BARs need to be aligned to the power-of-2, but the actual memory on the device may not. A read or write access to the physical address from the last device PFN up to the next power-of-2 aligned physical address results in reading ~0 and dropped writes. Note that the GPU device driver [6] is capable of knowing the exact device memory size through separate means. The device memory size is primarily kept in the system ACPI tables for use by the VFIO PCI variant module.

Note that the usemem memory is added by the VM Nvidia device driver
[5] to the VM kernel as memblocks. Hence make the usable memory size memblock aligned.

Currently there is no provision in KVM for a S2 mapping with MemAttr[2:0]=0b101, but there is an ongoing effort to provide the same [3]. As previously mentioned, resmem is mapped pgprot_writecombine(), that sets the Qemu VMA page properties (pgprot) as NORMAL_NC. Using the proposed changes in [4] and [3], KVM marks the region with MemAttr[2:0]=0b101 in S2.

If the bare metal properties are not present, the driver registers the vfio-pci-core function pointers.

This goes along with a qemu series [6] to provides the necessary implementation of the Grace Hopper Superchip firmware specification so that the guest operating system can see the correct ACPI modeling for the coherent GPU device. Verified with the CUDA workload in the VM.

[1] https://www.nvidia.com/en-in/technologies/multi-instance-gpu/
[2] section D8.5.5 of
https://developer.arm.com/documentation/ddi0487/latest/ [3]
https://lore.kernel.org/all/[email protected]/
[4]
https://lore.kernel.org/all/[email protected]/
[5] https://github.com/NVIDIA/open-gpu-kernel-modules [6]
https://lore.kernel.org/all/[email protected]/

---
 MAINTAINERS                           |   6 +
 drivers/vfio/pci/Kconfig              |   2 +
 drivers/vfio/pci/Makefile             |   2 +
 drivers/vfio/pci/nvgrace-gpu/Kconfig  |  10 +
 drivers/vfio/pci/nvgrace-gpu/Makefile |   3 +
 drivers/vfio/pci/nvgrace-gpu/main.c   | 856
++++++++++++++++++++++++++ 6 files changed, 879 insertions(+)
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/Kconfig
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/Makefile
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/main.c
MAINTAINERS +6 -0
index 8999497011a2..529ec8966f58 100644
@@ -23103,6 +23103,12 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/vfio/platform/
 
+VFIO NVIDIA GRACE GPU DRIVER
+M:	Ankit Agrawal <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	drivers/vfio/pci/nvgrace-gpu/
+
 VGA_SWITCHEROO
 R:	Lukas Wunner <[email protected]>
 S:	Maintained
drivers/vfio/pci/Kconfig +2 -0
index 18c397df566d..15821a2d77d2 100644
@@ -67,4 +67,6 @@ source "drivers/vfio/pci/pds/Kconfig"
 
 source "drivers/vfio/pci/virtio/Kconfig"
 
+source "drivers/vfio/pci/nvgrace-gpu/Kconfig"
+
 endmenu
drivers/vfio/pci/Makefile +2 -0
index 046139a4eca5..ce7a61f1d912 100644
@@ -15,3 +15,5 @@ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/
 obj-$(CONFIG_PDS_VFIO_PCI) += pds/
 
 obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/
+
+obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/
(quoted diff) +0 -0

b/drivers/vfio/pci/nvgrace-gpu/Kconfig new file mode 100644 index 000000000000..936e88d8d41d

drivers/vfio/pci/nvgrace-gpu/Kconfig new file +3 -0
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config NVGRACE_GPU_VFIO_PCI
+	tristate "VFIO support for the GPU in the NVIDIA Grace

Hopper Superchip"

drivers/vfio/pci/nvgrace-gpu/Kconfig new file +4 -0
+	depends on ARM64 || (COMPILE_TEST && 64BIT)
+	select VFIO_PCI_CORE
+	help
+	  VFIO support for the GPU in the NVIDIA Grace Hopper

Superchip is

drivers/vfio/pci/nvgrace-gpu/Kconfig new file +3 -0
+	  required to assign the GPU device using KVM/qemu/etc.
+
+	  If you don't know what to do here, say N.
(quoted diff) +0 -0

b/drivers/vfio/pci/nvgrace-gpu/Makefile new file mode 100644 index 000000000000..3ca8c187897a

drivers/vfio/pci/nvgrace-gpu/Makefile new file +3 -0
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu-vfio-pci.o
+nvgrace-gpu-vfio-pci-y := main.o
(quoted diff) +0 -0

b/drivers/vfio/pci/nvgrace-gpu/main.c new file mode 100644 index 000000000000..6279af2bc6b8

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
@@ -0,0 +1,856 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights

reserved
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+
+#include <linux/vfio_pci_core.h>
+
+/*
+ * The device memory usable to the workloads running in the VM is

cached
+ * and showcased as a 64b device BAR (comprising of BAR4 and BAR5 region)
+ * to the VM and is represented as usemem.
+ * Moreover, the VM GPU device driver needs a non-cacheable region to
+ * support the MIG feature. This region is also exposed as a 64b BAR
+ * (comprising of BAR2 and BAR3 region) and represented as resmem.
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+#define RESMEM_REGION_INDEX VFIO_PCI_BAR2_REGION_INDEX
+#define USEMEM_REGION_INDEX VFIO_PCI_BAR4_REGION_INDEX
+
+/* Memory size expected as non cached and reserved by the VM driver

*/ +#define RESMEM_SIZE 0x40000000

drivers/vfio/pci/nvgrace-gpu/main.c

Quoted 2 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+#define MEMBLK_SIZE 0x20000000
+

Maybe use SZ_* definitions in linux/size.h

Quoted 60 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+/*
+ * The state of the two device memory region - resmem and usemem - is
+ * saved as struct mem_region.
+ */
+struct mem_region {
+	phys_addr_t memphys;    /* Base physical address of the

region */

drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+	size_t memlength;       /* Region size */
+	size_t bar_size;        /* Reported region BAR size */
+	__le64 bar_val;         /* Emulated BAR offset registers */
+	union {
+		void *memaddr;
+		void __iomem *ioaddr;
+	};                      /* Base virtual address of the

region */ +};

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+
+struct nvgrace_gpu_vfio_pci_core_device {
+	struct vfio_pci_core_device core_device;
+	/* Cached and usable memory for the VM. */
+	struct mem_region usemem;
+	/* Non cached memory carved out from the end of device

memory */

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	struct mem_region resmem;
+	/* Lock to control device memory kernel mapping */
+	struct mutex remap_lock;
+};
+
+static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device

*core_vdev) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+			     core_device.vdev);
+
+	nvdev->resmem.bar_val = 0;
+	nvdev->usemem.bar_val = 0;
+}
+
+/* Choose the structure corresponding to the fake BAR with a given

index. */ +static struct mem_region *

drivers/vfio/pci/nvgrace-gpu/main.c new file +16 -0
+nvgrace_gpu_memregion(int index,
+		      struct nvgrace_gpu_vfio_pci_core_device *nvdev)
+{
+	if (index == USEMEM_REGION_INDEX)
+		return &nvdev->usemem;
+
+	if (index == RESMEM_REGION_INDEX)
+		return &nvdev->resmem;
+
+	return NULL;
+}
+
+static int nvgrace_gpu_open_device(struct vfio_device *core_vdev)
+{
+	struct vfio_pci_core_device *vdev =
+		container_of(core_vdev, struct vfio_pci_core_device,

vdev);

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c line 86

Quoted 9 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+			     core_device.vdev);
+	int ret;
+
+	ret = vfio_pci_core_enable(vdev);
+	if (ret)
+		return ret;
+
+	vfio_pci_core_finish_enable(vdev);
+

---

drivers/vfio/pci/nvgrace-gpu/main.c line 91

Quoted 5 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+	if (nvdev->usemem.memlength) {
+		nvgrace_gpu_init_fake_bar_emu_regs(core_vdev);
+		mutex_init(&nvdev->remap_lock);
+	}
+

---

Better move this part to the place between vfio_pci_core_enable() and vfio_pci_core_finish_enable() like others for respecting the expected device initialization sequence of life cycle.

It doesn't bite something right now, but think about when someone changes the behavior of vfio_pci_core_finish_enable() in the future, they have to propose a patch for this.

drivers/vfio/pci/nvgrace-gpu/main.c

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+	return 0;
+}
+
+static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+			     core_device.vdev);
+

Shouldn't there be a lock/unlock sequence when using/freeing the map? As the vfio device open/close paths are protected by group/dev_set lock, probably they are fine without driver-level lock protection. But for vfio_device_read/write/ioctl, if they are racing with open/close path, I think there should be a lock/unlock to protect the map.

---

drivers/vfio/pci/nvgrace-gpu/main.c line 112

Quoted 13 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+	/* Unmap the mapping to the device memory cached region */
+	if (nvdev->usemem.memaddr) {
+		memunmap(nvdev->usemem.memaddr);
+		nvdev->usemem.memaddr = NULL;
+	}
+
+	/* Unmap the mapping to the device memory non-cached region

*/

drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+	if (nvdev->resmem.ioaddr) {
+		iounmap(nvdev->resmem.ioaddr);
+		nvdev->resmem.ioaddr = NULL;
+	}
+

---

Quoted 367 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +10 -0
+	mutex_destroy(&nvdev->remap_lock);
+
+	vfio_pci_core_close_device(core_vdev);
+}
+
+static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
+			    struct vm_area_struct *vma)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+			     core_device.vdev);
+
+	unsigned long start_pfn;
+	unsigned int index;
+	u64 req_len, pgoff, end;
+	int ret = 0;
+	struct mem_region *memregion;
+
+	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT -

PAGE_SHIFT); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return vfio_pci_core_mmap(core_vdev, vma);
+
+	/*
+	 * Request to mmap the BAR. Map to the CPU accessible memory

on the

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * GPU using the memory information gathered from the system

ACPI

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	 * tables.
+	 */
+	pgoff = vma->vm_pgoff &
+		((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
+
+	if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len)

||

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	    check_add_overflow(PHYS_PFN(memregion->memphys), pgoff,

&start_pfn) ||

drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+	    check_add_overflow(PFN_PHYS(pgoff), req_len, &end))
+		return -EOVERFLOW;
+
+	/*
+	 * Check that the mapping request does not go beyond

available device

drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+	 * memory size
+	 */
+	if (end > memregion->memlength)
+		return -EINVAL;
+
+	/*
+	 * The carved out region of the device memory needs the

NORMAL_NC

drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+	 * property. Communicate as such to the hypervisor.
+	 */
+	if (index == RESMEM_REGION_INDEX)
+		vma->vm_page_prot =

pgprot_writecombine(vma->vm_page_prot); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	/*
+	 * Perform a PFN map to the memory and back the device BAR

by the

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+	 * GPU memory.
+	 *
+	 * The available GPU memory size may not be power-of-2

aligned. The

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * remainder is only backed by vfio_device_ops read/write

handlers.

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	 *
+	 * During device reset, the GPU is safely disconnected to

the CPU

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * and access to the BAR will be immediately returned

preventing

drivers/vfio/pci/nvgrace-gpu/main.c new file +17 -0
+	 * machine check.
+	 */
+	ret = remap_pfn_range(vma, vma->vm_start, start_pfn,
+			      req_len, vma->vm_page_prot);
+	if (ret)
+		return ret;
+
+	vma->vm_pgoff = start_pfn;
+
+	return 0;
+}
+
+static long
+nvgrace_gpu_ioctl_get_region_info(struct vfio_device *core_vdev,
+				  unsigned long arg)
+{
+	unsigned long minsz = offsetofend(struct vfio_region_info,

offset);

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +18 -0
+			     core_device.vdev);
+	struct vfio_region_info_cap_sparse_mmap *sparse;
+	struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
+	struct vfio_region_info info;
+	struct mem_region *memregion;
+	u32 size;
+	int ret;
+
+	if (copy_from_user(&info, (void __user *)arg, minsz))
+		return -EFAULT;
+
+	if (info.argsz < minsz)
+		return -EINVAL;
+
+	memregion = nvgrace_gpu_memregion(info.index, nvdev);
+	if (!memregion)
+		return vfio_pci_core_ioctl(core_vdev,
+

VFIO_DEVICE_GET_REGION_INFO, arg); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +64 -0
+	/*
+	 * Request to determine the BAR region information. Send the
+	 * GPU memory information.
+	 */
+	size = struct_size(sparse, areas, 1);
+
+	/*
+	 * Setup for sparse mapping for the device memory. Only the
+	 * available device memory on the hardware is shown as a
+	 * mappable region.
+	 */
+	sparse = kzalloc(size, GFP_KERNEL);
+	if (!sparse)
+		return -ENOMEM;
+
+	sparse->nr_areas = 1;
+	sparse->areas[0].offset = 0;
+	sparse->areas[0].size = memregion->memlength;
+	sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
+	sparse->header.version = 1;
+
+	ret = vfio_info_add_capability(&caps, &sparse->header, size);
+	kfree(sparse);
+	if (ret)
+		return ret;
+
+	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
+	/*
+	 * The region memory size may not be power-of-2 aligned.
+	 * Given that the memory  as a BAR and may not be
+	 * aligned, roundup to the next power-of-2.
+	 */
+	info.size = memregion->bar_size;
+	info.flags = VFIO_REGION_INFO_FLAG_READ |
+		     VFIO_REGION_INFO_FLAG_WRITE |
+		     VFIO_REGION_INFO_FLAG_MMAP;
+
+	if (caps.size) {
+		info.flags |= VFIO_REGION_INFO_FLAG_CAPS;
+		if (info.argsz < sizeof(info) + caps.size) {
+			info.argsz = sizeof(info) + caps.size;
+			info.cap_offset = 0;
+		} else {
+			vfio_info_cap_shift(&caps, sizeof(info));
+			if (copy_to_user((void __user *)arg +
+					 sizeof(info), caps.buf,
+					 caps.size)) {
+				kfree(caps.buf);
+				return -EFAULT;
+			}
+			info.cap_offset = sizeof(info);
+		}
+		kfree(caps.buf);
+	}
+	return copy_to_user((void __user *)arg, &info, minsz) ?
+			    -EFAULT : 0;
+}
+
+static long nvgrace_gpu_ioctl(struct vfio_device *core_vdev,
+			      unsigned int cmd, unsigned long arg)
+{
+	switch (cmd) {
+	case VFIO_DEVICE_GET_REGION_INFO:
+		return nvgrace_gpu_ioctl_get_region_info(core_vdev,

arg);

drivers/vfio/pci/nvgrace-gpu/main.c new file +24 -0
+	case VFIO_DEVICE_IOEVENTFD:
+		return -ENOTTY;
+	case VFIO_DEVICE_RESET:
+		nvgrace_gpu_init_fake_bar_emu_regs(core_vdev);
+		fallthrough;
+	default:
+		return vfio_pci_core_ioctl(core_vdev, cmd, arg);
+	}
+}
+
+static __le64
+nvgrace_gpu_get_read_value(size_t bar_size, u64 flags, __le64 val64)
+{
+	u64 tmp_val;
+
+	tmp_val = le64_to_cpu(val64);
+	tmp_val &= ~(bar_size - 1);
+	tmp_val |= flags;
+
+	return cpu_to_le64(tmp_val);
+}
+
+/*
+ * Both the usable (usemem) and the reserved (resmem) device memory

region
+ * are exposed as a 64b fake BARs in the VM. These fake BARs must respond
+ * to the accesses on their respective PCI config space offsets.
+ *
+ * resmem BAR owns PCI_BASE_ADDRESS_2 & PCI_BASE_ADDRESS_3.
+ * usemem BAR owns PCI_BASE_ADDRESS_4 & PCI_BASE_ADDRESS_5.
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+static ssize_t
+nvgrace_gpu_read_config_emu(struct vfio_device *core_vdev,
+			    char __user *buf, size_t count, loff_t

*ppos) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +14 -0
+			     core_device.vdev);
+	struct mem_region *memregion = NULL;
+	u64 pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	__le64 val64;
+	size_t register_offset;
+	loff_t copy_offset;
+	size_t copy_count;
+	int ret;
+
+	ret = vfio_pci_core_read(core_vdev, buf, count, ppos);
+	if (ret < 0)
+		return ret;
+
+	if (vfio_pci_core_range_intersect_range(pos, count,

PCI_BASE_ADDRESS_2,

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+						sizeof(val64),
+						&copy_offset,

&copy_count,

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+						&register_offset))
+		memregion =

nvgrace_gpu_memregion(RESMEM_REGION_INDEX, nvdev);

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	else if (vfio_pci_core_range_intersect_range(pos, count,
+

PCI_BASE_ADDRESS_4,

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+						     sizeof(val64),
+						     &copy_offset,

&copy_count,
+
&register_offset))

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		memregion =

nvgrace_gpu_memregion(USEMEM_REGION_INDEX, nvdev); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	if (memregion) {
+		val64 =

nvgrace_gpu_get_read_value(memregion->bar_size,
+
PCI_BASE_ADDRESS_MEM_TYPE_64 |
+
PCI_BASE_ADDRESS_MEM_PREFETCH,
+
memregion->bar_val);

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+		if (copy_to_user(buf + copy_offset,
+				 (void *)&val64 + register_offset,

copy_count))

drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+			return -EFAULT;
+	}
+
+	return count;
+}
+
+static ssize_t
+nvgrace_gpu_write_config_emu(struct vfio_device *core_vdev,
+			     const char __user *buf, size_t count,

loff_t *ppos) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +8 -0
+			     core_device.vdev);
+	u64 pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	size_t register_offset;
+	loff_t copy_offset;
+	size_t copy_count;
+	struct mem_region *memregion = NULL;
+
+	if (vfio_pci_core_range_intersect_range(pos, count,

PCI_BASE_ADDRESS_2,

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+						sizeof(u64),

&copy_offset,

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+						&copy_count,

&register_offset))

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		memregion =

nvgrace_gpu_memregion(RESMEM_REGION_INDEX, nvdev);

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	else if (vfio_pci_core_range_intersect_range(pos, count,

PCI_BASE_ADDRESS_4,

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+						     sizeof(u64),

&copy_offset,

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+						     &copy_count,

&register_offset))

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		memregion =

nvgrace_gpu_memregion(USEMEM_REGION_INDEX, nvdev); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	if (memregion) {
+		if (copy_from_user((void *)&memregion->bar_val +

register_offset,

drivers/vfio/pci/nvgrace-gpu/main.c new file +11 -0
+				   buf + copy_offset, copy_count))
+			return -EFAULT;
+		*ppos += copy_count;
+		return copy_count;
+	}
+
+	return vfio_pci_core_write(core_vdev, buf, count, ppos);
+}
+
+/*
+ * Ad hoc map the device memory in the module kernel VA space.

Primarily needed
+ * as vfio does not require the userspace driver to only perform accesses through
+ * mmaps of the vfio-pci BAR regions and such accesses should be supported using
+ * vfio_device_ops read/write implementations.
+ *
+ * The usemem region is cacheable memory and hence is memremaped.
+ * The resmem region is non-cached and is mapped using ioremap_wc (NORMAL_NC).
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+static int
+nvgrace_gpu_map_device_mem(int index,
+			   struct nvgrace_gpu_vfio_pci_core_device

*nvdev) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +15 -0
+	struct mem_region *memregion;
+	int ret = 0;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return -EINVAL;
+
+	mutex_lock(&nvdev->remap_lock);
+	if (index == USEMEM_REGION_INDEX && !memregion->memaddr) {
+		memregion->memaddr = memremap(memregion->memphys,
+					      memregion->memlength,
+					      MEMREMAP_WB);
+		if (!memregion->memaddr)
+			ret = -ENOMEM;
+	} else if (index == RESMEM_REGION_INDEX &&

!memregion->ioaddr) {

drivers/vfio/pci/nvgrace-gpu/main.c new file +12 -0
+		memregion->ioaddr = ioremap_wc(memregion->memphys,
+					       memregion->memlength);
+		if (!memregion->ioaddr)
+			ret = -ENOMEM;
+	}
+	mutex_unlock(&nvdev->remap_lock);
+
+	return ret;
+}
+
+/*
+ * Read the data from the device memory (mapped either through

ioremap
+ * or memremap) into the user buffer.
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+static int
+nvgrace_gpu_map_and_read(struct nvgrace_gpu_vfio_pci_core_device

*nvdev,

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+			 char __user *buf, size_t mem_count, loff_t

*ppos) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
+	int ret;
+
+	/*
+	 * Handle read on the BAR regions. Map to the target device

memory

drivers/vfio/pci/nvgrace-gpu/main.c line 439

Quoted 6 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	 * physical address and copy to the request read buffer.
+	 */
+	ret = nvgrace_gpu_map_device_mem(index, nvdev);
+	if (ret)
+		return ret;
+

Wouldn't it be better to do the map in the open path?

Quoted 70 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+	if (index == USEMEM_REGION_INDEX) {
+		if (copy_to_user(buf,
+				 (u8 *)nvdev->usemem.memaddr +

offset,

drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+				 mem_count))
+			ret = -EFAULT;
+	} else {
+		/*
+		 * The hardware ensures that the system does not

crash when

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * the device memory is accessed with the memory

enable

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * turned off. It synthesizes ~0 on such read. So

there is

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * no need to check or support the

disablement/enablement of

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * BAR through PCI_COMMAND config space register.

Pass

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+		 * test_mem flag as false.
+		 */
+		ret = vfio_pci_core_do_io_rw(&nvdev->core_device,

false,

drivers/vfio/pci/nvgrace-gpu/main.c new file +10 -0
+					     nvdev->resmem.ioaddr,
+					     buf, offset, mem_count,
+					     0, 0, false);
+	}
+
+	return ret;
+}
+
+/*
+ * Read count bytes from the device memory at an offset. The actual

device
+ * memory size (available) may not be a power-of-2. So the driver fakes
+ * the size to a power-of-2 (reported) when exposing to a user space driver.
+ *
+ * Reads extending beyond the reported size are truncated; reads starting
+ * beyond the reported size generate -EINVAL; reads extending beyond the
+ * actual device size is filled with ~0.
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +23 -0
+static ssize_t
+nvgrace_gpu_read_mem(struct nvgrace_gpu_vfio_pci_core_device *nvdev,
+		     char __user *buf, size_t count, loff_t *ppos)
+{
+	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct mem_region *memregion;
+	size_t mem_count, i;
+	u8 val = 0xFF;
+	int ret;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return -EINVAL;
+
+	if (offset >= memregion->bar_size)
+		return -EINVAL;
+
+	/* Clip short the read request beyond reported BAR size */
+	count = min(count, memregion->bar_size - (size_t)offset);
+
+	/*
+	 * Determine how many bytes to be actually read from the

device memory.

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * Read request beyond the actual device memory size is

filled with ~0,

drivers/vfio/pci/nvgrace-gpu/main.c line 499

Quoted 4 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+	 * while those beyond the actual reported size is skipped.
+	 */
+	if (offset >= memregion->memlength)
+		mem_count = 0;

If mem_count == 0, going through nvgrace_gpu_map_and_read() is not necessary.

Quoted 399 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	else
+		mem_count = min(count, memregion->memlength -

(size_t)offset); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	ret = nvgrace_gpu_map_and_read(nvdev, buf, mem_count, ppos);
+	if (ret)
+		return ret;
+
+	/*
+	 * Only the device memory present on the hardware is mapped,

which may

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * not be power-of-2 aligned. A read to an offset beyond the

device memory

drivers/vfio/pci/nvgrace-gpu/main.c new file +16 -0
+	 * size is filled with ~0.
+	 */
+	for (i = mem_count; i < count; i++)
+		put_user(val, (unsigned char __user *)(buf + i));
+
+	*ppos += count;
+	return count;
+}
+
+static ssize_t
+nvgrace_gpu_read(struct vfio_device *core_vdev,
+		 char __user *buf, size_t count, loff_t *ppos)
+{
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+			     core_device.vdev);
+
+	if (nvgrace_gpu_memregion(index, nvdev))
+		return nvgrace_gpu_read_mem(nvdev, buf, count, ppos);
+
+	if (index == VFIO_PCI_CONFIG_REGION_INDEX)
+		return nvgrace_gpu_read_config_emu(core_vdev, buf,

count, ppos); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+	return vfio_pci_core_read(core_vdev, buf, count, ppos);
+}
+
+/*
+ * Write the data to the device memory (mapped either through ioremap
+ * or memremap) from the user buffer.
+ */
+static int
+nvgrace_gpu_map_and_write(struct nvgrace_gpu_vfio_pci_core_device

*nvdev,

drivers/vfio/pci/nvgrace-gpu/main.c new file +18 -0
+			  const char __user *buf, size_t mem_count,
+			  loff_t *ppos)
+{
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	int ret;
+
+	ret = nvgrace_gpu_map_device_mem(index, nvdev);
+	if (ret)
+		return ret;
+
+	if (index == USEMEM_REGION_INDEX) {
+		if (copy_from_user((u8 *)nvdev->usemem.memaddr + pos,
+				   buf, mem_count))
+			return -EFAULT;
+	} else {
+		/*
+		 * The hardware ensures that the system does not

crash when

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * the device memory is accessed with the memory

enable

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * turned off. It drops such writes. So there is no

need to

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+		 * check or support the disablement/enablement of BAR
+		 * through PCI_COMMAND config space register. Pass

test_mem

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+		 * flag as false.
+		 */
+		ret = vfio_pci_core_do_io_rw(&nvdev->core_device,

false,

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+					     nvdev->resmem.ioaddr,
+					     (char __user *)buf,

pos, mem_count,

drivers/vfio/pci/nvgrace-gpu/main.c new file +8 -0
+					     0, 0, true);
+	}
+
+	return ret;
+}
+
+/*
+ * Write count bytes to the device memory at a given offset. The

actual device
+ * memory size (available) may not be a power-of-2. So the driver fakes the
+ * size to a power-of-2 (reported) when exposing to a user space driver.
+ *
+ * Writes extending beyond the reported size are truncated; writes starting
+ * beyond the reported size generate -EINVAL.
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+static ssize_t
+nvgrace_gpu_write_mem(struct nvgrace_gpu_vfio_pci_core_device *nvdev,
+		      size_t count, loff_t *ppos, const char __user

*buf) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +18 -0
+	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct mem_region *memregion;
+	size_t mem_count;
+	int ret = 0;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return -EINVAL;
+
+	if (offset >= memregion->bar_size)
+		return -EINVAL;
+
+	/* Clip short the write request beyond reported BAR size */
+	count = min(count, memregion->bar_size - (size_t)offset);
+
+	/*
+	 * Determine how many bytes to be actually written to the

device memory.

drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+	 * Do not write to the offset beyond available size.
+	 */
+	if (offset >= memregion->memlength)
+		goto exitfn;
+
+	/*
+	 * Only the device memory present on the hardware is mapped,

which may

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * not be power-of-2 aligned. Drop access outside the

available device

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+	 * memory on the hardware.
+	 */
+	mem_count = min(count, memregion->memlength -

(size_t)offset); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +16 -0
+	ret = nvgrace_gpu_map_and_write(nvdev, buf, mem_count, ppos);
+	if (ret)
+		return ret;
+
+exitfn:
+	*ppos += count;
+	return count;
+}
+
+static ssize_t
+nvgrace_gpu_write(struct vfio_device *core_vdev,
+		  const char __user *buf, size_t count, loff_t *ppos)
+{
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+			     core_device.vdev);
+
+	if (nvgrace_gpu_memregion(index, nvdev))
+		return nvgrace_gpu_write_mem(nvdev, count, ppos,

buf); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	if (index == VFIO_PCI_CONFIG_REGION_INDEX)
+		return nvgrace_gpu_write_config_emu(core_vdev, buf,

count, ppos); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +42 -0
+	return vfio_pci_core_write(core_vdev, buf, count, ppos);
+}
+
+static const struct vfio_device_ops nvgrace_gpu_vfio_pci_ops = {
+	.name		= "nvgrace-gpu-vfio-pci",
+	.init		= vfio_pci_core_init_dev,
+	.release	= vfio_pci_core_release_dev,
+	.open_device	= nvgrace_gpu_open_device,
+	.close_device	= nvgrace_gpu_close_device,
+	.ioctl		= nvgrace_gpu_ioctl,
+	.read		= nvgrace_gpu_read,
+	.write		= nvgrace_gpu_write,
+	.mmap		= nvgrace_gpu_mmap,
+	.request	= vfio_pci_core_request,
+	.match		= vfio_pci_core_match,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
+	.detach_ioas	= vfio_iommufd_physical_detach_ioas,
+};
+
+static const struct vfio_device_ops nvgrace_gpu_vfio_pci_core_ops = {
+	.name		= "nvgrace-gpu-vfio-pci-core",
+	.init		= vfio_pci_core_init_dev,
+	.release	= vfio_pci_core_release_dev,
+	.open_device	= nvgrace_gpu_open_device,
+	.close_device	= vfio_pci_core_close_device,
+	.ioctl		= vfio_pci_core_ioctl,
+	.device_feature	= vfio_pci_core_ioctl_feature,
+	.read		= vfio_pci_core_read,
+	.write		= vfio_pci_core_write,
+	.mmap		= vfio_pci_core_mmap,
+	.request	= vfio_pci_core_request,
+	.match		= vfio_pci_core_match,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
+	.detach_ioas	= vfio_iommufd_physical_detach_ioas,
+};
+
+static struct
+nvgrace_gpu_vfio_pci_core_device *nvgrace_gpu_drvdata(struct pci_dev

*pdev) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	struct vfio_pci_core_device *core_device =

dev_get_drvdata(&pdev->dev); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	return container_of(core_device, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +11 -0
+			    core_device);
+}
+
+static int
+nvgrace_gpu_fetch_memory_property(struct pci_dev *pdev,
+				  u64 *pmemphys, u64 *pmemlength)
+{
+	int ret;
+
+	/*
+	 * The memory information is present in the system ACPI

tables as DSD

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+	 * properties nvidia,gpu-mem-base-pa and nvidia,gpu-mem-size.
+	 */
+	ret = device_property_read_u64(&pdev->dev,

"nvidia,gpu-mem-base-pa",

drivers/vfio/pci/nvgrace-gpu/main.c new file +8 -0
+				       pmemphys);
+	if (ret)
+		return ret;
+
+	if (*pmemphys > type_max(phys_addr_t))
+		return -EOVERFLOW;
+
+	ret = device_property_read_u64(&pdev->dev,

"nvidia,gpu-mem-size",

drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+				       pmemlength);
+	if (ret)
+		return ret;
+
+	if (*pmemlength > type_max(size_t))
+		return -EOVERFLOW;
+
+	/*
+	 * If the C2C link is not up due to an error, the coherent

device

drivers/vfio/pci/nvgrace-gpu/main.c new file +11 -0
+	 * memory size is returned as 0. Fail in such case.
+	 */
+	if (*pmemlength == 0)
+		return -ENOMEM;
+
+	return ret;
+}
+
+static int
+nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
+			      struct

nvgrace_gpu_vfio_pci_core_device *nvdev,

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+			      u64 memphys, u64 memlength)
+{
+	int ret = 0;
+
+	/*
+	 * The VM GPU device driver needs a non-cacheable region to

support

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * the MIG feature. Since the device memory is mapped as

NORMAL cached,

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	 * carve out a region from the end with a different NORMAL_NC
+	 * property (called as reserved memory and represented as

resmem). This

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * region then is exposed as a 64b BAR (region 2 and 3) to

the VM, while

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * exposing the rest (termed as usable memory and

represented using usemem)

drivers/vfio/pci/nvgrace-gpu/main.c new file +11 -0
+	 * as cacheable 64b BAR (region 4 and 5).
+	 *
+	 *               devmem (memlength)
+	 * |-------------------------------------------------|
+	 * |                                           |
+	 * usemem.phys/memphys                         resmem.phys
+	 */
+	nvdev->usemem.memphys = memphys;
+
+	/*
+	 * The device memory exposed to the VM is added to the

kernel by the

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * VM driver module in chunks of memory block size. Only the

usable

drivers/vfio/pci/nvgrace-gpu/main.c new file +24 -0
+	 * memory (usemem) is added to the kernel for usage by the VM
+	 * workloads. Make the usable memory size memblock aligned.
+	 */
+	if (check_sub_overflow(memlength, RESMEM_SIZE,
+			       &nvdev->usemem.memlength)) {
+		ret = -EOVERFLOW;
+		goto done;
+	}
+	nvdev->usemem.memlength = round_down(nvdev->usemem.memlength,
+					     MEMBLK_SIZE);
+	if ((check_add_overflow(nvdev->usemem.memphys,
+				nvdev->usemem.memlength,
+				&nvdev->resmem.memphys)) ||
+	    (check_sub_overflow(memlength, nvdev->usemem.memlength,
+				&nvdev->resmem.memlength))) {
+		ret = -EOVERFLOW;
+		goto done;
+	}
+
+	/*
+	 * The memory regions are exposed as BARs. Calculate and save
+	 * the BAR size for them.
+	 */
+	nvdev->usemem.bar_size =

roundup_pow_of_two(nvdev->usemem.memlength);

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	nvdev->resmem.bar_size =

roundup_pow_of_two(nvdev->resmem.memlength); +done:

drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+	return ret;
+}
+
+static int nvgrace_gpu_probe(struct pci_dev *pdev,
+			     const struct pci_device_id *id)
+{
+	const struct vfio_device_ops *ops =

&nvgrace_gpu_vfio_pci_core_ops;

drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev;
+	u64 memphys, memlength;
+	int ret;
+
+	ret = nvgrace_gpu_fetch_memory_property(pdev, &memphys,

&memlength);

drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+	if (!ret)
+		ops = &nvgrace_gpu_vfio_pci_ops;
+
+	nvdev = vfio_alloc_device(nvgrace_gpu_vfio_pci_core_device,

core_device.vdev,

drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+				  &pdev->dev, ops);
+	if (IS_ERR(nvdev))
+		return PTR_ERR(nvdev);
+
+	dev_set_drvdata(&pdev->dev, &nvdev->core_device);
+
+	if (ops == &nvgrace_gpu_vfio_pci_ops) {
+		/*
+		 * Device memory properties are identified in the

host ACPI

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * table. Set the nvgrace_gpu_vfio_pci_core_device

structure.

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+		 */
+		ret = nvgrace_gpu_init_nvdev_struct(pdev, nvdev,
+						    memphys,

memlength);

drivers/vfio/pci/nvgrace-gpu/main.c new file +18 -0
+		if (ret)
+			goto out_put_vdev;
+	}
+
+	ret = vfio_pci_core_register_device(&nvdev->core_device);
+	if (ret)
+		goto out_put_vdev;
+
+	return ret;
+
+out_put_vdev:
+	vfio_put_device(&nvdev->core_device.vdev);
+	return ret;
+}
+
+static void nvgrace_gpu_remove(struct pci_dev *pdev)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =

nvgrace_gpu_drvdata(pdev);

drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+	struct vfio_pci_core_device *vdev = &nvdev->core_device;
+
+	vfio_pci_core_unregister_device(vdev);
+	vfio_put_device(&vdev->vdev);
+}
+
+static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = {
+	/* GH200 120GB */
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA,

0x2342) },

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	/* GH200 480GB */
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA,

0x2345) },

drivers/vfio/pci/nvgrace-gpu/main.c new file +20 -0
+	{}
+};
+
+MODULE_DEVICE_TABLE(pci, nvgrace_gpu_vfio_pci_table);
+
+static struct pci_driver nvgrace_gpu_vfio_pci_driver = {
+	.name = KBUILD_MODNAME,
+	.id_table = nvgrace_gpu_vfio_pci_table,
+	.probe = nvgrace_gpu_probe,
+	.remove = nvgrace_gpu_remove,
+	.err_handler = &vfio_pci_core_err_handlers,
+	.driver_managed_dma = true,
+};
+
+module_pci_driver(nvgrace_gpu_vfio_pci_driver);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Ankit Agrawal <[email protected]>");
+MODULE_AUTHOR("Aniket Agashe <[email protected]>");
+MODULE_DESCRIPTION("VFIO NVGRACE GPU PF - User Level driver for

NVIDIA devices with CPU coherently accessible device memory");

#8 lore raw
Quoted 210 lines · On Thu, 8 Feb 2024 00:32:10 +0200 Zhi Wang <[email protected]> wrote:
Quoted 207 lines · On Tue, 6 Feb 2024 04:31:23 +0530 <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device for the on-chip GPU that is the logical OS representation of the internal proprietary chip-to-chip cache coherent interconnect.

The device is peculiar compared to a real PCI device in that whilst there is a real 64b PCI BAR1 (comprising region 2 & region 3) on the device, it is not used to access device memory once the faster chip-to-chip interconnect is initialized (occurs at the time of host system boot). The device memory is accessed instead using the chip-to-chip interconnect that is exposed as a contiguous physically addressable region on the host. This device memory aperture can be obtained from host ACPI table using device_property_read_u64(), according to the FW specification. Since the device memory is cache coherent with the CPU, it can be mmap into the user VMA with a cacheable mapping using remap_pfn_range() and used like a regular RAM. The device memory is not added to the host kernel, but mapped directly as this reduces memory wastage due to struct pages.

There is also a requirement of a reserved 1G uncached region (termed as resmem) to support the Multi-Instance GPU (MIG) feature [1]. This is to work around a HW defect. Based on [2], the requisite properties (uncached, unaligned access) can be achieved through a VM mapping (S1) of NORMAL_NC and host (S2) mapping with MemAttr[2:0]=0b101. To provide a different non-cached property to the reserved 1G region, it needs to be carved out from the device memory and mapped as a separate region in Qemu VMA with pgprot_writecombine(). pgprot_writecombine() sets the Qemu VMA page properties (pgprot) as NORMAL_NC.

Provide a VFIO PCI variant driver that adapts the unique device memory representation into a more standard PCI representation facing userspace.

The variant driver exposes these two regions - the non-cached reserved (resmem) and the cached rest of the device memory (termed as usemem) as separate VFIO 64b BAR regions. This is divergent from the baremetal approach, where the device memory is exposed as a device memory region. The decision for a different approach was taken in view of the fact that it would necessiate additional code in Qemu to discover and insert those regions in the VM IPA, along with the additional VM ACPI DSDT changes to communicate the device memory region IPA to the VM workloads. Moreover, this behavior would have to be added to a variety of emulators (beyond top of tree Qemu) out there desiring grace hopper support.

Since the device implements 64-bit BAR0, the VFIO PCI variant driver maps the uncached carved out region to the next available PCI BAR (i.e. comprising of region 2 and 3). The cached device memory aperture is assigned BAR region 4 and 5. Qemu will then naturally generate a PCI device in the VM with the uncached aperture reported as BAR2 region, the cacheable as BAR4. The variant driver provides emulation for these fake BARs' PCI config space offset registers.

The hardware ensures that the system does not crash when the memory is accessed with the memory enable turned off. It synthesis ~0 reads and dropped writes on such access. So there is no need to support the disablement/enablement of BAR through PCI_COMMAND config space register.

The memory layout on the host looks like the following:
devmem (memlength)

|--------------------------------------------------|
|-------------cached------------------------|--NC--|
|                                           |
usemem.phys/memphys                         resmem.phys

PCI BARs need to be aligned to the power-of-2, but the actual memory on the device may not. A read or write access to the physical address from the last device PFN up to the next power-of-2 aligned physical address results in reading ~0 and dropped writes. Note that the GPU device driver [6] is capable of knowing the exact device memory size through separate means. The device memory size is primarily kept in the system ACPI tables for use by the VFIO PCI variant module.

Note that the usemem memory is added by the VM Nvidia device driver
[5] to the VM kernel as memblocks. Hence make the usable memory size memblock aligned.

Currently there is no provision in KVM for a S2 mapping with MemAttr[2:0]=0b101, but there is an ongoing effort to provide the same [3]. As previously mentioned, resmem is mapped pgprot_writecombine(), that sets the Qemu VMA page properties (pgprot) as NORMAL_NC. Using the proposed changes in [4] and [3], KVM marks the region with MemAttr[2:0]=0b101 in S2.

If the bare metal properties are not present, the driver registers the vfio-pci-core function pointers.

This goes along with a qemu series [6] to provides the necessary implementation of the Grace Hopper Superchip firmware specification so that the guest operating system can see the correct ACPI modeling for the coherent GPU device. Verified with the CUDA workload in the VM.

[1] https://www.nvidia.com/en-in/technologies/multi-instance-gpu/
[2] section D8.5.5 of
https://developer.arm.com/documentation/ddi0487/latest/ [3]
https://lore.kernel.org/all/[email protected]/
[4]
https://lore.kernel.org/all/[email protected]/
[5] https://github.com/NVIDIA/open-gpu-kernel-modules [6]
https://lore.kernel.org/all/[email protected]/

---
 MAINTAINERS                           |   6 +
 drivers/vfio/pci/Kconfig              |   2 +
 drivers/vfio/pci/Makefile             |   2 +
 drivers/vfio/pci/nvgrace-gpu/Kconfig  |  10 +
 drivers/vfio/pci/nvgrace-gpu/Makefile |   3 +
 drivers/vfio/pci/nvgrace-gpu/main.c   | 856
++++++++++++++++++++++++++ 6 files changed, 879 insertions(+)
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/Kconfig
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/Makefile
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/main.c
MAINTAINERS +6 -0
index 8999497011a2..529ec8966f58 100644
@@ -23103,6 +23103,12 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/vfio/platform/
 
+VFIO NVIDIA GRACE GPU DRIVER
+M:	Ankit Agrawal <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	drivers/vfio/pci/nvgrace-gpu/
+
 VGA_SWITCHEROO
 R:	Lukas Wunner <[email protected]>
 S:	Maintained
drivers/vfio/pci/Kconfig +2 -0
index 18c397df566d..15821a2d77d2 100644
@@ -67,4 +67,6 @@ source "drivers/vfio/pci/pds/Kconfig"
 
 source "drivers/vfio/pci/virtio/Kconfig"
 
+source "drivers/vfio/pci/nvgrace-gpu/Kconfig"
+
 endmenu
drivers/vfio/pci/Makefile +2 -0
index 046139a4eca5..ce7a61f1d912 100644
@@ -15,3 +15,5 @@ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/
 obj-$(CONFIG_PDS_VFIO_PCI) += pds/
 
 obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/
+
+obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/
(quoted diff) +0 -0

b/drivers/vfio/pci/nvgrace-gpu/Kconfig new file mode 100644 index 000000000000..936e88d8d41d

drivers/vfio/pci/nvgrace-gpu/Kconfig new file +3 -0
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config NVGRACE_GPU_VFIO_PCI
+	tristate "VFIO support for the GPU in the NVIDIA Grace

Hopper Superchip"

drivers/vfio/pci/nvgrace-gpu/Kconfig new file +4 -0
+	depends on ARM64 || (COMPILE_TEST && 64BIT)
+	select VFIO_PCI_CORE
+	help
+	  VFIO support for the GPU in the NVIDIA Grace Hopper

Superchip is

drivers/vfio/pci/nvgrace-gpu/Kconfig new file +3 -0
+	  required to assign the GPU device using KVM/qemu/etc.
+
+	  If you don't know what to do here, say N.
(quoted diff) +0 -0

b/drivers/vfio/pci/nvgrace-gpu/Makefile new file mode 100644 index 000000000000..3ca8c187897a

drivers/vfio/pci/nvgrace-gpu/Makefile new file +3 -0
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu-vfio-pci.o
+nvgrace-gpu-vfio-pci-y := main.o
(quoted diff) +0 -0

b/drivers/vfio/pci/nvgrace-gpu/main.c new file mode 100644 index 000000000000..6279af2bc6b8

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
@@ -0,0 +1,856 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights

reserved
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+
+#include <linux/vfio_pci_core.h>
+
+/*
+ * The device memory usable to the workloads running in the VM is

cached
+ * and showcased as a 64b device BAR (comprising of BAR4 and BAR5 region)
+ * to the VM and is represented as usemem.
+ * Moreover, the VM GPU device driver needs a non-cacheable region to
+ * support the MIG feature. This region is also exposed as a 64b BAR
+ * (comprising of BAR2 and BAR3 region) and represented as resmem.
+ */
+#define RESMEM_REGION_INDEX VFIO_PCI_BAR2_REGION_INDEX

Quoted 8 lines

drivers/vfio/pci/nvgrace-gpu/main.c

Quoted 6 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+#define USEMEM_REGION_INDEX VFIO_PCI_BAR4_REGION_INDEX
+
+/* Memory size expected as non cached and reserved by the VM driver

*/ +#define RESMEM_SIZE 0x40000000

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+#define MEMBLK_SIZE 0x20000000
+  

Maybe use SZ_* definitions in linux/size.h

Good suggestion.

Quoted 78 lines
Quoted 61 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+/*
+ * The state of the two device memory region - resmem and usemem - is
+ * saved as struct mem_region.
+ */
+struct mem_region {
+	phys_addr_t memphys;    /* Base physical address of the

region */

drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+	size_t memlength;       /* Region size */
+	size_t bar_size;        /* Reported region BAR size */
+	__le64 bar_val;         /* Emulated BAR offset registers */
+	union {
+		void *memaddr;
+		void __iomem *ioaddr;
+	};                      /* Base virtual address of the

region */ +};

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+
+struct nvgrace_gpu_vfio_pci_core_device {
+	struct vfio_pci_core_device core_device;
+	/* Cached and usable memory for the VM. */
+	struct mem_region usemem;
+	/* Non cached memory carved out from the end of device

memory */

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	struct mem_region resmem;
+	/* Lock to control device memory kernel mapping */
+	struct mutex remap_lock;
+};
+
+static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device

*core_vdev) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+			     core_device.vdev);
+
+	nvdev->resmem.bar_val = 0;
+	nvdev->usemem.bar_val = 0;
+}
+
+/* Choose the structure corresponding to the fake BAR with a given

index. */ +static struct mem_region *

drivers/vfio/pci/nvgrace-gpu/main.c new file +16 -0
+nvgrace_gpu_memregion(int index,
+		      struct nvgrace_gpu_vfio_pci_core_device *nvdev)
+{
+	if (index == USEMEM_REGION_INDEX)
+		return &nvdev->usemem;
+
+	if (index == RESMEM_REGION_INDEX)
+		return &nvdev->resmem;
+
+	return NULL;
+}
+
+static int nvgrace_gpu_open_device(struct vfio_device *core_vdev)
+{
+	struct vfio_pci_core_device *vdev =
+		container_of(core_vdev, struct vfio_pci_core_device,

vdev);

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c line 86

Quoted 9 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+			     core_device.vdev);
+	int ret;
+
+	ret = vfio_pci_core_enable(vdev);
+	if (ret)
+		return ret;
+
+	vfio_pci_core_finish_enable(vdev);
+  

---

drivers/vfio/pci/nvgrace-gpu/main.c line 91

Quoted 5 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+	if (nvdev->usemem.memlength) {
+		nvgrace_gpu_init_fake_bar_emu_regs(core_vdev);
+		mutex_init(&nvdev->remap_lock);
+	}
+  

---

Quoted 7 lines

Better move this part to the place between vfio_pci_core_enable() and vfio_pci_core_finish_enable() like others for respecting the expected device initialization sequence of life cycle.

It doesn't bite something right now, but think about when someone changes the behavior of vfio_pci_core_finish_enable() in the future, they have to propose a patch for this.

Agree.

Quoted 8 lines
Quoted 8 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+	return 0;
+}
+
+static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

Quoted 8 lines

drivers/vfio/pci/nvgrace-gpu/main.c

Quoted 2 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+			     core_device.vdev);
+  

Shouldn't there be a lock/unlock sequence when using/freeing the map? As the vfio device open/close paths are protected by group/dev_set lock, probably they are fine without driver-level lock protection. But for vfio_device_read/write/ioctl, if they are racing with open/close path, I think there should be a lock/unlock to protect the map.

This should only be called when the vfio device files is no longer opened and therefore cannot have a race with other userspace access.

Quoted 383 lines

---

drivers/vfio/pci/nvgrace-gpu/main.c line 112

Quoted 13 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+	/* Unmap the mapping to the device memory cached region */
+	if (nvdev->usemem.memaddr) {
+		memunmap(nvdev->usemem.memaddr);
+		nvdev->usemem.memaddr = NULL;
+	}
+
+	/* Unmap the mapping to the device memory non-cached region

*/

drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+	if (nvdev->resmem.ioaddr) {
+		iounmap(nvdev->resmem.ioaddr);
+		nvdev->resmem.ioaddr = NULL;
+	}
+  

---

Quoted 367 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +10 -0
+	mutex_destroy(&nvdev->remap_lock);
+
+	vfio_pci_core_close_device(core_vdev);
+}
+
+static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
+			    struct vm_area_struct *vma)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+			     core_device.vdev);
+
+	unsigned long start_pfn;
+	unsigned int index;
+	u64 req_len, pgoff, end;
+	int ret = 0;
+	struct mem_region *memregion;
+
+	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT -

PAGE_SHIFT); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return vfio_pci_core_mmap(core_vdev, vma);
+
+	/*
+	 * Request to mmap the BAR. Map to the CPU accessible memory

on the

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * GPU using the memory information gathered from the system

ACPI

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	 * tables.
+	 */
+	pgoff = vma->vm_pgoff &
+		((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
+
+	if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len)

||

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	    check_add_overflow(PHYS_PFN(memregion->memphys), pgoff,

&start_pfn) ||

drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+	    check_add_overflow(PFN_PHYS(pgoff), req_len, &end))
+		return -EOVERFLOW;
+
+	/*
+	 * Check that the mapping request does not go beyond

available device

drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+	 * memory size
+	 */
+	if (end > memregion->memlength)
+		return -EINVAL;
+
+	/*
+	 * The carved out region of the device memory needs the

NORMAL_NC

drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+	 * property. Communicate as such to the hypervisor.
+	 */
+	if (index == RESMEM_REGION_INDEX)
+		vma->vm_page_prot =

pgprot_writecombine(vma->vm_page_prot); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	/*
+	 * Perform a PFN map to the memory and back the device BAR

by the

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+	 * GPU memory.
+	 *
+	 * The available GPU memory size may not be power-of-2

aligned. The

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * remainder is only backed by vfio_device_ops read/write

handlers.

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	 *
+	 * During device reset, the GPU is safely disconnected to

the CPU

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * and access to the BAR will be immediately returned

preventing

drivers/vfio/pci/nvgrace-gpu/main.c new file +17 -0
+	 * machine check.
+	 */
+	ret = remap_pfn_range(vma, vma->vm_start, start_pfn,
+			      req_len, vma->vm_page_prot);
+	if (ret)
+		return ret;
+
+	vma->vm_pgoff = start_pfn;
+
+	return 0;
+}
+
+static long
+nvgrace_gpu_ioctl_get_region_info(struct vfio_device *core_vdev,
+				  unsigned long arg)
+{
+	unsigned long minsz = offsetofend(struct vfio_region_info,

offset);

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +18 -0
+			     core_device.vdev);
+	struct vfio_region_info_cap_sparse_mmap *sparse;
+	struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
+	struct vfio_region_info info;
+	struct mem_region *memregion;
+	u32 size;
+	int ret;
+
+	if (copy_from_user(&info, (void __user *)arg, minsz))
+		return -EFAULT;
+
+	if (info.argsz < minsz)
+		return -EINVAL;
+
+	memregion = nvgrace_gpu_memregion(info.index, nvdev);
+	if (!memregion)
+		return vfio_pci_core_ioctl(core_vdev,
+

VFIO_DEVICE_GET_REGION_INFO, arg); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +64 -0
+	/*
+	 * Request to determine the BAR region information. Send the
+	 * GPU memory information.
+	 */
+	size = struct_size(sparse, areas, 1);
+
+	/*
+	 * Setup for sparse mapping for the device memory. Only the
+	 * available device memory on the hardware is shown as a
+	 * mappable region.
+	 */
+	sparse = kzalloc(size, GFP_KERNEL);
+	if (!sparse)
+		return -ENOMEM;
+
+	sparse->nr_areas = 1;
+	sparse->areas[0].offset = 0;
+	sparse->areas[0].size = memregion->memlength;
+	sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
+	sparse->header.version = 1;
+
+	ret = vfio_info_add_capability(&caps, &sparse->header, size);
+	kfree(sparse);
+	if (ret)
+		return ret;
+
+	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
+	/*
+	 * The region memory size may not be power-of-2 aligned.
+	 * Given that the memory  as a BAR and may not be
+	 * aligned, roundup to the next power-of-2.
+	 */
+	info.size = memregion->bar_size;
+	info.flags = VFIO_REGION_INFO_FLAG_READ |
+		     VFIO_REGION_INFO_FLAG_WRITE |
+		     VFIO_REGION_INFO_FLAG_MMAP;
+
+	if (caps.size) {
+		info.flags |= VFIO_REGION_INFO_FLAG_CAPS;
+		if (info.argsz < sizeof(info) + caps.size) {
+			info.argsz = sizeof(info) + caps.size;
+			info.cap_offset = 0;
+		} else {
+			vfio_info_cap_shift(&caps, sizeof(info));
+			if (copy_to_user((void __user *)arg +
+					 sizeof(info), caps.buf,
+					 caps.size)) {
+				kfree(caps.buf);
+				return -EFAULT;
+			}
+			info.cap_offset = sizeof(info);
+		}
+		kfree(caps.buf);
+	}
+	return copy_to_user((void __user *)arg, &info, minsz) ?
+			    -EFAULT : 0;
+}
+
+static long nvgrace_gpu_ioctl(struct vfio_device *core_vdev,
+			      unsigned int cmd, unsigned long arg)
+{
+	switch (cmd) {
+	case VFIO_DEVICE_GET_REGION_INFO:
+		return nvgrace_gpu_ioctl_get_region_info(core_vdev,

arg);

drivers/vfio/pci/nvgrace-gpu/main.c new file +24 -0
+	case VFIO_DEVICE_IOEVENTFD:
+		return -ENOTTY;
+	case VFIO_DEVICE_RESET:
+		nvgrace_gpu_init_fake_bar_emu_regs(core_vdev);
+		fallthrough;
+	default:
+		return vfio_pci_core_ioctl(core_vdev, cmd, arg);
+	}
+}
+
+static __le64
+nvgrace_gpu_get_read_value(size_t bar_size, u64 flags, __le64 val64)
+{
+	u64 tmp_val;
+
+	tmp_val = le64_to_cpu(val64);
+	tmp_val &= ~(bar_size - 1);
+	tmp_val |= flags;
+
+	return cpu_to_le64(tmp_val);
+}
+
+/*
+ * Both the usable (usemem) and the reserved (resmem) device memory

region
+ * are exposed as a 64b fake BARs in the VM. These fake BARs must respond
+ * to the accesses on their respective PCI config space offsets.
+ *
+ * resmem BAR owns PCI_BASE_ADDRESS_2 & PCI_BASE_ADDRESS_3.
+ * usemem BAR owns PCI_BASE_ADDRESS_4 & PCI_BASE_ADDRESS_5.
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+static ssize_t
+nvgrace_gpu_read_config_emu(struct vfio_device *core_vdev,
+			    char __user *buf, size_t count, loff_t

*ppos) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +14 -0
+			     core_device.vdev);
+	struct mem_region *memregion = NULL;
+	u64 pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	__le64 val64;
+	size_t register_offset;
+	loff_t copy_offset;
+	size_t copy_count;
+	int ret;
+
+	ret = vfio_pci_core_read(core_vdev, buf, count, ppos);
+	if (ret < 0)
+		return ret;
+
+	if (vfio_pci_core_range_intersect_range(pos, count,

PCI_BASE_ADDRESS_2,

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+						sizeof(val64),
+						&copy_offset,

&copy_count,

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+						&register_offset))
+		memregion =

nvgrace_gpu_memregion(RESMEM_REGION_INDEX, nvdev);

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	else if (vfio_pci_core_range_intersect_range(pos, count,
+

PCI_BASE_ADDRESS_4,

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+						     sizeof(val64),
+						     &copy_offset,

&copy_count,
+
&register_offset))

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		memregion =

nvgrace_gpu_memregion(USEMEM_REGION_INDEX, nvdev); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	if (memregion) {
+		val64 =

nvgrace_gpu_get_read_value(memregion->bar_size,
+
PCI_BASE_ADDRESS_MEM_TYPE_64 |
+
PCI_BASE_ADDRESS_MEM_PREFETCH,
+
memregion->bar_val);

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+		if (copy_to_user(buf + copy_offset,
+				 (void *)&val64 + register_offset,

copy_count))

drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+			return -EFAULT;
+	}
+
+	return count;
+}
+
+static ssize_t
+nvgrace_gpu_write_config_emu(struct vfio_device *core_vdev,
+			     const char __user *buf, size_t count,

loff_t *ppos) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +8 -0
+			     core_device.vdev);
+	u64 pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	size_t register_offset;
+	loff_t copy_offset;
+	size_t copy_count;
+	struct mem_region *memregion = NULL;
+
+	if (vfio_pci_core_range_intersect_range(pos, count,

PCI_BASE_ADDRESS_2,

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+						sizeof(u64),

&copy_offset,

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+						&copy_count,

&register_offset))

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		memregion =

nvgrace_gpu_memregion(RESMEM_REGION_INDEX, nvdev);

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	else if (vfio_pci_core_range_intersect_range(pos, count,

PCI_BASE_ADDRESS_4,

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+						     sizeof(u64),

&copy_offset,

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+						     &copy_count,

&register_offset))

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		memregion =

nvgrace_gpu_memregion(USEMEM_REGION_INDEX, nvdev); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	if (memregion) {
+		if (copy_from_user((void *)&memregion->bar_val +

register_offset,

drivers/vfio/pci/nvgrace-gpu/main.c new file +11 -0
+				   buf + copy_offset, copy_count))
+			return -EFAULT;
+		*ppos += copy_count;
+		return copy_count;
+	}
+
+	return vfio_pci_core_write(core_vdev, buf, count, ppos);
+}
+
+/*
+ * Ad hoc map the device memory in the module kernel VA space.

Primarily needed
+ * as vfio does not require the userspace driver to only perform accesses through
+ * mmaps of the vfio-pci BAR regions and such accesses should be supported using
+ * vfio_device_ops read/write implementations.
+ *
+ * The usemem region is cacheable memory and hence is memremaped.
+ * The resmem region is non-cached and is mapped using ioremap_wc (NORMAL_NC).
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+static int
+nvgrace_gpu_map_device_mem(int index,
+			   struct nvgrace_gpu_vfio_pci_core_device

*nvdev) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +15 -0
+	struct mem_region *memregion;
+	int ret = 0;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return -EINVAL;
+
+	mutex_lock(&nvdev->remap_lock);
+	if (index == USEMEM_REGION_INDEX && !memregion->memaddr) {
+		memregion->memaddr = memremap(memregion->memphys,
+					      memregion->memlength,
+					      MEMREMAP_WB);
+		if (!memregion->memaddr)
+			ret = -ENOMEM;
+	} else if (index == RESMEM_REGION_INDEX &&

!memregion->ioaddr) {

drivers/vfio/pci/nvgrace-gpu/main.c new file +12 -0
+		memregion->ioaddr = ioremap_wc(memregion->memphys,
+					       memregion->memlength);
+		if (!memregion->ioaddr)
+			ret = -ENOMEM;
+	}
+	mutex_unlock(&nvdev->remap_lock);
+
+	return ret;
+}
+
+/*
+ * Read the data from the device memory (mapped either through

ioremap
+ * or memremap) into the user buffer.
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+static int
+nvgrace_gpu_map_and_read(struct nvgrace_gpu_vfio_pci_core_device

*nvdev,

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+			 char __user *buf, size_t mem_count, loff_t

*ppos) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
+	int ret;
+
+	/*
+	 * Handle read on the BAR regions. Map to the target device

memory

Quoted 8 lines

drivers/vfio/pci/nvgrace-gpu/main.c line 439

Quoted 6 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	 * physical address and copy to the request read buffer.
+	 */
+	ret = nvgrace_gpu_map_device_mem(index, nvdev);
+	if (ret)
+		return ret;
+  

Wouldn't it be better to do the map in the open path?

AIUI the device would typically be used exclusively through the mmap of these ranges, these mappings are only for pread/pwrite type accesses, so I think it makes sense to map them on demand.

Quoted 69 lines
Quoted 69 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+	if (index == USEMEM_REGION_INDEX) {
+		if (copy_to_user(buf,
+				 (u8 *)nvdev->usemem.memaddr +

offset,

drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+				 mem_count))
+			ret = -EFAULT;
+	} else {
+		/*
+		 * The hardware ensures that the system does not

crash when

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * the device memory is accessed with the memory

enable

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * turned off. It synthesizes ~0 on such read. So

there is

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * no need to check or support the

disablement/enablement of

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * BAR through PCI_COMMAND config space register.

Pass

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+		 * test_mem flag as false.
+		 */
+		ret = vfio_pci_core_do_io_rw(&nvdev->core_device,

false,

drivers/vfio/pci/nvgrace-gpu/main.c new file +10 -0
+					     nvdev->resmem.ioaddr,
+					     buf, offset, mem_count,
+					     0, 0, false);
+	}
+
+	return ret;
+}
+
+/*
+ * Read count bytes from the device memory at an offset. The actual

device
+ * memory size (available) may not be a power-of-2. So the driver fakes
+ * the size to a power-of-2 (reported) when exposing to a user space driver.
+ *
+ * Reads extending beyond the reported size are truncated; reads starting
+ * beyond the reported size generate -EINVAL; reads extending beyond the
+ * actual device size is filled with ~0.
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +23 -0
+static ssize_t
+nvgrace_gpu_read_mem(struct nvgrace_gpu_vfio_pci_core_device *nvdev,
+		     char __user *buf, size_t count, loff_t *ppos)
+{
+	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct mem_region *memregion;
+	size_t mem_count, i;
+	u8 val = 0xFF;
+	int ret;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return -EINVAL;
+
+	if (offset >= memregion->bar_size)
+		return -EINVAL;
+
+	/* Clip short the read request beyond reported BAR size */
+	count = min(count, memregion->bar_size - (size_t)offset);
+
+	/*
+	 * Determine how many bytes to be actually read from the

device memory.

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * Read request beyond the actual device memory size is
Quoted 8 lines

drivers/vfio/pci/nvgrace-gpu/main.c line 499

Quoted 5 lines

filled with ~0,

drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+	 * while those beyond the actual reported size is skipped.
+	 */
+	if (offset >= memregion->memlength)
+		mem_count = 0;  

If mem_count == 0, going through nvgrace_gpu_map_and_read() is not necessary.

Harmless, other than the possibly unnecessary call through to nvgrace_gpu_map_device_mem(). Maybe both nvgrace_gpu_map_and_read() and nvgrace_gpu_map_and_write() could conditionally return 0 as their first operation when !mem_count. Thanks,

Alex

Quoted 400 lines
Quoted 399 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	else
+		mem_count = min(count, memregion->memlength -

(size_t)offset); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	ret = nvgrace_gpu_map_and_read(nvdev, buf, mem_count, ppos);
+	if (ret)
+		return ret;
+
+	/*
+	 * Only the device memory present on the hardware is mapped,

which may

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * not be power-of-2 aligned. A read to an offset beyond the

device memory

drivers/vfio/pci/nvgrace-gpu/main.c new file +16 -0
+	 * size is filled with ~0.
+	 */
+	for (i = mem_count; i < count; i++)
+		put_user(val, (unsigned char __user *)(buf + i));
+
+	*ppos += count;
+	return count;
+}
+
+static ssize_t
+nvgrace_gpu_read(struct vfio_device *core_vdev,
+		 char __user *buf, size_t count, loff_t *ppos)
+{
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+			     core_device.vdev);
+
+	if (nvgrace_gpu_memregion(index, nvdev))
+		return nvgrace_gpu_read_mem(nvdev, buf, count, ppos);
+
+	if (index == VFIO_PCI_CONFIG_REGION_INDEX)
+		return nvgrace_gpu_read_config_emu(core_vdev, buf,

count, ppos); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+	return vfio_pci_core_read(core_vdev, buf, count, ppos);
+}
+
+/*
+ * Write the data to the device memory (mapped either through ioremap
+ * or memremap) from the user buffer.
+ */
+static int
+nvgrace_gpu_map_and_write(struct nvgrace_gpu_vfio_pci_core_device

*nvdev,

drivers/vfio/pci/nvgrace-gpu/main.c new file +18 -0
+			  const char __user *buf, size_t mem_count,
+			  loff_t *ppos)
+{
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	int ret;
+
+	ret = nvgrace_gpu_map_device_mem(index, nvdev);
+	if (ret)
+		return ret;
+
+	if (index == USEMEM_REGION_INDEX) {
+		if (copy_from_user((u8 *)nvdev->usemem.memaddr + pos,
+				   buf, mem_count))
+			return -EFAULT;
+	} else {
+		/*
+		 * The hardware ensures that the system does not

crash when

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * the device memory is accessed with the memory

enable

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * turned off. It drops such writes. So there is no

need to

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+		 * check or support the disablement/enablement of BAR
+		 * through PCI_COMMAND config space register. Pass

test_mem

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+		 * flag as false.
+		 */
+		ret = vfio_pci_core_do_io_rw(&nvdev->core_device,

false,

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+					     nvdev->resmem.ioaddr,
+					     (char __user *)buf,

pos, mem_count,

drivers/vfio/pci/nvgrace-gpu/main.c new file +8 -0
+					     0, 0, true);
+	}
+
+	return ret;
+}
+
+/*
+ * Write count bytes to the device memory at a given offset. The

actual device
+ * memory size (available) may not be a power-of-2. So the driver fakes the
+ * size to a power-of-2 (reported) when exposing to a user space driver.
+ *
+ * Writes extending beyond the reported size are truncated; writes starting
+ * beyond the reported size generate -EINVAL.
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+static ssize_t
+nvgrace_gpu_write_mem(struct nvgrace_gpu_vfio_pci_core_device *nvdev,
+		      size_t count, loff_t *ppos, const char __user

*buf) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +18 -0
+	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct mem_region *memregion;
+	size_t mem_count;
+	int ret = 0;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return -EINVAL;
+
+	if (offset >= memregion->bar_size)
+		return -EINVAL;
+
+	/* Clip short the write request beyond reported BAR size */
+	count = min(count, memregion->bar_size - (size_t)offset);
+
+	/*
+	 * Determine how many bytes to be actually written to the

device memory.

drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+	 * Do not write to the offset beyond available size.
+	 */
+	if (offset >= memregion->memlength)
+		goto exitfn;
+
+	/*
+	 * Only the device memory present on the hardware is mapped,

which may

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * not be power-of-2 aligned. Drop access outside the

available device

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+	 * memory on the hardware.
+	 */
+	mem_count = min(count, memregion->memlength -

(size_t)offset); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +16 -0
+	ret = nvgrace_gpu_map_and_write(nvdev, buf, mem_count, ppos);
+	if (ret)
+		return ret;
+
+exitfn:
+	*ppos += count;
+	return count;
+}
+
+static ssize_t
+nvgrace_gpu_write(struct vfio_device *core_vdev,
+		  const char __user *buf, size_t count, loff_t *ppos)
+{
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+			     core_device.vdev);
+
+	if (nvgrace_gpu_memregion(index, nvdev))
+		return nvgrace_gpu_write_mem(nvdev, count, ppos,

buf); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	if (index == VFIO_PCI_CONFIG_REGION_INDEX)
+		return nvgrace_gpu_write_config_emu(core_vdev, buf,

count, ppos); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +42 -0
+	return vfio_pci_core_write(core_vdev, buf, count, ppos);
+}
+
+static const struct vfio_device_ops nvgrace_gpu_vfio_pci_ops = {
+	.name		= "nvgrace-gpu-vfio-pci",
+	.init		= vfio_pci_core_init_dev,
+	.release	= vfio_pci_core_release_dev,
+	.open_device	= nvgrace_gpu_open_device,
+	.close_device	= nvgrace_gpu_close_device,
+	.ioctl		= nvgrace_gpu_ioctl,
+	.read		= nvgrace_gpu_read,
+	.write		= nvgrace_gpu_write,
+	.mmap		= nvgrace_gpu_mmap,
+	.request	= vfio_pci_core_request,
+	.match		= vfio_pci_core_match,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
+	.detach_ioas	= vfio_iommufd_physical_detach_ioas,
+};
+
+static const struct vfio_device_ops nvgrace_gpu_vfio_pci_core_ops = {
+	.name		= "nvgrace-gpu-vfio-pci-core",
+	.init		= vfio_pci_core_init_dev,
+	.release	= vfio_pci_core_release_dev,
+	.open_device	= nvgrace_gpu_open_device,
+	.close_device	= vfio_pci_core_close_device,
+	.ioctl		= vfio_pci_core_ioctl,
+	.device_feature	= vfio_pci_core_ioctl_feature,
+	.read		= vfio_pci_core_read,
+	.write		= vfio_pci_core_write,
+	.mmap		= vfio_pci_core_mmap,
+	.request	= vfio_pci_core_request,
+	.match		= vfio_pci_core_match,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
+	.detach_ioas	= vfio_iommufd_physical_detach_ioas,
+};
+
+static struct
+nvgrace_gpu_vfio_pci_core_device *nvgrace_gpu_drvdata(struct pci_dev

*pdev) +{

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	struct vfio_pci_core_device *core_device =

dev_get_drvdata(&pdev->dev); +

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	return container_of(core_device, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +11 -0
+			    core_device);
+}
+
+static int
+nvgrace_gpu_fetch_memory_property(struct pci_dev *pdev,
+				  u64 *pmemphys, u64 *pmemlength)
+{
+	int ret;
+
+	/*
+	 * The memory information is present in the system ACPI

tables as DSD

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+	 * properties nvidia,gpu-mem-base-pa and nvidia,gpu-mem-size.
+	 */
+	ret = device_property_read_u64(&pdev->dev,

"nvidia,gpu-mem-base-pa",

drivers/vfio/pci/nvgrace-gpu/main.c new file +8 -0
+				       pmemphys);
+	if (ret)
+		return ret;
+
+	if (*pmemphys > type_max(phys_addr_t))
+		return -EOVERFLOW;
+
+	ret = device_property_read_u64(&pdev->dev,

"nvidia,gpu-mem-size",

drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+				       pmemlength);
+	if (ret)
+		return ret;
+
+	if (*pmemlength > type_max(size_t))
+		return -EOVERFLOW;
+
+	/*
+	 * If the C2C link is not up due to an error, the coherent

device

drivers/vfio/pci/nvgrace-gpu/main.c new file +11 -0
+	 * memory size is returned as 0. Fail in such case.
+	 */
+	if (*pmemlength == 0)
+		return -ENOMEM;
+
+	return ret;
+}
+
+static int
+nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
+			      struct

nvgrace_gpu_vfio_pci_core_device *nvdev,

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+			      u64 memphys, u64 memlength)
+{
+	int ret = 0;
+
+	/*
+	 * The VM GPU device driver needs a non-cacheable region to

support

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * the MIG feature. Since the device memory is mapped as

NORMAL cached,

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	 * carve out a region from the end with a different NORMAL_NC
+	 * property (called as reserved memory and represented as

resmem). This

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * region then is exposed as a 64b BAR (region 2 and 3) to

the VM, while

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * exposing the rest (termed as usable memory and

represented using usemem)

drivers/vfio/pci/nvgrace-gpu/main.c new file +11 -0
+	 * as cacheable 64b BAR (region 4 and 5).
+	 *
+	 *               devmem (memlength)
+	 * |-------------------------------------------------|
+	 * |                                           |
+	 * usemem.phys/memphys                         resmem.phys
+	 */
+	nvdev->usemem.memphys = memphys;
+
+	/*
+	 * The device memory exposed to the VM is added to the

kernel by the

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	 * VM driver module in chunks of memory block size. Only the

usable

drivers/vfio/pci/nvgrace-gpu/main.c new file +24 -0
+	 * memory (usemem) is added to the kernel for usage by the VM
+	 * workloads. Make the usable memory size memblock aligned.
+	 */
+	if (check_sub_overflow(memlength, RESMEM_SIZE,
+			       &nvdev->usemem.memlength)) {
+		ret = -EOVERFLOW;
+		goto done;
+	}
+	nvdev->usemem.memlength = round_down(nvdev->usemem.memlength,
+					     MEMBLK_SIZE);
+	if ((check_add_overflow(nvdev->usemem.memphys,
+				nvdev->usemem.memlength,
+				&nvdev->resmem.memphys)) ||
+	    (check_sub_overflow(memlength, nvdev->usemem.memlength,
+				&nvdev->resmem.memlength))) {
+		ret = -EOVERFLOW;
+		goto done;
+	}
+
+	/*
+	 * The memory regions are exposed as BARs. Calculate and save
+	 * the BAR size for them.
+	 */
+	nvdev->usemem.bar_size =

roundup_pow_of_two(nvdev->usemem.memlength);

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+	nvdev->resmem.bar_size =

roundup_pow_of_two(nvdev->resmem.memlength); +done:

drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+	return ret;
+}
+
+static int nvgrace_gpu_probe(struct pci_dev *pdev,
+			     const struct pci_device_id *id)
+{
+	const struct vfio_device_ops *ops =

&nvgrace_gpu_vfio_pci_core_ops;

drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev;
+	u64 memphys, memlength;
+	int ret;
+
+	ret = nvgrace_gpu_fetch_memory_property(pdev, &memphys,

&memlength);

drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+	if (!ret)
+		ops = &nvgrace_gpu_vfio_pci_ops;
+
+	nvdev = vfio_alloc_device(nvgrace_gpu_vfio_pci_core_device,

core_device.vdev,

drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+				  &pdev->dev, ops);
+	if (IS_ERR(nvdev))
+		return PTR_ERR(nvdev);
+
+	dev_set_drvdata(&pdev->dev, &nvdev->core_device);
+
+	if (ops == &nvgrace_gpu_vfio_pci_ops) {
+		/*
+		 * Device memory properties are identified in the

host ACPI

drivers/vfio/pci/nvgrace-gpu/main.c new file +1 -0
+		 * table. Set the nvgrace_gpu_vfio_pci_core_device

structure.

drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+		 */
+		ret = nvgrace_gpu_init_nvdev_struct(pdev, nvdev,
+						    memphys,

memlength);

drivers/vfio/pci/nvgrace-gpu/main.c new file +18 -0
+		if (ret)
+			goto out_put_vdev;
+	}
+
+	ret = vfio_pci_core_register_device(&nvdev->core_device);
+	if (ret)
+		goto out_put_vdev;
+
+	return ret;
+
+out_put_vdev:
+	vfio_put_device(&nvdev->core_device.vdev);
+	return ret;
+}
+
+static void nvgrace_gpu_remove(struct pci_dev *pdev)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =

nvgrace_gpu_drvdata(pdev);

drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+	struct vfio_pci_core_device *vdev = &nvdev->core_device;
+
+	vfio_pci_core_unregister_device(vdev);
+	vfio_put_device(&vdev->vdev);
+}
+
+static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = {
+	/* GH200 120GB */
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA,

0x2342) },

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+	/* GH200 480GB */
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA,

0x2345) },

drivers/vfio/pci/nvgrace-gpu/main.c new file +20 -0
+	{}
+};
+
+MODULE_DEVICE_TABLE(pci, nvgrace_gpu_vfio_pci_table);
+
+static struct pci_driver nvgrace_gpu_vfio_pci_driver = {
+	.name = KBUILD_MODNAME,
+	.id_table = nvgrace_gpu_vfio_pci_table,
+	.probe = nvgrace_gpu_probe,
+	.remove = nvgrace_gpu_remove,
+	.err_handler = &vfio_pci_core_err_handlers,
+	.driver_managed_dma = true,
+};
+
+module_pci_driver(nvgrace_gpu_vfio_pci_driver);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Ankit Agrawal <[email protected]>");
+MODULE_AUTHOR("Aniket Agashe <[email protected]>");
+MODULE_DESCRIPTION("VFIO NVGRACE GPU PF - User Level driver for

NVIDIA devices with CPU coherently accessible device memory");

author maintainer
#9 lore raw
Quoted 8 lines
Quoted 6 lines

Review

Quoted 4 lines

+/* Memory size expected as non cached and reserved by the VM driver */ +#define RESMEM_SIZE 0x40000000

(quoted diff) +2 -0
+#define MEMBLK_SIZE 0x20000000
+

Maybe use SZ_* definitions in linux/size.h

Good suggestion.

Ack.

Quoted 10 lines
Quoted 8 lines

Better move this part to the place between vfio_pci_core_enable() and vfio_pci_core_finish_enable() like others for respecting the expected device initialization sequence of life cycle.

It doesn't bite something right now, but think about when someone changes the behavior of vfio_pci_core_finish_enable() in the future, they have to propose a patch for this.

Agree.

Good point, will move it.

Quoted 6 lines
Quoted 2 lines

Wouldn't it be better to do the map in the open path?

AIUI the device would typically be used exclusively through the mmap of these ranges, these mappings are only for pread/pwrite type accesses, so I think it makes sense to map them on demand.

That's right, agree with Alex to keep it on-demand.

Quoted 12 lines
Quoted 11 lines
Quoted 8 lines

+    * Determine how many bytes to be actually read from the device memory.
+    * Read request beyond the actual device memory size is filled with ~0,
+    * while those beyond the actual reported size is skipped. +    */
+   if (offset >= memregion->memlength)
+           mem_count = 0;

If mem_count == 0, going through nvgrace_gpu_map_and_read() is not necessary.

Quoted 6 lines

Harmless, other than the possibly unnecessary call through to nvgrace_gpu_map_device_mem().  Maybe both nvgrace_gpu_map_and_read() and nvgrace_gpu_map_and_write() could conditionally return 0 as their first operation when !mem_count.  Thanks,

Alex

IMO, this seems like adding too much code to reduce the call length for a very specific case. If there aren't any strong opinion on this, I'm planning to leave this code as it is.

reviewer
#10 lore raw
Quoted 17 lines

From: Ankit Agrawal <[email protected]>
Sent: Thursday, February 8, 2024 3:13 PM

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

+    * Determine how many bytes to be actually read from the device memory.
+    * Read request beyond the actual device memory size is filled with ~0,
+    * while those beyond the actual reported size is skipped. +    */
+   if (offset >= memregion->memlength)
+           mem_count = 0;

Quoted 5 lines
Quoted 2 lines

If mem_count == 0, going through nvgrace_gpu_map_and_read() is not necessary.

Harmless, other than the possibly unnecessary call through to nvgrace_gpu_map_device_mem().  Maybe both

nvgrace_gpu_map_and_read()

Quoted 8 lines
Quoted 4 lines

and nvgrace_gpu_map_and_write() could conditionally return 0 as their first operation when !mem_count.  Thanks,

Alex

IMO, this seems like adding too much code to reduce the call length for a very specific case. If there aren't any strong opinion on this, I'm planning to leave this code as it is.

a slight difference. if mem_count==0 the result should always succeed no matter nvgrace_gpu_map_device_mem() succeeds or not. Of course if it fails it's already a big problem probably nobody cares about the subtle difference when reading non-exist range.

but regarding to readability it's still clearer:

if (mem_count)

	nvgrace_gpu_map_and_read();
author maintainer
#11 lore raw
Quoted 21 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

If mem_count == 0, going through nvgrace_gpu_map_and_read() is not necessary.

Harmless, other than the possibly unnecessary call through to nvgrace_gpu_map_device_mem().  Maybe both

nvgrace_gpu_map_and_read()

Quoted 8 lines
Quoted 4 lines

and nvgrace_gpu_map_and_write() could conditionally return 0 as their first operation when !mem_count.  Thanks,

Alex

IMO, this seems like adding too much code to reduce the call length for a very specific case. If there aren't any strong opinion on this, I'm planning to leave this code as it is.

a slight difference. if mem_count==0 the result should always succeed no matter nvgrace_gpu_map_device_mem() succeeds or not. Of course if it fails it's already a big problem probably nobody cares about the subtle difference when reading non-exist range.

Quoted 4 lines

but regarding to readability it's still clearer:

if (mem_count)
        nvgrace_gpu_map_and_read();

Makes sense. I'll change it.

#12 lore raw
In reply to Tian, Kevin
Quoted 31 lines · Kevin
Quoted 17 lines

From: Ankit Agrawal <[email protected]>
Sent: Thursday, February 8, 2024 3:13 PM

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

+    * Determine how many bytes to be actually read from the device memory.
+    * Read request beyond the actual device memory size is filled with ~0,
+    * while those beyond the actual reported size is skipped. +    */
+   if (offset >= memregion->memlength)
+           mem_count = 0;

Quoted 5 lines
Quoted 2 lines

If mem_count == 0, going through nvgrace_gpu_map_and_read() is not necessary.

Harmless, other than the possibly unnecessary call through to nvgrace_gpu_map_device_mem().  Maybe both

nvgrace_gpu_map_and_read()

Quoted 8 lines
Quoted 4 lines

and nvgrace_gpu_map_and_write() could conditionally return 0 as their first operation when !mem_count.  Thanks,

Alex

IMO, this seems like adding too much code to reduce the call length for a very specific case. If there aren't any strong opinion on this, I'm planning to leave this code as it is.

a slight difference. if mem_count==0 the result should always succeed no matter nvgrace_gpu_map_device_mem() succeeds or not. Of course if it fails it's already a big problem probably nobody cares about the subtle difference when reading non-exist range.

Quoted 5 lines

but regarding to readability it's still clearer:

if (mem_count)

	nvgrace_gpu_map_and_read();

The below has better flow imo vs conditionalizing the call to map_and_read/write and subsequent error handling, but I don't think either adds too much code. Thanks,

Alex

drivers/vfio/pci/nvgrace-gpu/main.c +6 -0
@@ -429,6 +429,9 @@ nvgrace_gpu_map_and_read(struct nvgrace_gpu_vfio_pci_core_device *nvdev,
        u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
        int ret;
 
+       if (!mem_count)
+               return 0;
+
        /*
         * Handle read on the BAR regions. Map to the target device memory
         * physical address and copy to the request read buffer.
@@ -547,6 +550,9 @@ nvgrace_gpu_map_and_write(struct nvgrace_gpu_vfio_pci_core_device *nvdev,
        loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
        int ret;
 
+       if (!mem_count)
+               return 0;
+
        ret = nvgrace_gpu_map_device_mem(index, nvdev);
        if (ret)
                return ret;
author maintainer
#13 lore raw
Quoted 35 lines
Quoted 10 lines
Quoted 4 lines

IMO, this seems like adding too much code to reduce the call length for a very specific case. If there aren't any strong opinion on this, I'm planning to leave this code as it is.

a slight difference. if mem_count==0 the result should always succeed no matter nvgrace_gpu_map_device_mem() succeeds or not. Of course if it fails it's already a big problem probably nobody cares about the subtle difference when reading non-exist range.

Quoted 5 lines

but regarding to readability it's still clearer:

if (mem_count)
       nvgrace_gpu_map_and_read();

The below has better flow imo vs conditionalizing the call to map_and_read/write and subsequent error handling, but I don't think either adds too much code.  Thanks,

Alex

drivers/vfio/pci/nvgrace-gpu/main.c +3 -0
@@ -429,6 +429,9 @@ nvgrace_gpu_map_and_read(struct nvgrace_gpu_vfio_pci_core_device *nvdev,
          u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
          int ret;
 
+       if (!mem_count)
+               return 0;
+
         /*
         * Handle read on the BAR regions. Map to the target device memory
          * physical address and copy to the request read buffer.

drivers/vfio/pci/nvgrace-gpu/main.c nvgrace_gpu_map_and_write(struct nvgrace_gpu_vfio_pci_core_device *nvdev, · line 558

Quoted 9 lines
drivers/vfio/pci/nvgrace-gpu/main.c +3 -0
@@ -547,6 +550,9 @@ nvgrace_gpu_map_and_write(struct nvgrace_gpu_vfio_pci_core_device *nvdev,
         loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
         int ret;
 
+       if (!mem_count)
+               return 0;
+
         ret = nvgrace_gpu_map_device_mem(index, nvdev);
         if (ret)
                 return ret;

Sure, will update it as mentioned above.

#14 lore raw
In reply to [email protected]
Quoted 117 lines · On Tue, 6 Feb 2024 04:31:23 +0530 <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device for the on-chip GPU that is the logical OS representation of the internal proprietary chip-to-chip cache coherent interconnect.

The device is peculiar compared to a real PCI device in that whilst there is a real 64b PCI BAR1 (comprising region 2 & region 3) on the device, it is not used to access device memory once the faster chip-to-chip interconnect is initialized (occurs at the time of host system boot). The device memory is accessed instead using the chip-to-chip interconnect that is exposed as a contiguous physically addressable region on the host. This device memory aperture can be obtained from host ACPI table using device_property_read_u64(), according to the FW specification. Since the device memory is cache coherent with the CPU, it can be mmap into the user VMA with a cacheable mapping using remap_pfn_range() and used like a regular RAM. The device memory is not added to the host kernel, but mapped directly as this reduces memory wastage due to struct pages.

There is also a requirement of a reserved 1G uncached region (termed as resmem) to support the Multi-Instance GPU (MIG) feature [1]. This is to work around a HW defect. Based on [2], the requisite properties (uncached, unaligned access) can be achieved through a VM mapping (S1) of NORMAL_NC and host (S2) mapping with MemAttr[2:0]=0b101. To provide a different non-cached property to the reserved 1G region, it needs to be carved out from the device memory and mapped as a separate region in Qemu VMA with pgprot_writecombine(). pgprot_writecombine() sets the Qemu VMA page properties (pgprot) as NORMAL_NC.

Provide a VFIO PCI variant driver that adapts the unique device memory representation into a more standard PCI representation facing userspace.

The variant driver exposes these two regions - the non-cached reserved (resmem) and the cached rest of the device memory (termed as usemem) as separate VFIO 64b BAR regions. This is divergent from the baremetal approach, where the device memory is exposed as a device memory region. The decision for a different approach was taken in view of the fact that it would necessiate additional code in Qemu to discover and insert those regions in the VM IPA, along with the additional VM ACPI DSDT changes to communicate the device memory region IPA to the VM workloads. Moreover, this behavior would have to be added to a variety of emulators (beyond top of tree Qemu) out there desiring grace hopper support.

Since the device implements 64-bit BAR0, the VFIO PCI variant driver maps the uncached carved out region to the next available PCI BAR (i.e. comprising of region 2 and 3). The cached device memory aperture is assigned BAR region 4 and 5. Qemu will then naturally generate a PCI device in the VM with the uncached aperture reported as BAR2 region, the cacheable as BAR4. The variant driver provides emulation for these fake BARs' PCI config space offset registers.

The hardware ensures that the system does not crash when the memory is accessed with the memory enable turned off. It synthesis ~0 reads and dropped writes on such access. So there is no need to support the disablement/enablement of BAR through PCI_COMMAND config space register.

The memory layout on the host looks like the following:
devmem (memlength)

|--------------------------------------------------|
|-------------cached------------------------|--NC--|
|                                           |
usemem.phys/memphys                         resmem.phys

PCI BARs need to be aligned to the power-of-2, but the actual memory on the device may not. A read or write access to the physical address from the last device PFN up to the next power-of-2 aligned physical address results in reading ~0 and dropped writes. Note that the GPU device driver [6] is capable of knowing the exact device memory size through separate means. The device memory size is primarily kept in the system ACPI tables for use by the VFIO PCI variant module.

Note that the usemem memory is added by the VM Nvidia device driver [5] to the VM kernel as memblocks. Hence make the usable memory size memblock aligned.

Currently there is no provision in KVM for a S2 mapping with MemAttr[2:0]=0b101, but there is an ongoing effort to provide the same [3]. As previously mentioned, resmem is mapped pgprot_writecombine(), that sets the Qemu VMA page properties (pgprot) as NORMAL_NC. Using the proposed changes in [4] and [3], KVM marks the region with MemAttr[2:0]=0b101 in S2.

If the bare metal properties are not present, the driver registers the vfio-pci-core function pointers.

This goes along with a qemu series [6] to provides the necessary implementation of the Grace Hopper Superchip firmware specification so that the guest operating system can see the correct ACPI modeling for the coherent GPU device. Verified with the CUDA workload in the VM.

[1] https://www.nvidia.com/en-in/technologies/multi-instance-gpu/
[2] section D8.5.5 of https://developer.arm.com/documentation/ddi0487/latest/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/
[5] https://github.com/NVIDIA/open-gpu-kernel-modules
[6] https://lore.kernel.org/all/[email protected]/

---
 MAINTAINERS                           |   6 +
 drivers/vfio/pci/Kconfig              |   2 +
 drivers/vfio/pci/Makefile             |   2 +
 drivers/vfio/pci/nvgrace-gpu/Kconfig  |  10 +
 drivers/vfio/pci/nvgrace-gpu/Makefile |   3 +
 drivers/vfio/pci/nvgrace-gpu/main.c   | 856 ++++++++++++++++++++++++++
 6 files changed, 879 insertions(+)
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/Kconfig
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/Makefile
 create mode 100644 drivers/vfio/pci/nvgrace-gpu/main.c
MAINTAINERS +0 -0
index 8999497011a2..529ec8966f58 100644

MAINTAINERS L: [email protected] · line 23111

Quoted 9 lines
MAINTAINERS +6 -0
@@ -23103,6 +23103,12 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/vfio/platform/
 
+VFIO NVIDIA GRACE GPU DRIVER
+M:	Ankit Agrawal <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	drivers/vfio/pci/nvgrace-gpu/
+

Entries should be alphabetical. This will end up colliding with [1] so I'll plan to fix it either way.

Otherwise just a couple optional comments from me below. I see Zhi also has a few good comments. I'd suggest soliciting a review from the other variant driver reviewers for this version and maybe we can make v18 the final version. Thanks,

Alex

[1]https://lore.kernel.org/all/[email protected]/

Quoted 405 lines
MAINTAINERS +0 -0
 VGA_SWITCHEROO
 R:	Lukas Wunner <[email protected]>
 S:	Maintained
drivers/vfio/pci/Kconfig +2 -0
index 18c397df566d..15821a2d77d2 100644
@@ -67,4 +67,6 @@ source "drivers/vfio/pci/pds/Kconfig"
 
 source "drivers/vfio/pci/virtio/Kconfig"
 
+source "drivers/vfio/pci/nvgrace-gpu/Kconfig"
+
 endmenu
drivers/vfio/pci/Makefile +2 -0
index 046139a4eca5..ce7a61f1d912 100644
@@ -15,3 +15,5 @@ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/
 obj-$(CONFIG_PDS_VFIO_PCI) += pds/
 
 obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/
+
+obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/
drivers/vfio/pci/nvgrace-gpu/Kconfig new file +10 -0
new file mode 100644
index 000000000000..936e88d8d41d
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config NVGRACE_GPU_VFIO_PCI
+	tristate "VFIO support for the GPU in the NVIDIA Grace Hopper Superchip"
+	depends on ARM64 || (COMPILE_TEST && 64BIT)
+	select VFIO_PCI_CORE
+	help
+	  VFIO support for the GPU in the NVIDIA Grace Hopper Superchip is
+	  required to assign the GPU device using KVM/qemu/etc.
+
+	  If you don't know what to do here, say N.
drivers/vfio/pci/nvgrace-gpu/Makefile new file +3 -0
new file mode 100644
index 000000000000..3ca8c187897a
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu-vfio-pci.o
+nvgrace-gpu-vfio-pci-y := main.o
drivers/vfio/pci/nvgrace-gpu/main.c new file +350 -0
new file mode 100644
index 000000000000..6279af2bc6b8
@@ -0,0 +1,856 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ */
+
+#include <linux/vfio_pci_core.h>
+
+/*
+ * The device memory usable to the workloads running in the VM is cached
+ * and showcased as a 64b device BAR (comprising of BAR4 and BAR5 region)
+ * to the VM and is represented as usemem.
+ * Moreover, the VM GPU device driver needs a non-cacheable region to
+ * support the MIG feature. This region is also exposed as a 64b BAR
+ * (comprising of BAR2 and BAR3 region) and represented as resmem.
+ */
+#define RESMEM_REGION_INDEX VFIO_PCI_BAR2_REGION_INDEX
+#define USEMEM_REGION_INDEX VFIO_PCI_BAR4_REGION_INDEX
+
+/* Memory size expected as non cached and reserved by the VM driver */
+#define RESMEM_SIZE 0x40000000
+#define MEMBLK_SIZE 0x20000000
+
+/*
+ * The state of the two device memory region - resmem and usemem - is
+ * saved as struct mem_region.
+ */
+struct mem_region {
+	phys_addr_t memphys;    /* Base physical address of the region */
+	size_t memlength;       /* Region size */
+	size_t bar_size;        /* Reported region BAR size */
+	__le64 bar_val;         /* Emulated BAR offset registers */
+	union {
+		void *memaddr;
+		void __iomem *ioaddr;
+	};                      /* Base virtual address of the region */
+};
+
+struct nvgrace_gpu_vfio_pci_core_device {
+	struct vfio_pci_core_device core_device;
+	/* Cached and usable memory for the VM. */
+	struct mem_region usemem;
+	/* Non cached memory carved out from the end of device memory */
+	struct mem_region resmem;
+	/* Lock to control device memory kernel mapping */
+	struct mutex remap_lock;
+};
+
+static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device *core_vdev)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvgrace_gpu_vfio_pci_core_device,
+			     core_device.vdev);
+
+	nvdev->resmem.bar_val = 0;
+	nvdev->usemem.bar_val = 0;
+}
+
+/* Choose the structure corresponding to the fake BAR with a given index. */
+static struct mem_region *
+nvgrace_gpu_memregion(int index,
+		      struct nvgrace_gpu_vfio_pci_core_device *nvdev)
+{
+	if (index == USEMEM_REGION_INDEX)
+		return &nvdev->usemem;
+
+	if (index == RESMEM_REGION_INDEX)
+		return &nvdev->resmem;
+
+	return NULL;
+}
+
+static int nvgrace_gpu_open_device(struct vfio_device *core_vdev)
+{
+	struct vfio_pci_core_device *vdev =
+		container_of(core_vdev, struct vfio_pci_core_device, vdev);
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvgrace_gpu_vfio_pci_core_device,
+			     core_device.vdev);
+	int ret;
+
+	ret = vfio_pci_core_enable(vdev);
+	if (ret)
+		return ret;
+
+	vfio_pci_core_finish_enable(vdev);
+
+	if (nvdev->usemem.memlength) {
+		nvgrace_gpu_init_fake_bar_emu_regs(core_vdev);
+		mutex_init(&nvdev->remap_lock);
+	}
+
+	return 0;
+}
+
+static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvgrace_gpu_vfio_pci_core_device,
+			     core_device.vdev);
+
+	/* Unmap the mapping to the device memory cached region */
+	if (nvdev->usemem.memaddr) {
+		memunmap(nvdev->usemem.memaddr);
+		nvdev->usemem.memaddr = NULL;
+	}
+
+	/* Unmap the mapping to the device memory non-cached region */
+	if (nvdev->resmem.ioaddr) {
+		iounmap(nvdev->resmem.ioaddr);
+		nvdev->resmem.ioaddr = NULL;
+	}
+
+	mutex_destroy(&nvdev->remap_lock);
+
+	vfio_pci_core_close_device(core_vdev);
+}
+
+static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
+			    struct vm_area_struct *vma)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvgrace_gpu_vfio_pci_core_device,
+			     core_device.vdev);
+
+	unsigned long start_pfn;
+	unsigned int index;
+	u64 req_len, pgoff, end;
+	int ret = 0;
+	struct mem_region *memregion;
+
+	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return vfio_pci_core_mmap(core_vdev, vma);
+
+	/*
+	 * Request to mmap the BAR. Map to the CPU accessible memory on the
+	 * GPU using the memory information gathered from the system ACPI
+	 * tables.
+	 */
+	pgoff = vma->vm_pgoff &
+		((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
+
+	if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len) ||
+	    check_add_overflow(PHYS_PFN(memregion->memphys), pgoff, &start_pfn) ||
+	    check_add_overflow(PFN_PHYS(pgoff), req_len, &end))
+		return -EOVERFLOW;
+
+	/*
+	 * Check that the mapping request does not go beyond available device
+	 * memory size
+	 */
+	if (end > memregion->memlength)
+		return -EINVAL;
+
+	/*
+	 * The carved out region of the device memory needs the NORMAL_NC
+	 * property. Communicate as such to the hypervisor.
+	 */
+	if (index == RESMEM_REGION_INDEX)
+		vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
+
+	/*
+	 * Perform a PFN map to the memory and back the device BAR by the
+	 * GPU memory.
+	 *
+	 * The available GPU memory size may not be power-of-2 aligned. The
+	 * remainder is only backed by vfio_device_ops read/write handlers.
+	 *
+	 * During device reset, the GPU is safely disconnected to the CPU
+	 * and access to the BAR will be immediately returned preventing
+	 * machine check.
+	 */
+	ret = remap_pfn_range(vma, vma->vm_start, start_pfn,
+			      req_len, vma->vm_page_prot);
+	if (ret)
+		return ret;
+
+	vma->vm_pgoff = start_pfn;
+
+	return 0;
+}
+
+static long
+nvgrace_gpu_ioctl_get_region_info(struct vfio_device *core_vdev,
+				  unsigned long arg)
+{
+	unsigned long minsz = offsetofend(struct vfio_region_info, offset);
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvgrace_gpu_vfio_pci_core_device,
+			     core_device.vdev);
+	struct vfio_region_info_cap_sparse_mmap *sparse;
+	struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
+	struct vfio_region_info info;
+	struct mem_region *memregion;
+	u32 size;
+	int ret;
+
+	if (copy_from_user(&info, (void __user *)arg, minsz))
+		return -EFAULT;
+
+	if (info.argsz < minsz)
+		return -EINVAL;
+
+	memregion = nvgrace_gpu_memregion(info.index, nvdev);
+	if (!memregion)
+		return vfio_pci_core_ioctl(core_vdev,
+					   VFIO_DEVICE_GET_REGION_INFO, arg);
+
+	/*
+	 * Request to determine the BAR region information. Send the
+	 * GPU memory information.
+	 */
+	size = struct_size(sparse, areas, 1);
+
+	/*
+	 * Setup for sparse mapping for the device memory. Only the
+	 * available device memory on the hardware is shown as a
+	 * mappable region.
+	 */
+	sparse = kzalloc(size, GFP_KERNEL);
+	if (!sparse)
+		return -ENOMEM;
+
+	sparse->nr_areas = 1;
+	sparse->areas[0].offset = 0;
+	sparse->areas[0].size = memregion->memlength;
+	sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
+	sparse->header.version = 1;
+
+	ret = vfio_info_add_capability(&caps, &sparse->header, size);
+	kfree(sparse);
+	if (ret)
+		return ret;
+
+	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
+	/*
+	 * The region memory size may not be power-of-2 aligned.
+	 * Given that the memory  as a BAR and may not be
+	 * aligned, roundup to the next power-of-2.
+	 */
+	info.size = memregion->bar_size;
+	info.flags = VFIO_REGION_INFO_FLAG_READ |
+		     VFIO_REGION_INFO_FLAG_WRITE |
+		     VFIO_REGION_INFO_FLAG_MMAP;
+
+	if (caps.size) {
+		info.flags |= VFIO_REGION_INFO_FLAG_CAPS;
+		if (info.argsz < sizeof(info) + caps.size) {
+			info.argsz = sizeof(info) + caps.size;
+			info.cap_offset = 0;
+		} else {
+			vfio_info_cap_shift(&caps, sizeof(info));
+			if (copy_to_user((void __user *)arg +
+					 sizeof(info), caps.buf,
+					 caps.size)) {
+				kfree(caps.buf);
+				return -EFAULT;
+			}
+			info.cap_offset = sizeof(info);
+		}
+		kfree(caps.buf);
+	}
+	return copy_to_user((void __user *)arg, &info, minsz) ?
+			    -EFAULT : 0;
+}
+
+static long nvgrace_gpu_ioctl(struct vfio_device *core_vdev,
+			      unsigned int cmd, unsigned long arg)
+{
+	switch (cmd) {
+	case VFIO_DEVICE_GET_REGION_INFO:
+		return nvgrace_gpu_ioctl_get_region_info(core_vdev, arg);
+	case VFIO_DEVICE_IOEVENTFD:
+		return -ENOTTY;
+	case VFIO_DEVICE_RESET:
+		nvgrace_gpu_init_fake_bar_emu_regs(core_vdev);
+		fallthrough;
+	default:
+		return vfio_pci_core_ioctl(core_vdev, cmd, arg);
+	}
+}
+
+static __le64
+nvgrace_gpu_get_read_value(size_t bar_size, u64 flags, __le64 val64)
+{
+	u64 tmp_val;
+
+	tmp_val = le64_to_cpu(val64);
+	tmp_val &= ~(bar_size - 1);
+	tmp_val |= flags;
+
+	return cpu_to_le64(tmp_val);
+}
+
+/*
+ * Both the usable (usemem) and the reserved (resmem) device memory region
+ * are exposed as a 64b fake BARs in the VM. These fake BARs must respond
+ * to the accesses on their respective PCI config space offsets.
+ *
+ * resmem BAR owns PCI_BASE_ADDRESS_2 & PCI_BASE_ADDRESS_3.
+ * usemem BAR owns PCI_BASE_ADDRESS_4 & PCI_BASE_ADDRESS_5.
+ */
+static ssize_t
+nvgrace_gpu_read_config_emu(struct vfio_device *core_vdev,
+			    char __user *buf, size_t count, loff_t *ppos)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvgrace_gpu_vfio_pci_core_device,
+			     core_device.vdev);
+	struct mem_region *memregion = NULL;
+	u64 pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	__le64 val64;
+	size_t register_offset;
+	loff_t copy_offset;
+	size_t copy_count;
+	int ret;
+
+	ret = vfio_pci_core_read(core_vdev, buf, count, ppos);
+	if (ret < 0)
+		return ret;
+
+	if (vfio_pci_core_range_intersect_range(pos, count, PCI_BASE_ADDRESS_2,
+						sizeof(val64),
+						&copy_offset, &copy_count,
+						&register_offset))
+		memregion = nvgrace_gpu_memregion(RESMEM_REGION_INDEX, nvdev);
+	else if (vfio_pci_core_range_intersect_range(pos, count,
+						     PCI_BASE_ADDRESS_4,
+						     sizeof(val64),
+						     &copy_offset, &copy_count,
+						     &register_offset))
+		memregion = nvgrace_gpu_memregion(USEMEM_REGION_INDEX, nvdev);
+
+	if (memregion) {
+		val64 = nvgrace_gpu_get_read_value(memregion->bar_size,
+						   PCI_BASE_ADDRESS_MEM_TYPE_64 |
+						   PCI_BASE_ADDRESS_MEM_PREFETCH,
+						   memregion->bar_val);
+		if (copy_to_user(buf + copy_offset,
+				 (void *)&val64 + register_offset, copy_count))
+			return -EFAULT;
+	}
+
+	return count;
+}
+
+static ssize_t
+nvgrace_gpu_write_config_emu(struct vfio_device *core_vdev,

drivers/vfio/pci/nvgrace-gpu/main.c line 360

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +10 -0
+			     const char __user *buf, size_t count, loff_t *ppos)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvgrace_gpu_vfio_pci_core_device,
+			     core_device.vdev);
+	u64 pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	size_t register_offset;
+	loff_t copy_offset;
+	size_t copy_count;
+	struct mem_region *memregion = NULL;

Nit, consistency and reverse Christmas tree variable declaration would suggest pushing this up in the list, but it's not strictly required.

Quoted 44 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +44 -0
+
+	if (vfio_pci_core_range_intersect_range(pos, count, PCI_BASE_ADDRESS_2,
+						sizeof(u64), &copy_offset,
+						&copy_count, &register_offset))
+		memregion = nvgrace_gpu_memregion(RESMEM_REGION_INDEX, nvdev);
+	else if (vfio_pci_core_range_intersect_range(pos, count, PCI_BASE_ADDRESS_4,
+						     sizeof(u64), &copy_offset,
+						     &copy_count, &register_offset))
+		memregion = nvgrace_gpu_memregion(USEMEM_REGION_INDEX, nvdev);
+
+	if (memregion) {
+		if (copy_from_user((void *)&memregion->bar_val + register_offset,
+				   buf + copy_offset, copy_count))
+			return -EFAULT;
+		*ppos += copy_count;
+		return copy_count;
+	}
+
+	return vfio_pci_core_write(core_vdev, buf, count, ppos);
+}
+
+/*
+ * Ad hoc map the device memory in the module kernel VA space. Primarily needed
+ * as vfio does not require the userspace driver to only perform accesses through
+ * mmaps of the vfio-pci BAR regions and such accesses should be supported using
+ * vfio_device_ops read/write implementations.
+ *
+ * The usemem region is cacheable memory and hence is memremaped.
+ * The resmem region is non-cached and is mapped using ioremap_wc (NORMAL_NC).
+ */
+static int
+nvgrace_gpu_map_device_mem(int index,
+			   struct nvgrace_gpu_vfio_pci_core_device *nvdev)
+{
+	struct mem_region *memregion;
+	int ret = 0;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return -EINVAL;
+
+	mutex_lock(&nvdev->remap_lock);
+	if (index == USEMEM_REGION_INDEX && !memregion->memaddr) {
+		memregion->memaddr = memremap(memregion->memphys,

drivers/vfio/pci/nvgrace-gpu/main.c line 414

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +10 -0
+					      memregion->memlength,
+					      MEMREMAP_WB);
+		if (!memregion->memaddr)
+			ret = -ENOMEM;
+	} else if (index == RESMEM_REGION_INDEX && !memregion->ioaddr) {
+		memregion->ioaddr = ioremap_wc(memregion->memphys,
+					       memregion->memlength);
+		if (!memregion->ioaddr)
+			ret = -ENOMEM;
+	}

As .memaddr and .ioaddr are a union we can consolidate the NULL test.

Quoted 442 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +442 -0
+	mutex_unlock(&nvdev->remap_lock);
+
+	return ret;
+}
+
+/*
+ * Read the data from the device memory (mapped either through ioremap
+ * or memremap) into the user buffer.
+ */
+static int
+nvgrace_gpu_map_and_read(struct nvgrace_gpu_vfio_pci_core_device *nvdev,
+			 char __user *buf, size_t mem_count, loff_t *ppos)
+{
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
+	int ret;
+
+	/*
+	 * Handle read on the BAR regions. Map to the target device memory
+	 * physical address and copy to the request read buffer.
+	 */
+	ret = nvgrace_gpu_map_device_mem(index, nvdev);
+	if (ret)
+		return ret;
+
+	if (index == USEMEM_REGION_INDEX) {
+		if (copy_to_user(buf,
+				 (u8 *)nvdev->usemem.memaddr + offset,
+				 mem_count))
+			ret = -EFAULT;
+	} else {
+		/*
+		 * The hardware ensures that the system does not crash when
+		 * the device memory is accessed with the memory enable
+		 * turned off. It synthesizes ~0 on such read. So there is
+		 * no need to check or support the disablement/enablement of
+		 * BAR through PCI_COMMAND config space register. Pass
+		 * test_mem flag as false.
+		 */
+		ret = vfio_pci_core_do_io_rw(&nvdev->core_device, false,
+					     nvdev->resmem.ioaddr,
+					     buf, offset, mem_count,
+					     0, 0, false);
+	}
+
+	return ret;
+}
+
+/*
+ * Read count bytes from the device memory at an offset. The actual device
+ * memory size (available) may not be a power-of-2. So the driver fakes
+ * the size to a power-of-2 (reported) when exposing to a user space driver.
+ *
+ * Reads extending beyond the reported size are truncated; reads starting
+ * beyond the reported size generate -EINVAL; reads extending beyond the
+ * actual device size is filled with ~0.
+ */
+static ssize_t
+nvgrace_gpu_read_mem(struct nvgrace_gpu_vfio_pci_core_device *nvdev,
+		     char __user *buf, size_t count, loff_t *ppos)
+{
+	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct mem_region *memregion;
+	size_t mem_count, i;
+	u8 val = 0xFF;
+	int ret;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return -EINVAL;
+
+	if (offset >= memregion->bar_size)
+		return -EINVAL;
+
+	/* Clip short the read request beyond reported BAR size */
+	count = min(count, memregion->bar_size - (size_t)offset);
+
+	/*
+	 * Determine how many bytes to be actually read from the device memory.
+	 * Read request beyond the actual device memory size is filled with ~0,
+	 * while those beyond the actual reported size is skipped.
+	 */
+	if (offset >= memregion->memlength)
+		mem_count = 0;
+	else
+		mem_count = min(count, memregion->memlength - (size_t)offset);
+
+	ret = nvgrace_gpu_map_and_read(nvdev, buf, mem_count, ppos);
+	if (ret)
+		return ret;
+
+	/*
+	 * Only the device memory present on the hardware is mapped, which may
+	 * not be power-of-2 aligned. A read to an offset beyond the device memory
+	 * size is filled with ~0.
+	 */
+	for (i = mem_count; i < count; i++)
+		put_user(val, (unsigned char __user *)(buf + i));
+
+	*ppos += count;
+	return count;
+}
+
+static ssize_t
+nvgrace_gpu_read(struct vfio_device *core_vdev,
+		 char __user *buf, size_t count, loff_t *ppos)
+{
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvgrace_gpu_vfio_pci_core_device,
+			     core_device.vdev);
+
+	if (nvgrace_gpu_memregion(index, nvdev))
+		return nvgrace_gpu_read_mem(nvdev, buf, count, ppos);
+
+	if (index == VFIO_PCI_CONFIG_REGION_INDEX)
+		return nvgrace_gpu_read_config_emu(core_vdev, buf, count, ppos);
+
+	return vfio_pci_core_read(core_vdev, buf, count, ppos);
+}
+
+/*
+ * Write the data to the device memory (mapped either through ioremap
+ * or memremap) from the user buffer.
+ */
+static int
+nvgrace_gpu_map_and_write(struct nvgrace_gpu_vfio_pci_core_device *nvdev,
+			  const char __user *buf, size_t mem_count,
+			  loff_t *ppos)
+{
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	int ret;
+
+	ret = nvgrace_gpu_map_device_mem(index, nvdev);
+	if (ret)
+		return ret;
+
+	if (index == USEMEM_REGION_INDEX) {
+		if (copy_from_user((u8 *)nvdev->usemem.memaddr + pos,
+				   buf, mem_count))
+			return -EFAULT;
+	} else {
+		/*
+		 * The hardware ensures that the system does not crash when
+		 * the device memory is accessed with the memory enable
+		 * turned off. It drops such writes. So there is no need to
+		 * check or support the disablement/enablement of BAR
+		 * through PCI_COMMAND config space register. Pass test_mem
+		 * flag as false.
+		 */
+		ret = vfio_pci_core_do_io_rw(&nvdev->core_device, false,
+					     nvdev->resmem.ioaddr,
+					     (char __user *)buf, pos, mem_count,
+					     0, 0, true);
+	}
+
+	return ret;
+}
+
+/*
+ * Write count bytes to the device memory at a given offset. The actual device
+ * memory size (available) may not be a power-of-2. So the driver fakes the
+ * size to a power-of-2 (reported) when exposing to a user space driver.
+ *
+ * Writes extending beyond the reported size are truncated; writes starting
+ * beyond the reported size generate -EINVAL.
+ */
+static ssize_t
+nvgrace_gpu_write_mem(struct nvgrace_gpu_vfio_pci_core_device *nvdev,
+		      size_t count, loff_t *ppos, const char __user *buf)
+{
+	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct mem_region *memregion;
+	size_t mem_count;
+	int ret = 0;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return -EINVAL;
+
+	if (offset >= memregion->bar_size)
+		return -EINVAL;
+
+	/* Clip short the write request beyond reported BAR size */
+	count = min(count, memregion->bar_size - (size_t)offset);
+
+	/*
+	 * Determine how many bytes to be actually written to the device memory.
+	 * Do not write to the offset beyond available size.
+	 */
+	if (offset >= memregion->memlength)
+		goto exitfn;
+
+	/*
+	 * Only the device memory present on the hardware is mapped, which may
+	 * not be power-of-2 aligned. Drop access outside the available device
+	 * memory on the hardware.
+	 */
+	mem_count = min(count, memregion->memlength - (size_t)offset);
+
+	ret = nvgrace_gpu_map_and_write(nvdev, buf, mem_count, ppos);
+	if (ret)
+		return ret;
+
+exitfn:
+	*ppos += count;
+	return count;
+}
+
+static ssize_t
+nvgrace_gpu_write(struct vfio_device *core_vdev,
+		  const char __user *buf, size_t count, loff_t *ppos)
+{
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct nvgrace_gpu_vfio_pci_core_device,
+			     core_device.vdev);
+
+	if (nvgrace_gpu_memregion(index, nvdev))
+		return nvgrace_gpu_write_mem(nvdev, count, ppos, buf);
+
+	if (index == VFIO_PCI_CONFIG_REGION_INDEX)
+		return nvgrace_gpu_write_config_emu(core_vdev, buf, count, ppos);
+
+	return vfio_pci_core_write(core_vdev, buf, count, ppos);
+}
+
+static const struct vfio_device_ops nvgrace_gpu_vfio_pci_ops = {
+	.name		= "nvgrace-gpu-vfio-pci",
+	.init		= vfio_pci_core_init_dev,
+	.release	= vfio_pci_core_release_dev,
+	.open_device	= nvgrace_gpu_open_device,
+	.close_device	= nvgrace_gpu_close_device,
+	.ioctl		= nvgrace_gpu_ioctl,
+	.read		= nvgrace_gpu_read,
+	.write		= nvgrace_gpu_write,
+	.mmap		= nvgrace_gpu_mmap,
+	.request	= vfio_pci_core_request,
+	.match		= vfio_pci_core_match,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
+	.detach_ioas	= vfio_iommufd_physical_detach_ioas,
+};
+
+static const struct vfio_device_ops nvgrace_gpu_vfio_pci_core_ops = {
+	.name		= "nvgrace-gpu-vfio-pci-core",
+	.init		= vfio_pci_core_init_dev,
+	.release	= vfio_pci_core_release_dev,
+	.open_device	= nvgrace_gpu_open_device,
+	.close_device	= vfio_pci_core_close_device,
+	.ioctl		= vfio_pci_core_ioctl,
+	.device_feature	= vfio_pci_core_ioctl_feature,
+	.read		= vfio_pci_core_read,
+	.write		= vfio_pci_core_write,
+	.mmap		= vfio_pci_core_mmap,
+	.request	= vfio_pci_core_request,
+	.match		= vfio_pci_core_match,
+	.bind_iommufd	= vfio_iommufd_physical_bind,
+	.unbind_iommufd	= vfio_iommufd_physical_unbind,
+	.attach_ioas	= vfio_iommufd_physical_attach_ioas,
+	.detach_ioas	= vfio_iommufd_physical_detach_ioas,
+};
+
+static struct
+nvgrace_gpu_vfio_pci_core_device *nvgrace_gpu_drvdata(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev);
+
+	return container_of(core_device, struct nvgrace_gpu_vfio_pci_core_device,
+			    core_device);
+}
+
+static int
+nvgrace_gpu_fetch_memory_property(struct pci_dev *pdev,
+				  u64 *pmemphys, u64 *pmemlength)
+{
+	int ret;
+
+	/*
+	 * The memory information is present in the system ACPI tables as DSD
+	 * properties nvidia,gpu-mem-base-pa and nvidia,gpu-mem-size.
+	 */
+	ret = device_property_read_u64(&pdev->dev, "nvidia,gpu-mem-base-pa",
+				       pmemphys);
+	if (ret)
+		return ret;
+
+	if (*pmemphys > type_max(phys_addr_t))
+		return -EOVERFLOW;
+
+	ret = device_property_read_u64(&pdev->dev, "nvidia,gpu-mem-size",
+				       pmemlength);
+	if (ret)
+		return ret;
+
+	if (*pmemlength > type_max(size_t))
+		return -EOVERFLOW;
+
+	/*
+	 * If the C2C link is not up due to an error, the coherent device
+	 * memory size is returned as 0. Fail in such case.
+	 */
+	if (*pmemlength == 0)
+		return -ENOMEM;
+
+	return ret;
+}
+
+static int
+nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
+			      struct nvgrace_gpu_vfio_pci_core_device *nvdev,
+			      u64 memphys, u64 memlength)
+{
+	int ret = 0;
+
+	/*
+	 * The VM GPU device driver needs a non-cacheable region to support
+	 * the MIG feature. Since the device memory is mapped as NORMAL cached,
+	 * carve out a region from the end with a different NORMAL_NC
+	 * property (called as reserved memory and represented as resmem). This
+	 * region then is exposed as a 64b BAR (region 2 and 3) to the VM, while
+	 * exposing the rest (termed as usable memory and represented using usemem)
+	 * as cacheable 64b BAR (region 4 and 5).
+	 *
+	 *               devmem (memlength)
+	 * |-------------------------------------------------|
+	 * |                                           |
+	 * usemem.phys/memphys                         resmem.phys
+	 */
+	nvdev->usemem.memphys = memphys;
+
+	/*
+	 * The device memory exposed to the VM is added to the kernel by the
+	 * VM driver module in chunks of memory block size. Only the usable
+	 * memory (usemem) is added to the kernel for usage by the VM
+	 * workloads. Make the usable memory size memblock aligned.
+	 */
+	if (check_sub_overflow(memlength, RESMEM_SIZE,
+			       &nvdev->usemem.memlength)) {
+		ret = -EOVERFLOW;
+		goto done;
+	}
+	nvdev->usemem.memlength = round_down(nvdev->usemem.memlength,
+					     MEMBLK_SIZE);
+	if ((check_add_overflow(nvdev->usemem.memphys,
+				nvdev->usemem.memlength,
+				&nvdev->resmem.memphys)) ||
+	    (check_sub_overflow(memlength, nvdev->usemem.memlength,
+				&nvdev->resmem.memlength))) {
+		ret = -EOVERFLOW;
+		goto done;
+	}
+
+	/*
+	 * The memory regions are exposed as BARs. Calculate and save
+	 * the BAR size for them.
+	 */
+	nvdev->usemem.bar_size = roundup_pow_of_two(nvdev->usemem.memlength);
+	nvdev->resmem.bar_size = roundup_pow_of_two(nvdev->resmem.memlength);
+done:
+	return ret;
+}
+
+static int nvgrace_gpu_probe(struct pci_dev *pdev,
+			     const struct pci_device_id *id)
+{
+	const struct vfio_device_ops *ops = &nvgrace_gpu_vfio_pci_core_ops;
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev;
+	u64 memphys, memlength;
+	int ret;
+
+	ret = nvgrace_gpu_fetch_memory_property(pdev, &memphys, &memlength);
+	if (!ret)
+		ops = &nvgrace_gpu_vfio_pci_ops;
+
+	nvdev = vfio_alloc_device(nvgrace_gpu_vfio_pci_core_device, core_device.vdev,
+				  &pdev->dev, ops);
+	if (IS_ERR(nvdev))
+		return PTR_ERR(nvdev);
+
+	dev_set_drvdata(&pdev->dev, &nvdev->core_device);
+
+	if (ops == &nvgrace_gpu_vfio_pci_ops) {
+		/*
+		 * Device memory properties are identified in the host ACPI
+		 * table. Set the nvgrace_gpu_vfio_pci_core_device structure.
+		 */
+		ret = nvgrace_gpu_init_nvdev_struct(pdev, nvdev,
+						    memphys, memlength);
+		if (ret)
+			goto out_put_vdev;
+	}
+
+	ret = vfio_pci_core_register_device(&nvdev->core_device);
+	if (ret)
+		goto out_put_vdev;
+
+	return ret;
+
+out_put_vdev:
+	vfio_put_device(&nvdev->core_device.vdev);
+	return ret;
+}
+
+static void nvgrace_gpu_remove(struct pci_dev *pdev)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev = nvgrace_gpu_drvdata(pdev);
+	struct vfio_pci_core_device *vdev = &nvdev->core_device;
+
+	vfio_pci_core_unregister_device(vdev);
+	vfio_put_device(&vdev->vdev);
+}
+
+static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = {
+	/* GH200 120GB */
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA, 0x2342) },
+	/* GH200 480GB */
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA, 0x2345) },
+	{}
+};
+
+MODULE_DEVICE_TABLE(pci, nvgrace_gpu_vfio_pci_table);
+
+static struct pci_driver nvgrace_gpu_vfio_pci_driver = {
+	.name = KBUILD_MODNAME,
+	.id_table = nvgrace_gpu_vfio_pci_table,
+	.probe = nvgrace_gpu_probe,
+	.remove = nvgrace_gpu_remove,
+	.err_handler = &vfio_pci_core_err_handlers,
+	.driver_managed_dma = true,
+};
+
+module_pci_driver(nvgrace_gpu_vfio_pci_driver);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Ankit Agrawal <[email protected]>");
+MODULE_AUTHOR("Aniket Agashe <[email protected]>");
+MODULE_DESCRIPTION("VFIO NVGRACE GPU PF - User Level driver for NVIDIA devices with CPU coherently accessible device memory");
author maintainer
#15 lore raw
Quoted 8 lines
Quoted 8 lines
MAINTAINERS +0 -0
index 8999497011a2..529ec8966f58 100644
@@ -23103,6 +23103,12 @@ L:   [email protected]

  S:   Maintained
  F:   drivers/vfio/platform/

Quoted 9 lines

MAINTAINERS L: [email protected] · line 23111

Quoted 6 lines
MAINTAINERS +6 -0
+VFIO NVIDIA GRACE GPU DRIVER
+M:   Ankit Agrawal <[email protected]>
+L:   [email protected]
+S:   Supported
+F:   drivers/vfio/pci/nvgrace-gpu/
+

Entries should be alphabetical.  This will end up colliding with [1] so I'll plan to fix it either way.

I will make the change to put it at the right place.

Quoted 8 lines

Otherwise just a couple optional comments from me below.  I see Zhi also has a few good comments.  I'd suggest soliciting a review from the other variant driver reviewers for this version and maybe we can make v18 the final version.  Thanks,

Alex

[1]https://lore.kernel.org/all/[email protected]/

Great!

Quoted 7 lines
Quoted 7 lines
MAINTAINERS +7 -0
+static ssize_t
+nvgrace_gpu_write_config_emu(struct vfio_device *core_vdev,
+                          const char __user *buf, size_t count, loff_t *ppos)
+{
+     struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+             container_of(core_vdev, struct nvgrace_gpu_vfio_pci_core_device,
+                          core_device.vdev);
Quoted 8 lines
Quoted 5 lines

+     u64 pos = *ppos & VFIO_PCI_OFFSET_MASK;
+     size_t register_offset;
+     loff_t copy_offset;
+     size_t copy_count;
+     struct mem_region *memregion = NULL;

Nit, consistency and reverse Christmas tree variable declaration would suggest pushing this up in the list, but it's not strictly required.

Ack, I'll make the change.

Quoted 13 lines
Quoted 4 lines

+     if (index == USEMEM_REGION_INDEX && !memregion->memaddr) { +             memregion->memaddr = memremap(memregion->memphys, +                                           memregion->memlength, +                                           MEMREMAP_WB);

Quoted 8 lines

+             if (!memregion->memaddr)
+                     ret = -ENOMEM;
+     } else if (index == RESMEM_REGION_INDEX && !memregion->ioaddr) { +             memregion->ioaddr = ioremap_wc(memregion->memphys, +                                            memregion->memlength); +             if (!memregion->ioaddr)
+                     ret = -ENOMEM;
+     }

Quoted 1 line

As .memaddr and .ioaddr are a union we can consolidate the NULL test.

Ok, will do that.

reviewer
#16 lore raw
In reply to [email protected]
Quoted 7 lines

From: [email protected] <[email protected]>
Sent: Tuesday, February 6, 2024 7:01 AM

Note that the usemem memory is added by the VM Nvidia device driver [5] to the VM kernel as memblocks. Hence make the usable memory size memblock
aligned.

Is memblock size defined in spec or purely a guest implementation choice?

Quoted 3 lines

If the bare metal properties are not present, the driver registers the vfio-pci-core function pointers.

so if qemu doesn't generate such property the variant driver running inside guest will always go to use core functions and guest vfio userspace will observe both resmem and usemem bars. But then there is nothing in field to prohibit mapping resmem bar as cacheable.

should this driver check the presence of either ACPI property or resmem/usemem bars to enable variant function pointers?

drivers/vfio/pci/nvgrace-gpu/Kconfig line 8

Quoted 8 lines
(quoted diff) +2 -0
+config NVGRACE_GPU_VFIO_PCI
+	tristate "VFIO support for the GPU in the NVIDIA Grace Hopper

Superchip"

drivers/vfio/pci/nvgrace-gpu/Kconfig new file +5 -0
+	depends on ARM64 || (COMPILE_TEST && 64BIT)
+	select VFIO_PCI_CORE
+	help
+	  VFIO support for the GPU in the NVIDIA Grace Hopper Superchip is
+	  required to assign the GPU device using KVM/qemu/etc.

"assign the GPU device to userspace"

drivers/vfio/pci/nvgrace-gpu/main.c line 21

Quoted 4 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+
+/* Memory size expected as non cached and reserved by the VM driver */
+#define RESMEM_SIZE 0x40000000
+#define MEMBLK_SIZE 0x20000000

also add a comment for MEMBLK_SIZE

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+struct nvgrace_gpu_vfio_pci_core_device {

will nvgrace refer to a non-gpu device? if not probably all prefixes with 'nvgrace_gpu' can be simplified to 'nvgrace'.

btw following other variant drivers 'vfio' can be removed too.

Quoted 5 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+
+/*
+ * Both the usable (usemem) and the reserved (resmem) device memory

region
+ * are exposed as a 64b fake BARs in the VM. These fake BARs must respond

s/VM/device/

Quoted 14 lines

+ * to the accesses on their respective PCI config space offsets.
+ *
+ * resmem BAR owns PCI_BASE_ADDRESS_2 & PCI_BASE_ADDRESS_3.
+ * usemem BAR owns PCI_BASE_ADDRESS_4 & PCI_BASE_ADDRESS_5.
+ */

drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+static ssize_t
+nvgrace_gpu_read_config_emu(struct vfio_device *core_vdev,
+			    char __user *buf, size_t count, loff_t *ppos)
+{
+	struct nvgrace_gpu_vfio_pci_core_device *nvdev =
+		container_of(core_vdev, struct

nvgrace_gpu_vfio_pci_core_device,

drivers/vfio/pci/nvgrace-gpu/main.c new file +2 -0
+			     core_device.vdev);
+	struct mem_region *memregion = NULL;

drivers/vfio/pci/nvgrace-gpu/main.c line 322

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +10 -0
+	u64 pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	__le64 val64;
+	size_t register_offset;
+	loff_t copy_offset;
+	size_t copy_count;
+	int ret;
+
+	ret = vfio_pci_core_read(core_vdev, buf, count, ppos);
+	if (ret < 0)
+		return ret;

here if core_read succeeds *ppos has been updated...

Quoted 23 lines
(quoted diff) +2 -0
+
+	if (vfio_pci_core_range_intersect_range(pos, count,

PCI_BASE_ADDRESS_2,

drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+						sizeof(val64),
+						&copy_offset, &copy_count,
+						&register_offset))
+		memregion =

nvgrace_gpu_memregion(RESMEM_REGION_INDEX, nvdev);

drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+	else if (vfio_pci_core_range_intersect_range(pos, count,
+						     PCI_BASE_ADDRESS_4,
+						     sizeof(val64),
+						     &copy_offset,

&copy_count,

(quoted diff) +2 -0
+						     &register_offset))
+		memregion =

nvgrace_gpu_memregion(USEMEM_REGION_INDEX, nvdev);

drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+
+	if (memregion) {
+		val64 = nvgrace_gpu_get_read_value(memregion->bar_size,
+

PCI_BASE_ADDRESS_MEM_TYPE_64 |
+
PCI_BASE_ADDRESS_MEM_PREFETCH,

drivers/vfio/pci/nvgrace-gpu/main.c line 343

Quoted 4 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+						   memregion->bar_val);
+		if (copy_to_user(buf + copy_offset,
+				 (void *)&val64 + register_offset, copy_count))
+			return -EFAULT;

...but here it's not adjusted back upon error.

Quoted 8 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +7 -0
+
+/*
+ * Read the data from the device memory (mapped either through ioremap
+ * or memremap) into the user buffer.
+ */
+static int
+nvgrace_gpu_map_and_read(struct nvgrace_gpu_vfio_pci_core_device

*nvdev,

drivers/vfio/pci/nvgrace-gpu/main.c line 435

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +10 -0
+			 char __user *buf, size_t mem_count, loff_t *ppos)
+{
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
+	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
+	int ret;
+
+	/*
+	 * Handle read on the BAR regions. Map to the target device memory
+	 * physical address and copy to the request read buffer.
+	 */

duplicate with the earlier comment for the function.

Quoted 8 lines

+/*
+ * Read count bytes from the device memory at an offset. The actual device
+ * memory size (available) may not be a power-of-2. So the driver fakes
+ * the size to a power-of-2 (reported) when exposing to a user space driver.
+ *
+ * Reads extending beyond the reported size are truncated; reads starting
+ * beyond the reported size generate -EINVAL; reads extending beyond the
+ * actual device size is filled with ~0.

slightly clearer to order the description: read starting beyond reported size, then read extending beyond device size, and read extending beyond reported size.

Quoted 19 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +9 -0
+static int
+nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
+			      struct nvgrace_gpu_vfio_pci_core_device *nvdev,
+			      u64 memphys, u64 memlength)
+{
+	int ret = 0;
+
+	/*
+	 * The VM GPU device driver needs a non-cacheable region to

support

(quoted diff) +1 -0
+	 * the MIG feature. Since the device memory is mapped as NORMAL

cached,

(quoted diff) +2 -0
+	 * carve out a region from the end with a different NORMAL_NC
+	 * property (called as reserved memory and represented as resmem).

This

(quoted diff) +1 -0
+	 * region then is exposed as a 64b BAR (region 2 and 3) to the VM,

while

(quoted diff) +1 -0
+	 * exposing the rest (termed as usable memory and represented

using usemem)

drivers/vfio/pci/nvgrace-gpu/main.c line 746

Quoted 6 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +6 -0
+	 * as cacheable 64b BAR (region 4 and 5).
+	 *
+	 *               devmem (memlength)
+	 * |-------------------------------------------------|
+	 * |                                           |
+	 * usemem.phys/memphys                         resmem.phys

there is no usemem.phys and resmem.phys

drivers/vfio/pci/nvgrace-gpu/main.c line 755

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+	 */
+	nvdev->usemem.memphys = memphys;
+
+	/*
+	 * The device memory exposed to the VM is added to the kernel by

the

drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+	 * VM driver module in chunks of memory block size. Only the usable
+	 * memory (usemem) is added to the kernel for usage by the VM
+	 * workloads. Make the usable memory size memblock aligned.
+	 */

If memblock size is defined by hw spec then say so.

otherwise this sounds a broken contract if it's a guest-decided value.

drivers/vfio/pci/nvgrace-gpu/main.c line 760

Quoted 5 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +5 -0
+	if (check_sub_overflow(memlength, RESMEM_SIZE,
+			       &nvdev->usemem.memlength)) {
+		ret = -EOVERFLOW;
+		goto done;
+	}

does resmem require 1G-aligned?

if usemem.memlength becomes 0 then should return error too.

author maintainer
#17 lore raw

Thanks Kevin for the review. Comments inline.

Quoted 7 lines
Quoted 5 lines

Note that the usemem memory is added by the VM Nvidia device driver [5] to the VM kernel as memblocks. Hence make the usable memory size memblock
aligned.

Is memblock size defined in spec or purely a guest implementation choice?

The MEMBLOCK value is a hardwired and a constant ABI value between the GPU FW and VFIO driver.

Quoted 11 lines
Quoted 3 lines

If the bare metal properties are not present, the driver registers the vfio-pci-core function pointers.

so if qemu doesn't generate such property the variant driver running inside guest will always go to use core functions and guest vfio userspace will observe both resmem and usemem bars. But then there is nothing in field to prohibit mapping resmem bar as cacheable.

should this driver check the presence of either ACPI property or resmem/usemem bars to enable variant function pointers?

Maybe I am missing something here; but if the ACPI property is absent, the real physical BARs present on the device will be exposed by the vfio-pci-core functions to the VM. So I think if the variant driver is ran within the VM, it should not see the fake usemem and resmem BARs.

Quoted 10 lines
Quoted 8 lines

+config NVGRACE_GPU_VFIO_PCI
+     tristate "VFIO support for the GPU in the NVIDIA Grace Hopper Superchip"
+     depends on ARM64 || (COMPILE_TEST && 64BIT)
+     select VFIO_PCI_CORE
+     help
+       VFIO support for the GPU in the NVIDIA Grace Hopper Superchip is +       required to assign the GPU device using KVM/qemu/etc.

"assign the GPU device to userspace"

Ack, will make the change.

Quoted 6 lines

drivers/vfio/pci/nvgrace-gpu/main.c line 21

Quoted 4 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +4 -0
+
+/* Memory size expected as non cached and reserved by the VM driver */
+#define RESMEM_SIZE 0x40000000
+#define MEMBLK_SIZE 0x20000000

also add a comment for MEMBLK_SIZE

Yes.

Quoted 5 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+struct nvgrace_gpu_vfio_pci_core_device {

will nvgrace refer to a non-gpu device? if not probably all prefixes with 'nvgrace_gpu' can be simplified to 'nvgrace'.

nvgrace_gpu indicates the GPU associated with grace CPU here. That is a one of the reason behind keeping the nomenclature as nvgrace_gpu. Calling it nvgrace may not be apt as it is a CPU.

Quoted 1 line

btw following other variant drivers 'vfio' can be removed too.

Ack.

Quoted 7 lines
Quoted 5 lines
drivers/vfio/pci/nvgrace-gpu/main.c new file +3 -0
+
+/*
+ * Both the usable (usemem) and the reserved (resmem) device memory

region
+ * are 64b fake BARs in the VM. These fake BARs must respond

s/VM/device/

I can make it clearer to something like "64b fake device BARs in the VM".

Quoted 7 lines
Quoted 5 lines

+     int ret;
+
+     ret = vfio_pci_core_read(core_vdev, buf, count, ppos); +     if (ret < 0)
+             return ret;

here if core_read succeeds *ppos has been updated...

Thanks for pointing that out, will fix the code handling *ppos.

Quoted 10 lines
Quoted 10 lines

+ * Read the data from the device memory (mapped either through ioremap
+ * or memremap) into the user buffer.
+ */

(quoted diff) +2 -0
+static int
+nvgrace_gpu_map_and_read(struct nvgrace_gpu_vfio_pci_core_device

*nvdev,
+                      char __user *buf, size_t mem_count, loff_t *ppos) +{
+     unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos); +     u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;

Quoted 8 lines
Quoted 6 lines

+     int ret;
+
+     /*
+      * Handle read on the BAR regions. Map to the target device memory +      * physical address and copy to the request read buffer. +      */

duplicate with the earlier comment for the function.

Ack.

Quoted 9 lines
Quoted 8 lines

+/*
+ * Read count bytes from the device memory at an offset. The actual device
+ * memory size (available) may not be a power-of-2. So the driver fakes
+ * the size to a power-of-2 (reported) when exposing to a user space driver.
+ *
+ * Reads extending beyond the reported size are truncated; reads starting
+ * beyond the reported size generate -EINVAL; reads extending beyond the
+ * actual device size is filled with ~0.

Quoted 3 lines

slightly clearer to order the description: read starting beyond reported size, then read extending beyond device size, and read extending beyond reported size.

Yes, will make the change.

Quoted 10 lines
Quoted 8 lines

+      * exposing the rest (termed as usable memory and represented using usemem)
+      * as cacheable 64b BAR (region 4 and 5).
+      *
+      *               devmem (memlength)
+      * |-------------------------------------------------| +      * |                                           |
+      * usemem.phys/memphys                         resmem.phys

there is no usemem.phys and resmem.phys

Silly miss. Will fix that.

Quoted 11 lines
Quoted 10 lines

+      */
+     nvdev->usemem.memphys = memphys;
+
+     /*
+      * The device memory exposed to the VM is added to the kernel by the
+      * VM driver module in chunks of memory block size. Only the usable +      * memory (usemem) is added to the kernel for usage by the VM +      * workloads. Make the usable memory size memblock aligned. +      */

Quoted 2 lines

If memblock size is defined by hw spec then say so. otherwise this sounds a broken contract if it's a guest-decided value.

Answered above. Will update the comment to mention that.

Quoted 7 lines
Quoted 5 lines

+     if (check_sub_overflow(memlength, RESMEM_SIZE, +                            &nvdev->usemem.memlength)) { +             ret = -EOVERFLOW;
+             goto done;
+     }

does resmem require 1G-aligned?

No, the requirement is it to be 1G+. No alignment restrictions there.

Quoted 1 line

if usemem.memlength becomes 0 then should return error too.

I suppose you mean if it becomes 0 after the subtraction above. We are checking for the 0 size in the host ACPI table otherwise.

#18 lore raw
Quoted 22 lines · On Fri, 9 Feb 2024 09:20:22 +0000 Ankit Agrawal <[email protected]> wrote:

Thanks Kevin for the review. Comments inline.

Quoted 7 lines
Quoted 5 lines

Note that the usemem memory is added by the VM Nvidia device driver [5] to the VM kernel as memblocks. Hence make the usable memory size memblock
aligned.

Is memblock size defined in spec or purely a guest implementation choice?

The MEMBLOCK value is a hardwired and a constant ABI value between the GPU FW and VFIO driver.

Quoted 9 lines
Quoted 3 lines

If the bare metal properties are not present, the driver registers the vfio-pci-core function pointers.

so if qemu doesn't generate such property the variant driver running inside guest will always go to use core functions and guest vfio userspace will observe both resmem and usemem bars. But then there is nothing in field to prohibit mapping resmem bar as cacheable.

Quoted 7 lines
Quoted 2 lines

should this driver check the presence of either ACPI property or resmem/usemem bars to enable variant function pointers?

Maybe I am missing something here; but if the ACPI property is absent, the real physical BARs present on the device will be exposed by the vfio-pci-core functions to the VM. So I think if the variant driver is ran within the VM, it should not see the fake usemem and resmem BARs.

There are two possibilities here, either we're assigning the pure physical device from a host that does not have the ACPI properties or we're performing a nested assignment. In the former case we're simply passing along the unmodified physical BARs. In the latter case we're actually passing through the fake BARs, the virtualization of the device has already happened in the level 1 assignment.

I think Kevin's point is also relative to this latter scenario, in the L1 instance of the nvgrace-gpu driver the mmap of the usemem BAR is cachable, but in the L2 instance of the driver where we only use the vfio-pci-core ops nothing maintains that cachable mapping. Is that a problem? An uncached mapping on top of a cachable mapping is often prone to problems. Thanks,

Alex

#19 lore raw
Quoted 6 lines · Alex Williamson

I think Kevin's point is also relative to this latter scenario, in the L1 instance of the nvgrace-gpu driver the mmap of the usemem BAR is cachable, but in the L2 instance of the driver where we only use the vfio-pci-core ops nothing maintains that cachable mapping. Is that a problem? An uncached mapping on top of a cachable mapping is often prone to problems.

On these CPUs the ARM architecture won't permit it, the L0 level blocks uncachable using FWB and page table attributes. The VM, no matter what it does, cannot make the cachable memory uncachable.

Jason

#20 lore raw
Quoted 11 lines · On Fri, 9 Feb 2024 13:19:03 -0400 Jason Gunthorpe <[email protected]> wrote:
Quoted 6 lines · Alex Williamson

I think Kevin's point is also relative to this latter scenario, in the L1 instance of the nvgrace-gpu driver the mmap of the usemem BAR is cachable, but in the L2 instance of the driver where we only use the vfio-pci-core ops nothing maintains that cachable mapping. Is that a problem? An uncached mapping on top of a cachable mapping is often prone to problems.

On these CPUs the ARM architecture won't permit it, the L0 level blocks uncachable using FWB and page table attributes. The VM, no matter what it does, cannot make the cachable memory uncachable.

Great, thanks,

Alex