Skip to contentSkip to conversation

vfio/nvgrace-gpu: Enable grace blackwell boards

PATCH Superseded [email protected] 2 participants 8 messages Jan 5, 2025 – Jan 9, 2025

Subsystems (4)

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

v2 cover vfio/nvgrace-gpu: Enable grace blackwell boards

From: Ankit Agrawal <[email protected]>

NVIDIA's recently introduced Grace Blackwell (GB) Superchip in continuation with the Grace Hopper (GH) superchip that provides a cache coherent access to CPU and GPU to each other's memory with an internal proprietary chip-to-chip (C2C) cache coherent interconnect. The in-tree nvgrace-gpu driver manages the GH devices. The intention is to extend the support to the new Grace Blackwell boards.

There is a HW defect on GH to support the Multi-Instance GPU (MIG) feature [1] that necessiated the presence of a 1G carved out from the device memory and mapped uncached. The 1G region is shown as a fake BAR (comprising region 2 and 3) to workaround the issue.

The GB systems differ from GH systems in the following aspects.
1. The aforementioned HW defect is fixed on GB systems.
2. There is a usable BAR1 (region 2 and 3) on GB systems for the GPUdirect RDMA feature [2].

This patch series accommodate those GB changes by showing the real physical device BAR1 (region2 and 3) to the VM instead of the fake one. This takes care of both the differences.

The presence of the fix for the HW defect is communicated by the firmware through a DVSEC PCI config register. The module reads this to take a different codepath on GB vs GH.

To improve system bootup time, HBM training is moved out of UEFI in GB system. Poll for the register indicating the training state. Also check the C2C link status if it is ready. Fail the probe if either fails.

Applied over next-20241220 and the required KVM patch (under review on the mailing list) to map the GPU device memory as cacheable [3]. Tested on the Grace Blackwell platform by booting up VM, loading NVIDIA module [4] and running nvidia-smi in the VM.

To run CUDA workloads, there is a dependency on the IOMMUFD and the Nested Page Table patches being worked on separately by Nicolin Chen. ([email protected]). NVIDIA has provided git repositories which includes all the requisite kernel [5] and Qemu [6] patches in case one wants to try.

v1 -> v2
* Rebased to next-20241220.

v1:

Ankit Agrawal (3):
vfio/nvgrace-gpu: Read dvsec register to determine need for uncached
resmem
vfio/nvgrace-gpu: Expose the blackwell device PF BAR1 to the VM vfio/nvgrace-gpu: Check the HBM training and C2C link status

 drivers/vfio/pci/nvgrace-gpu/main.c | 115 ++++++++++++++++++++++++++--
 1 file changed, 107 insertions(+), 8 deletions(-)
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH v2 1/3 vfio/nvgrace-gpu: Read dvsec register to determine need for uncached resmem

From: Ankit Agrawal <[email protected]>

NVIDIA's recently introduced Grace Blackwell (GB) Superchip is a continuation with the Grace Hopper (GH) superchip that provides a cache coherent access to CPU and GPU to each other's memory with an internal proprietary chip-to-chip cache coherent interconnect.

There is a HW defect on GH systems to support the Multi-Instance GPU (MIG) feature [1] that necessiated the presence of a 1G region with uncached mapping carved out from the device memory. The 1G region is shown as a fake BAR (comprising region 2 and 3) to workaround the issue. This is fixed on the GB systems.

The presence of the fix for the HW defect is communicated by the device firmware through the DVSEC PCI config register with ID 3. The module reads this to take a different codepath on GB vs GH.

Scan through the DVSEC registers to identify the correct one and use it to determine the presence of the fix. Save the value in the device's nvgrace_gpu_pci_core_device structure.

---
 drivers/vfio/pci/nvgrace-gpu/main.c | 30 +++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
drivers/vfio/pci/nvgrace-gpu/main.c +30 -0
index a467085038f0..85eacafaffdf 100644
@@ -23,6 +23,11 @@
 /* A hardwired and constant ABI value between the GPU FW and VFIO driver. */
 #define MEMBLK_SIZE SZ_512M
 
+#define DVSEC_BITMAP_OFFSET 0xA
+#define MIG_SUPPORTED_WITH_CACHED_RESMEM BIT(0)
+
+#define GPU_CAP_DVSEC_REGISTER 3
+
 /*
  * The state of the two device memory region - resmem and usemem - is
  * saved as struct mem_region.
@@ -46,6 +51,7 @@ struct nvgrace_gpu_pci_core_device {
 	struct mem_region resmem;
 	/* Lock to control device memory kernel mapping */
 	struct mutex remap_lock;
+	bool has_mig_hw_bug_fix;
 };
 
 static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device *core_vdev)
@@ -812,6 +818,26 @@ nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
 	return ret;
 }
 
+static bool nvgrace_gpu_has_mig_hw_bug_fix(struct pci_dev *pdev)
+{
+	int pcie_dvsec;
+	u16 dvsec_ctrl16;
+
+	pcie_dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_NVIDIA,
+					       GPU_CAP_DVSEC_REGISTER);
+
+	if (pcie_dvsec) {
+		pci_read_config_word(pdev,
+				     pcie_dvsec + DVSEC_BITMAP_OFFSET,
+				     &dvsec_ctrl16);
+
+		if (dvsec_ctrl16 & MIG_SUPPORTED_WITH_CACHED_RESMEM)
+			return true;
+	}
+
+	return false;
+}
+
 static int nvgrace_gpu_probe(struct pci_dev *pdev,
 			     const struct pci_device_id *id)
 {
@@ -832,6 +858,8 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev,
 	dev_set_drvdata(&pdev->dev, &nvdev->core_device);
 
 	if (ops == &nvgrace_gpu_pci_ops) {
+		nvdev->has_mig_hw_bug_fix = nvgrace_gpu_has_mig_hw_bug_fix(pdev);
+
 		/*
 		 * Device memory properties are identified in the host ACPI
 		 * table. Set the nvgrace_gpu_pci_core_device structure.
@@ -868,6 +896,8 @@ static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = {
 	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA, 0x2345) },
 	/* GH200 SKU */
 	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA, 0x2348) },
+	/* GB200 SKU */
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA, 0x2941) },
 	{}
 };
 
Signature · 1 line
2.34.1
author maintainer
#2 lore raw

PATCH v2 2/3 vfio/nvgrace-gpu: Expose the blackwell device PF BAR1 to the VM

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

There is a HW defect on Grace Hopper (GH) to support the Multi-Instance GPU (MIG) feature [1] that necessiated the presence of a 1G region carved out from the device memory and mapped as uncached. The 1G region is shown as a fake BAR (comprising region 2 and 3) to workaround the issue.

The Grace Blackwell systems (GB) differ from GH systems in the following aspects:
1. The aforementioned HW defect is fixed on GB systems.
2. There is a usable BAR1 (region 2 and 3) on GB systems for the GPUdirect RDMA feature [2].

This patch accommodate those GB changes by showing the 64b physical device BAR1 (region2 and 3) to the VM instead of the fake one. This takes care of both the differences.

Moreover, the entire device memory is exposed on GB as cacheable to the VM as there is no carveout required.

---
 drivers/vfio/pci/nvgrace-gpu/main.c | 32 +++++++++++++++++++++--------
 1 file changed, 24 insertions(+), 8 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +24 -8
index 85eacafaffdf..44a276c886e1 100644
@@ -72,7 +72,7 @@ nvgrace_gpu_memregion(int index,
 	if (index == USEMEM_REGION_INDEX)
 		return &nvdev->usemem;
 
-	if (index == RESMEM_REGION_INDEX)
+	if (!nvdev->has_mig_hw_bug_fix && index == RESMEM_REGION_INDEX)
 		return &nvdev->resmem;
 
 	return NULL;
@@ -715,6 +715,16 @@ static const struct vfio_device_ops nvgrace_gpu_pci_core_ops = {
 	.detach_ioas	= vfio_iommufd_physical_detach_ioas,
 };
 
+static void
+nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
+			      struct nvgrace_gpu_pci_core_device *nvdev,
+			      u64 memphys, u64 memlength)
+{
+	nvdev->usemem.memphys = memphys;
+	nvdev->usemem.memlength = memlength;
+	nvdev->usemem.bar_size = roundup_pow_of_two(nvdev->usemem.memlength);
+}
+
 static int
 nvgrace_gpu_fetch_memory_property(struct pci_dev *pdev,
 				  u64 *pmemphys, u64 *pmemlength)
@@ -752,9 +762,9 @@ nvgrace_gpu_fetch_memory_property(struct pci_dev *pdev,
 }
 
 static int
-nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
-			      struct nvgrace_gpu_pci_core_device *nvdev,
-			      u64 memphys, u64 memlength)
+nvgrace_gpu_nvdev_struct_workaround(struct pci_dev *pdev,
+				    struct nvgrace_gpu_pci_core_device *nvdev,
+				    u64 memphys, u64 memlength)
 {
 	int ret = 0;
 
@@ -864,10 +874,16 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev,
 		 * Device memory properties are identified in the host ACPI
 		 * table. Set the nvgrace_gpu_pci_core_device structure.
 		 */
-		ret = nvgrace_gpu_init_nvdev_struct(pdev, nvdev,
-						    memphys, memlength);
-		if (ret)
-			goto out_put_vdev;
+		if (nvdev->has_mig_hw_bug_fix) {
+			nvgrace_gpu_init_nvdev_struct(pdev, nvdev,
+						      memphys, memlength);
+		} else {
+			ret = nvgrace_gpu_nvdev_struct_workaround(pdev, nvdev,
+								  memphys,
+								  memlength);
+			if (ret)
+				goto out_put_vdev;
+		}
 	}
 
 	ret = vfio_pci_core_register_device(&nvdev->core_device);
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH v2 2/3] vfio/nvgrace-gpu: Expose the blackwell device PF BAR1 to the VM

Quoted 80 lines · On Sun, 5 Jan 2025 17:36:14 +0000 <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

There is a HW defect on Grace Hopper (GH) to support the Multi-Instance GPU (MIG) feature [1] that necessiated the presence of a 1G region carved out from the device memory and mapped as uncached. The 1G region is shown as a fake BAR (comprising region 2 and 3) to workaround the issue.

The Grace Blackwell systems (GB) differ from GH systems in the following aspects:
1. The aforementioned HW defect is fixed on GB systems.
2. There is a usable BAR1 (region 2 and 3) on GB systems for the GPUdirect RDMA feature [2].

This patch accommodate those GB changes by showing the 64b physical device BAR1 (region2 and 3) to the VM instead of the fake one. This takes care of both the differences.

Moreover, the entire device memory is exposed on GB as cacheable to the VM as there is no carveout required.

---
 drivers/vfio/pci/nvgrace-gpu/main.c | 32 +++++++++++++++++++++--------
 1 file changed, 24 insertions(+), 8 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +14 -8
index 85eacafaffdf..44a276c886e1 100644
@@ -72,7 +72,7 @@ nvgrace_gpu_memregion(int index,
 	if (index == USEMEM_REGION_INDEX)
 		return &nvdev->usemem;
 
-	if (index == RESMEM_REGION_INDEX)
+	if (!nvdev->has_mig_hw_bug_fix && index == RESMEM_REGION_INDEX)
 		return &nvdev->resmem;
 
 	return NULL;
@@ -715,6 +715,16 @@ static const struct vfio_device_ops nvgrace_gpu_pci_core_ops = {
 	.detach_ioas	= vfio_iommufd_physical_detach_ioas,
 };
 
+static void
+nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
+			      struct nvgrace_gpu_pci_core_device *nvdev,
+			      u64 memphys, u64 memlength)
+{
+	nvdev->usemem.memphys = memphys;
+	nvdev->usemem.memlength = memlength;
+	nvdev->usemem.bar_size = roundup_pow_of_two(nvdev->usemem.memlength);
+}
+
 static int
 nvgrace_gpu_fetch_memory_property(struct pci_dev *pdev,
 				  u64 *pmemphys, u64 *pmemlength)
@@ -752,9 +762,9 @@ nvgrace_gpu_fetch_memory_property(struct pci_dev *pdev,
 }
 
 static int
-nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
-			      struct nvgrace_gpu_pci_core_device *nvdev,
-			      u64 memphys, u64 memlength)
+nvgrace_gpu_nvdev_struct_workaround(struct pci_dev *pdev,
+				    struct nvgrace_gpu_pci_core_device *nvdev,
+				    u64 memphys, u64 memlength)
 {
 	int ret = 0;
 
@@ -864,10 +874,16 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev,
 		 * Device memory properties are identified in the host ACPI
 		 * table. Set the nvgrace_gpu_pci_core_device structure.
 		 */
-		ret = nvgrace_gpu_init_nvdev_struct(pdev, nvdev,
-						    memphys, memlength);
-		if (ret)
-			goto out_put_vdev;

drivers/vfio/pci/nvgrace-gpu/main.c static int nvgrace_gpu_probe(struct pci_dev *pdev, · line 886

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c +10 -0
+		if (nvdev->has_mig_hw_bug_fix) {
+			nvgrace_gpu_init_nvdev_struct(pdev, nvdev,
+						      memphys, memlength);
+		} else {
+			ret = nvgrace_gpu_nvdev_struct_workaround(pdev, nvdev,
+								  memphys,
+								  memlength);
+			if (ret)
+				goto out_put_vdev;
+		}

Doesn't this work out much more naturally if we just do something like:

(quoted diff) +0 -0

b/drivers/vfio/pci/nvgrace-gpu/main.c index 85eacafaffdf..43a9457442ff 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c
+++ b/drivers/vfio/pci/nvgrace-gpu/main.c
@@ -17,9 +17,6 @@
#define RESMEM_REGION_INDEX VFIO_PCI_BAR2_REGION_INDEX #define USEMEM_REGION_INDEX VFIO_PCI_BAR4_REGION_INDEX

-/* Memory size expected as non cached and reserved by the VM driver */ -#define RESMEM_SIZE SZ_1G
-
/* A hardwired and constant ABI value between the GPU FW and VFIO
driver. */ #define MEMBLK_SIZE SZ_512M

@@ -72,7 +69,7 @@ nvgrace_gpu_memregion(int index,
if (index == USEMEM_REGION_INDEX)
return &nvdev->usemem;

- if (index == RESMEM_REGION_INDEX)
+ if (nvdev->resmem.memlength && index == RESMEM_REGION_INDEX)
return &nvdev->resmem;

return NULL;
@@ -757,6 +754,13 @@ nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
u64 memphys, u64 memlength)
{
int ret = 0;
+ u64 resmem_size = 0;
+
+ /*
+ * Comment about the GH bug that requires this and fix in GB
+ */
+ if (!nvdev->has_mig_hw_bug_fix)
+ resmem_size = SZ_1G;

/*
* The VM GPU device driver needs a non-cacheable region to
support @@ -780,7 +784,7 @@ nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
* memory (usemem) is added to the kernel for usage by the VM
* workloads. Make the usable memory size memblock aligned. */
- if (check_sub_overflow(memlength, RESMEM_SIZE,
+ if (check_sub_overflow(memlength, resmem_size,
&nvdev->usemem.memlength)) {
ret = -EOVERFLOW;
goto done;
@@ -813,7 +817,9 @@ nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
* the BAR size for them.
*/
nvdev->usemem.bar_size =
roundup_pow_of_two(nvdev->usemem.memlength);
- nvdev->resmem.bar_size =
roundup_pow_of_two(nvdev->resmem.memlength);
+ if (nvdev->resmem.memlength)
+ nvdev->resmem.bar_size =
+ roundup_pow_of_two(nvdev->resmem.memlength);
done:
return ret;
}

Thanks,
Alex

author maintainer
#4 lore raw

Re: [PATCH v2 2/3] vfio/nvgrace-gpu: Expose the blackwell device PF BAR1 to the VM

Quoted 54 lines

Doesn't this work out much more naturally if we just do something like:

(quoted diff) +0 -0

b/drivers/vfio/pci/nvgrace-gpu/main.c index 85eacafaffdf..43a9457442ff 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c
+++ b/drivers/vfio/pci/nvgrace-gpu/main.c

(quoted diff) +0 -3
@@ -17,9 +17,6 @@
 #define RESMEM_REGION_INDEX VFIO_PCI_BAR2_REGION_INDEX
 #define USEMEM_REGION_INDEX VFIO_PCI_BAR4_REGION_INDEX
 
-/* Memory size expected as non cached and reserved by the VM driver */
-#define RESMEM_SIZE SZ_1G
-
 /* A hardwired and constant ABI value between the GPU FW and VFIO

driver. */ #define MEMBLK_SIZE SZ_512M

(quoted diff) +8 -1
@@ -72,7 +69,7 @@ nvgrace_gpu_memregion(int index,
         if (index == USEMEM_REGION_INDEX)
                 return &nvdev->usemem;
 
-       if (index == RESMEM_REGION_INDEX)
+       if (nvdev->resmem.memlength && index == RESMEM_REGION_INDEX)
                 return &nvdev->resmem;
 
         return NULL;
@@ -757,6 +754,13 @@ nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
                               u64 memphys, u64 memlength)
 {
         int ret = 0;
+       u64 resmem_size = 0;
+
+       /*
+        * Comment about the GH bug that requires this and fix in GB
+        */
+       if (!nvdev->has_mig_hw_bug_fix)
+               resmem_size = SZ_1G;
 
         /*
          * The VM GPU device driver needs a non-cacheable region to

support @@ -780,7 +784,7 @@ nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
         * memory (usemem) is added to the kernel for usage by the VM
         * workloads. Make the usable memory size memblock aligned. */
-       if (check_sub_overflow(memlength, RESMEM_SIZE,
+       if (check_sub_overflow(memlength, resmem_size, &nvdev->usemem.memlength)) { ret = -EOVERFLOW;
                goto done;

(quoted diff) +0 -0
@@ -813,7 +817,9 @@ nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev,
          * the BAR size for them.
          */

        nvdev->usemem.bar_size =
roundup_pow_of_two(nvdev->usemem.memlength);

Quoted 9 lines

-       nvdev->resmem.bar_size = roundup_pow_of_two(nvdev->resmem.memlength);
+       if (nvdev->resmem.memlength)
+               nvdev->resmem.bar_size =
+                       roundup_pow_of_two(nvdev->resmem.memlength); done:
        return ret;
 }

Thanks Alex, you suggestion does looks simpler and better.

I'll test that out and send out an updated version of the patch.

author maintainer
#5 lore raw

PATCH v2 3/3 vfio/nvgrace-gpu: Check the HBM training and C2C link status

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

In contrast to Grace Hopper systems, the HBM training has been moved out of the UEFI on the Grace Blackwell systems. This reduces the system bootup time significantly.

The onus of checking whether the HBM training has completed thus falls on the module.

The HBM training status can be determined from a BAR0 register. Similarly, another BAR0 register exposes the status of the CPU-GPU chip-to-chip (C2C) cache coherent interconnect.

Based on testing, 30s is determined to be sufficient to ensure initialization completion on all the Grace based systems. Thus poll these register and check for 30s. If the HBM training is not complete or if the C2C link is not ready, fail the probe.

While the time is not required on Grace Hopper systems, it is beneficial to make the check to ensure the device is in an expected state. Hence keeping it generalized to both the generations.

---
 drivers/vfio/pci/nvgrace-gpu/main.c | 53 +++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)
drivers/vfio/pci/nvgrace-gpu/main.c +53 -0
index 44a276c886e1..cf020496743e 100644
@@ -5,6 +5,7 @@
 
 #include <linux/sizes.h>
 #include <linux/vfio_pci_core.h>
+#include <linux/delay.h>
 
 /*
  * The device memory usable to the workloads running in the VM is cached
@@ -28,6 +29,13 @@
 
 #define GPU_CAP_DVSEC_REGISTER 3
 
+#define C2C_LINK_BAR0_OFFSET 0x1498
+#define HBM_TRAINING_BAR0_OFFSET 0x200BC
+#define STATUS_READY 0xFF
+
+#define POLL_QUANTUM_MS 1000
+#define POLL_TIMEOUT_MS (30 * 1000)
+
 /*
  * The state of the two device memory region - resmem and usemem - is
  * saved as struct mem_region.
@@ -848,6 +856,47 @@ static bool nvgrace_gpu_has_mig_hw_bug_fix(struct pci_dev *pdev)
 	return false;
 }
 
+/*
+ * To reduce the system bootup time, the HBM training has
+ * been moved out of the UEFI on the Grace-Blackwell systems.
+ *
+ * The onus of checking whether the HBM training has completed
+ * thus falls on the module. The HBM training status can be
+ * determined from a BAR0 register.
+ *
+ * Similarly, another BAR0 register exposes the status of the
+ * CPU-GPU chip-to-chip (C2C) cache coherent interconnect.
+ *
+ * Poll these register and check for 30s. If the HBM training is
+ * not complete or if the C2C link is not ready, fail the probe.
+ *
+ * While the wait is not required on Grace Hopper systems, it
+ * is beneficial to make the check to ensure the device is in an
+ * expected state.
+ */
+static int nvgrace_gpu_check_device_status(struct pci_dev *pdev)
+{
+	void __iomem *io;
+	int time_elasped;
+
+	io = pci_iomap(pdev, 0, ~0UL);
+	if (!io)
+		return -ENOMEM;
+
+	for (time_elasped = 0; time_elasped < POLL_TIMEOUT_MS;
+	     time_elasped += POLL_QUANTUM_MS) {
+		if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) &&
+		    (ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY)) {
+			pci_iounmap(pdev, io);
+			return 0;
+		}
+		msleep(POLL_QUANTUM_MS);
+	}
+
+	pci_iounmap(pdev, io);
+	return -ENODEV;
+}
+
 static int nvgrace_gpu_probe(struct pci_dev *pdev,
 			     const struct pci_device_id *id)
 {
@@ -856,6 +905,10 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev,
 	u64 memphys, memlength;
 	int ret;
 
+	ret = nvgrace_gpu_check_device_status(pdev);
+	if (ret)
+		return ret;
+
 	ret = nvgrace_gpu_fetch_memory_property(pdev, &memphys, &memlength);
 	if (!ret)
 		ops = &nvgrace_gpu_pci_ops;
Signature · 1 line
2.34.1
#6 lore raw

Re: [PATCH v2 3/3] vfio/nvgrace-gpu: Check the HBM training and C2C link status

Quoted 66 lines · On Sun, 5 Jan 2025 17:36:15 +0000 <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

In contrast to Grace Hopper systems, the HBM training has been moved out of the UEFI on the Grace Blackwell systems. This reduces the system bootup time significantly.

The onus of checking whether the HBM training has completed thus falls on the module.

The HBM training status can be determined from a BAR0 register. Similarly, another BAR0 register exposes the status of the CPU-GPU chip-to-chip (C2C) cache coherent interconnect.

Based on testing, 30s is determined to be sufficient to ensure initialization completion on all the Grace based systems. Thus poll these register and check for 30s. If the HBM training is not complete or if the C2C link is not ready, fail the probe.

While the time is not required on Grace Hopper systems, it is beneficial to make the check to ensure the device is in an expected state. Hence keeping it generalized to both the generations.

---
 drivers/vfio/pci/nvgrace-gpu/main.c | 53 +++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)
drivers/vfio/pci/nvgrace-gpu/main.c +17 -0
index 44a276c886e1..cf020496743e 100644
@@ -5,6 +5,7 @@
 
 #include <linux/sizes.h>
 #include <linux/vfio_pci_core.h>
+#include <linux/delay.h>
 
 /*
  * The device memory usable to the workloads running in the VM is cached
@@ -28,6 +29,13 @@
 
 #define GPU_CAP_DVSEC_REGISTER 3
 
+#define C2C_LINK_BAR0_OFFSET 0x1498
+#define HBM_TRAINING_BAR0_OFFSET 0x200BC
+#define STATUS_READY 0xFF
+
+#define POLL_QUANTUM_MS 1000
+#define POLL_TIMEOUT_MS (30 * 1000)
+
 /*
  * The state of the two device memory region - resmem and usemem - is
  * saved as struct mem_region.
@@ -848,6 +856,47 @@ static bool nvgrace_gpu_has_mig_hw_bug_fix(struct pci_dev *pdev)
 	return false;
 }
 
+/*
+ * To reduce the system bootup time, the HBM training has
+ * been moved out of the UEFI on the Grace-Blackwell systems.
+ *
+ * The onus of checking whether the HBM training has completed
+ * thus falls on the module. The HBM training status can be
+ * determined from a BAR0 register.
+ *
+ * Similarly, another BAR0 register exposes the status of the

drivers/vfio/pci/nvgrace-gpu/main.c static bool nvgrace_gpu_has_mig_hw_bug_fix(struct pci_dev *pdev) · line 877

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c +10 -0
+ * CPU-GPU chip-to-chip (C2C) cache coherent interconnect.
+ *
+ * Poll these register and check for 30s. If the HBM training is
+ * not complete or if the C2C link is not ready, fail the probe.
+ *
+ * While the wait is not required on Grace Hopper systems, it
+ * is beneficial to make the check to ensure the device is in an
+ * expected state.
+ */
+static int nvgrace_gpu_check_device_status(struct pci_dev *pdev)

"nvgrace_gpu_wait_device_ready()"?

drivers/vfio/pci/nvgrace-gpu/main.c static bool nvgrace_gpu_has_mig_hw_bug_fix(struct pci_dev *pdev) · line 882

Quoted 5 lines
drivers/vfio/pci/nvgrace-gpu/main.c +5 -0
+{
+	void __iomem *io;
+	int time_elasped;
+
+	io = pci_iomap(pdev, 0, ~0UL);

The documentation is unclear here, but existing code suggests passing 0 here rather than -1 to map the full BAR. It ends up being equivalent since the code doesn't error attempting to map longer than the BAR, but there's no reason to add a bad example.

drivers/vfio/pci/nvgrace-gpu/main.c static bool nvgrace_gpu_has_mig_hw_bug_fix(struct pci_dev *pdev) · line 894

Quoted 12 lines
drivers/vfio/pci/nvgrace-gpu/main.c +12 -0
+	if (!io)
+		return -ENOMEM;
+
+	for (time_elasped = 0; time_elasped < POLL_TIMEOUT_MS;
+	     time_elasped += POLL_QUANTUM_MS) {
+		if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) &&
+		    (ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY)) {
+			pci_iounmap(pdev, io);
+			return 0;
+		}
+		msleep(POLL_QUANTUM_MS);
+	}

time_after() would simplify things here. I'd also suggest a common exit path.

drivers/vfio/pci/nvgrace-gpu/main.c static bool nvgrace_gpu_has_mig_hw_bug_fix(struct pci_dev *pdev) · line 897

Quoted 3 lines
drivers/vfio/pci/nvgrace-gpu/main.c +3 -0
+
+	pci_iounmap(pdev, io);
+	return -ENODEV;

ETIME could work for the error code too. Thanks,

Alex

Quoted 16 lines
drivers/vfio/pci/nvgrace-gpu/main.c +6 -0
+}
+
 static int nvgrace_gpu_probe(struct pci_dev *pdev,
 			     const struct pci_device_id *id)
 {
@@ -856,6 +905,10 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev,
 	u64 memphys, memlength;
 	int ret;
 
+	ret = nvgrace_gpu_check_device_status(pdev);
+	if (ret)
+		return ret;
+
 	ret = nvgrace_gpu_fetch_memory_property(pdev, &memphys, &memlength);
 	if (!ret)
 		ops = &nvgrace_gpu_pci_ops;
author maintainer
#7 lore raw

Re: [PATCH v2 3/3] vfio/nvgrace-gpu: Check the HBM training and C2C link status

Thanks Alex for the review.

Quoted 3 lines
Quoted 1 line

+static int nvgrace_gpu_check_device_status(struct pci_dev *pdev)

"nvgrace_gpu_wait_device_ready()"?

Ok, I'll rename it.

Quoted 10 lines

drivers/vfio/pci/nvgrace-gpu/main.c static bool nvgrace_gpu_has_mig_hw_bug_fix(struct pci_dev *pdev) · line 882

Quoted 5 lines
drivers/vfio/pci/nvgrace-gpu/main.c +5 -0
+{
+     void __iomem *io;
+     int time_elasped;
+
+     io = pci_iomap(pdev, 0, ~0UL);

The documentation is unclear here, but existing code suggests passing 0 here rather than -1 to map the full BAR.  It ends up being equivalent since the code doesn't error attempting to map longer than the BAR, but there's no reason to add a bad example.

Ack, will change to use 0.

Quoted 6 lines
Quoted 3 lines

+             }
+             msleep(POLL_QUANTUM_MS);
+     }

time_after() would simplify things here.  I'd also suggest a common exit path.

Understood, will change.

Quoted 5 lines
Quoted 3 lines

+
+     pci_iounmap(pdev, io);
+     return -ENODEV;

ETIME could work for the error code too.  Thanks,

Ack.