Skip to contentSkip to conversation

vfio/nvgrace-gpu: Support huge PFNMAP and wait for GPU ready post reset

PATCH Superseded [email protected] 4 participants 24 messages Nov 24, 2025 – Nov 25, 2025

Subsystems (4)

v5 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

v5 cover vfio/nvgrace-gpu: Support huge PFNMAP and wait for GPU ready post reset

From: Ankit Agrawal <[email protected]>

NVIDIA's Grace based system have large GPU device memory. The device memory is mapped as VM_PFNMAP in the VMM VMA. The nvgrace-gpu module could make use of the huge PFNMAP support added in mm [1].

To achieve this, nvgrace-gpu module is updated to implement huge_fault ops. The implementation establishes mapping according to the order request. Note that if the PFN or the VMA address is unaligned to the order, the mapping fallbacks to the PTE level.

Secondly, it is expected that the mapping not be re-established until the GPU is ready post reset. Presence of the mappings during that time could potentially leads to harmless corrected RAS events to be logged if the CPU attempts to do speculative reads on the GPU memory on the Grace systems.

It can take several seconds for the GPU to be ready. So it is desirable that the time overlaps as much of the VM startup as possible to reduce impact on the VM bootup time. The GPU readiness state is thus checked on the first fault/huge_fault request which amortizes the GPU readiness time. The GPU readiness is checked through BAR0 registers as is done at the device probe.

Patch 1 updates the mapping mechanism to be done through faults.

Patch 2 splits the code to map at the various levels.

Patch 3 implements support for huge pfnmap.

Patch 4 vfio_pci_core_mmap cleanup.

Patch 5 split the code to check the device readiness.

Patch 6 reset_done handler implementation

Patch 7 Ensures that the GPU is ready before re-establishing the mapping after reset.

Applied over 6.18-rc6.

Changelog:
v5:
- Updated gpu_mem_mapped with reset_done flag for clearer semantics. (6/7) (Thanks Alex Williamson)
- Renamed vfio_pci_map_pfn to vfio_pci_vmf_insert_pfn. (2/7) (Thanks Alex Williamson)
- Updated to hold memory_lock across the vmf_insert_pfn and the read/write access of the device. (7/7) (Thanks Alex Williamson)
- Used scoped_guard to simplify critical region. (1/7, 7/7) [v4]
- Implemented reset_done handler to set gpu_mem_mapped flag. Cleaned up FLR detection path (Thanks Alex Williamson)
- Moved the premap check of the device readiness to a new function. Added locking to avoid races. (Thanks Alex Williamson)
- vfio_pci_core_mmap cleanup.
- Added ioremap to BAR0 during open.

- Moved the code for BAR mapping to a separate function.
- Added BAR0 mapping during open. Ensures BAR0 is mapped when registers are checked. (Thanks Alex Williamson, Jason Gunthorpe for suggestion)
- Added check for GPU readiness on nvgrace_gpu_map_device_mem. (Thanks Alex Williamson for the suggestion.

- Fixed build kernel warning
- subject text changes
- Rebased to 6.18-rc6.

Ankit Agrawal (7):
vfio/nvgrace-gpu: Use faults to map device memory
vfio: export function to map the VMA vfio/nvgrace-gpu: Add support for huge pfnmap
vfio: use vfio_pci_core_setup_barmap to map bar in mmap vfio/nvgrace-gpu: split the code to wait for GPU ready vfio/nvgrace-gpu: Inform devmem unmapped after reset vfio/nvgrace-gpu: wait for the GPU mem to be ready

 drivers/vfio/pci/nvgrace-gpu/main.c | 216 ++++++++++++++++++++++------
 drivers/vfio/pci/vfio_pci_core.c    |  61 ++++----
 include/linux/vfio_pci_core.h       |   2 +
 3 files changed, 207 insertions(+), 72 deletions(-)
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH v5 1/7 vfio/nvgrace-gpu: Use faults to map device memory

From: Ankit Agrawal <[email protected]>

To make use of the huge pfnmap support and to support zap/remap sequence, fault/huge_fault ops based mapping mechanism needs to be implemented.

Currently nvgrace-gpu module relies on remap_pfn_range to do the mapping during VM bootup. Replace it to instead rely on fault and use vmf_insert_pfn to setup the mapping.

---
 drivers/vfio/pci/nvgrace-gpu/main.c | 50 +++++++++++++++++------------
 1 file changed, 30 insertions(+), 20 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +30 -20
index e346392b72f6..f74f3d8e1ebe 100644
@@ -130,6 +130,32 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)
 	vfio_pci_core_close_device(core_vdev);
 }
 
+static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+	struct nvgrace_gpu_pci_core_device *nvdev = vma->vm_private_data;
+	int index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
+	vm_fault_t ret = VM_FAULT_SIGBUS;
+	struct mem_region *memregion;
+	unsigned long pgoff, pfn;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return ret;
+
+	pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
+	pfn = PHYS_PFN(memregion->memphys) + pgoff;
+
+	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock)
+		ret = vmf_insert_pfn(vmf->vma, vmf->address, pfn);
+
+	return ret;
+}
+
+static const struct vm_operations_struct nvgrace_gpu_vfio_pci_mmap_ops = {
+	.fault = nvgrace_gpu_vfio_pci_fault,
+};
+
 static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
 			    struct vm_area_struct *vma)
 {
@@ -137,10 +163,8 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
 		container_of(core_vdev, struct nvgrace_gpu_pci_core_device,
 			     core_device.vdev);
 	struct mem_region *memregion;
-	unsigned long start_pfn;
 	u64 req_len, pgoff, end;
 	unsigned int index;
-	int ret = 0;
 
 	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
 
@@ -157,7 +181,6 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
 		((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
 
 	if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len) ||
-	    check_add_overflow(PHYS_PFN(memregion->memphys), pgoff, &start_pfn) ||
 	    check_add_overflow(PFN_PHYS(pgoff), req_len, &end))
 		return -EOVERFLOW;
 
@@ -168,6 +191,8 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
 	if (end > memregion->memlength)
 		return -EINVAL;
 
+	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
+
 	/*
 	 * The carved out region of the device memory needs the NORMAL_NC
 	 * property. Communicate as such to the hypervisor.
@@ -184,23 +209,8 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
 		vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
 	}
 
-	/*
-	 * Perform a PFN map to the memory and back the device BAR by the
-	 * GPU memory.
-	 *
-	 * The available GPU memory size may not be power-of-2 aligned. The
-	 * remainder is only backed by vfio_device_ops read/write handlers.
-	 *
-	 * During device reset, the GPU is safely disconnected to the CPU
-	 * and access to the BAR will be immediately returned preventing
-	 * machine check.
-	 */
-	ret = remap_pfn_range(vma, vma->vm_start, start_pfn,
-			      req_len, vma->vm_page_prot);
-	if (ret)
-		return ret;
-
-	vma->vm_pgoff = start_pfn;
+	vma->vm_ops = &nvgrace_gpu_vfio_pci_mmap_ops;
+	vma->vm_private_data = nvdev;
 
 	return 0;
 }
Signature · 1 line
2.34.1
#2 lore raw

RE: [PATCH v5 1/7] vfio/nvgrace-gpu: Use faults to map device memory

Quoted 51 lines

-----Original Message-----
From: Ankit Agrawal <[email protected]>
Sent: 24 November 2025 11:59
To: Ankit Agrawal <[email protected]>; [email protected]; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; [email protected]; Aniket Agashe <[email protected]>; Vikram Sethi <[email protected]>; Matt Ochs <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Avihai Horon <[email protected]>; [email protected]; [email protected]; [email protected]; Alistair Popple <[email protected]>; [email protected]; linux- [email protected]; Neo Jia <[email protected]>; Kirti Wankhede <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Dan Williams <[email protected]>; Dheeraj Nigam <[email protected]>; Krishnakant Jaju <[email protected]>
Subject: [PATCH v5 1/7] vfio/nvgrace-gpu: Use faults to map device memory

From: Ankit Agrawal <[email protected]>

To make use of the huge pfnmap support and to support zap/remap sequence, fault/huge_fault ops based mapping mechanism needs to be implemented.

Currently nvgrace-gpu module relies on remap_pfn_range to do the mapping during VM bootup. Replace it to instead rely on fault and use vmf_insert_pfn to setup the mapping.

---
 drivers/vfio/pci/nvgrace-gpu/main.c | 50 +++++++++++++++++------------
 1 file changed, 30 insertions(+), 20 deletions(-)
drivers/vfio/pci/nvgrace- renamed from drivers/vfio/pci/nvgrace-gpu/main.c +0 -0

gpu/main.c
index e346392b72f6..f74f3d8e1ebe 100644

drivers/vfio/pci/nvgrace-gpu/main.c +5 -0
@@ -130,6 +130,32 @@ static void nvgrace_gpu_close_device(struct
 vfio_device *core_vdev)
 	vfio_pci_core_close_device(core_vdev);
 }
 
+static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+	struct nvgrace_gpu_pci_core_device *nvdev = vma->vm_private_data;
+	int index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT -
 PAGE_SHIFT);

drivers/vfio/pci/nvgrace-gpu/main.c static void nvgrace_gpu_close_device(struct · line 149

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c +10 -0
+	vm_fault_t ret = VM_FAULT_SIGBUS;
+	struct mem_region *memregion;
+	unsigned long pgoff, pfn;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return ret;
+
+	pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
+	pfn = PHYS_PFN(memregion->memphys) + pgoff;

The core fault code seems to calculate the BAR offset in vma_to_pfn() which is missing here.

pgoff = vma->vm_pgoff &

                ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);

Is the assumption here is user space will always map at BAR offset 0?

drivers/vfio/pci/nvgrace-gpu/main.c static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) · line 152

Quoted 5 lines
drivers/vfio/pci/nvgrace-gpu/main.c +5 -0
+
+	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock)
+		ret = vmf_insert_pfn(vmf->vma, vmf->address, pfn);
+
+	return ret;

Could do return vmf_insert_pfn(vmf->vma, vmf->address, pfn); if you don't need it later.

Thanks,
Shameer

#3 lore raw

Re: [PATCH v5 1/7] vfio/nvgrace-gpu: Use faults to map device memory

Quoted 20 lines · Shameer Kolothum
Quoted 6 lines
drivers/vfio/pci/nvgrace-gpu/main.c +5 -0
+static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+	struct nvgrace_gpu_pci_core_device *nvdev = vma->vm_private_data;
+	int index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT -

PAGE_SHIFT);

drivers/vfio/pci/nvgrace-gpu/main.c static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) · line 147

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c +10 -0
+	vm_fault_t ret = VM_FAULT_SIGBUS;
+	struct mem_region *memregion;
+	unsigned long pgoff, pfn;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return ret;
+
+	pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
+	pfn = PHYS_PFN(memregion->memphys) + pgoff;

The core fault code seems to calculate the BAR offset in vma_to_pfn() which is missing here.

Quoted 2 lines

pgoff = vma->vm_pgoff &

                ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);

Yes, that should be included for this reason:

Quoted 1 line

Is the assumption here is user space will always map at BAR offset 0?

It should not be assumed.

Jason

#4 lore raw

Re: [PATCH v5 1/7] vfio/nvgrace-gpu: Use faults to map device memory

In reply to [email protected]
Quoted 20 lines · On Mon, Nov 24, 2025 at 11:59:20AM +0000, [email protected] wrote:

From: Ankit Agrawal <[email protected]>

To make use of the huge pfnmap support and to support zap/remap sequence, fault/huge_fault ops based mapping mechanism needs to be implemented.

Currently nvgrace-gpu module relies on remap_pfn_range to do the mapping during VM bootup. Replace it to instead rely on fault and use vmf_insert_pfn to setup the mapping.

---
 drivers/vfio/pci/nvgrace-gpu/main.c | 50 +++++++++++++++++------------
 1 file changed, 30 insertions(+), 20 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +0 -0
index e346392b72f6..f74f3d8e1ebe 100644

drivers/vfio/pci/nvgrace-gpu/main.c static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) · line 137

Quoted 8 lines
drivers/vfio/pci/nvgrace-gpu/main.c +5 -0
@@ -130,6 +130,32 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)
 	vfio_pci_core_close_device(core_vdev);
 }
 
+static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+	struct nvgrace_gpu_pci_core_device *nvdev = vma->vm_private_data;
+	int index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);

This should not be a signed value. I think the right type is unsigned long.

drivers/vfio/pci/nvgrace-gpu/main.c static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) · line 150

Quoted 13 lines
drivers/vfio/pci/nvgrace-gpu/main.c +13 -0
+	vm_fault_t ret = VM_FAULT_SIGBUS;
+	struct mem_region *memregion;
+	unsigned long pgoff, pfn;
+
+	memregion = nvgrace_gpu_memregion(index, nvdev);
+	if (!memregion)
+		return ret;
+
+	pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
+	pfn = PHYS_PFN(memregion->memphys) + pgoff;
+
+	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock)
+		ret = vmf_insert_pfn(vmf->vma, vmf->address, pfn);

This needs to check for this:

        if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev))
                goto out_unlock;

So I think your series is out of order.

Move patch 2 before this one, add the above lines to vfio_pci_vmf_insert_pfn() as well as a lockdep to check the memory_lock

Then just call vfio_pci_vmf_insert_pfn() here and consider squashing patch 3.

Jason

author maintainer
#5 lore raw

PATCH v5 2/7 vfio: export function to map the VMA

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

Take out the implementation to map the VMA to the PTE/PMD/PUD as a separate function.

Export the function to be used by nvgrace-gpu module.

---
 drivers/vfio/pci/vfio_pci_core.c | 46 ++++++++++++++++++++------------
 include/linux/vfio_pci_core.h    |  2 ++
 2 files changed, 31 insertions(+), 17 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +29 -17
index 7dcf5439dedc..ede410e0ae1c 100644
@@ -1640,6 +1640,34 @@ static unsigned long vma_to_pfn(struct vm_area_struct *vma)
 	return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) + pgoff;
 }
 
+vm_fault_t vfio_pci_vmf_insert_pfn(struct vm_fault *vmf,
+				   unsigned long pfn,
+				   unsigned int order)
+{
+	vm_fault_t ret;
+
+	switch (order) {
+	case 0:
+		ret = vmf_insert_pfn(vmf->vma, vmf->address, pfn);
+		break;
+#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP
+	case PMD_ORDER:
+		ret = vmf_insert_pfn_pmd(vmf, pfn, false);
+		break;
+#endif
+#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP
+	case PUD_ORDER:
+		ret = vmf_insert_pfn_pud(vmf, pfn, false);
+		break;
+#endif
+	default:
+		ret = VM_FAULT_FALLBACK;
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_vmf_insert_pfn);
+
 static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
 					   unsigned int order)
 {
@@ -1662,23 +1690,7 @@ static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
 	if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev))
 		goto out_unlock;
 
-	switch (order) {
-	case 0:
-		ret = vmf_insert_pfn(vma, vmf->address, pfn);
-		break;
-#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP
-	case PMD_ORDER:
-		ret = vmf_insert_pfn_pmd(vmf, pfn, false);
-		break;
-#endif
-#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP
-	case PUD_ORDER:
-		ret = vmf_insert_pfn_pud(vmf, pfn, false);
-		break;
-#endif
-	default:
-		ret = VM_FAULT_FALLBACK;
-	}
+	ret = vfio_pci_vmf_insert_pfn(vmf, pfn, order);
 
 out_unlock:
 	up_read(&vdev->memory_lock);
include/linux/vfio_pci_core.h +2 -0
index f541044e42a2..970b3775505e 100644
@@ -119,6 +119,8 @@ ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf,
 		size_t count, loff_t *ppos);
 ssize_t vfio_pci_core_write(struct vfio_device *core_vdev, const char __user *buf,
 		size_t count, loff_t *ppos);
+vm_fault_t vfio_pci_vmf_insert_pfn(struct vm_fault *vmf, unsigned long pfn,
+				   unsigned int order);
 int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma);
 void vfio_pci_core_request(struct vfio_device *core_vdev, unsigned int count);
 int vfio_pci_core_match(struct vfio_device *core_vdev, char *buf);
Signature · 1 line
2.34.1
reviewer Reviewed-by on v5 2/7
#6 lore raw

RE: [PATCH v5 2/7] vfio: export function to map the VMA

Quoted 38 lines

-----Original Message-----
From: Ankit Agrawal <[email protected]>
Sent: 24 November 2025 11:59
To: Ankit Agrawal <[email protected]>; [email protected]; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; [email protected]; Aniket Agashe <[email protected]>; Vikram Sethi <[email protected]>; Matt Ochs <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Avihai Horon <[email protected]>; [email protected]; [email protected]; [email protected]; Alistair Popple <[email protected]>; [email protected]; linux- [email protected]; Neo Jia <[email protected]>; Kirti Wankhede <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Dan Williams <[email protected]>; Dheeraj Nigam <[email protected]>; Krishnakant Jaju <[email protected]>
Subject: [PATCH v5 2/7] vfio: export function to map the VMA

From: Ankit Agrawal <[email protected]>

Take out the implementation to map the VMA to the PTE/PMD/PUD as a separate function.

Export the function to be used by nvgrace-gpu module.

---
 drivers/vfio/pci/vfio_pci_core.c | 46 ++++++++++++++++++++------------
 include/linux/vfio_pci_core.h    |  2 ++
 2 files changed, 31 insertions(+), 17 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +0 -0
index 7dcf5439dedc..ede410e0ae1c 100644

drivers/vfio/pci/vfio_pci_core.c static unsigned long vma_to_pfn(struct · line 1646

Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c +3 -0
@@ -1640,6 +1640,34 @@ static unsigned long vma_to_pfn(struct
 vm_area_struct *vma)
 	return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) +
 pgoff;
 }
+vm_fault_t vfio_pci_vmf_insert_pfn(struct vm_fault *vmf,
+				   unsigned long pfn,
+				   unsigned int order)

Does this need to be called with vdev->memory_lock held? Please add a comment if so.

Quoted 29 lines
drivers/vfio/pci/vfio_pci_core.c +25 -0
+{
+	vm_fault_t ret;
+
+	switch (order) {
+	case 0:
+		ret = vmf_insert_pfn(vmf->vma, vmf->address, pfn);
+		break;
+#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP
+	case PMD_ORDER:
+		ret = vmf_insert_pfn_pmd(vmf, pfn, false);
+		break;
+#endif
+#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP
+	case PUD_ORDER:
+		ret = vmf_insert_pfn_pud(vmf, pfn, false);
+		break;
+#endif
+	default:
+		ret = VM_FAULT_FALLBACK;
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_vmf_insert_pfn);
+
 static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
 					   unsigned int order)

{

drivers/vfio/pci/vfio_pci_core.c static vm_fault_t · line 1693

Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c +0 -0
@@ -1662,23 +1690,7 @@ static vm_fault_t
 vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
 	if (vdev->pm_runtime_engaged ||
 !__vfio_pci_memory_enabled(vdev))
 		goto out_unlock;

Is the above check not required for the common helper?

Quoted 35 lines
drivers/vfio/pci/vfio_pci_core.c +1 -17
-	switch (order) {
-	case 0:
-		ret = vmf_insert_pfn(vma, vmf->address, pfn);
-		break;
-#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP
-	case PMD_ORDER:
-		ret = vmf_insert_pfn_pmd(vmf, pfn, false);
-		break;
-#endif
-#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP
-	case PUD_ORDER:
-		ret = vmf_insert_pfn_pud(vmf, pfn, false);
-		break;
-#endif
-	default:
-		ret = VM_FAULT_FALLBACK;
-	}
+	ret = vfio_pci_vmf_insert_pfn(vmf, pfn, order);
 
 out_unlock:
drivers/vfio/pci/vfio_pci_core.c +0 -0
 	up_read(&vdev->memory_lock);
include/linux/vfio_pci_core.h +2 -0
index f541044e42a2..970b3775505e 100644
@@ -119,6 +119,8 @@ ssize_t vfio_pci_core_read(struct vfio_device
 *core_vdev, char __user *buf,
 		size_t count, loff_t *ppos);
 ssize_t vfio_pci_core_write(struct vfio_device *core_vdev, const char __user
 *buf,
 		size_t count, loff_t *ppos);
+vm_fault_t vfio_pci_vmf_insert_pfn(struct vm_fault *vmf, unsigned long pfn,
+				   unsigned int order);
 int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct
Quoted 4 lines

*vma);
void vfio_pci_core_request(struct vfio_device *core_vdev, unsigned int
count);
int vfio_pci_core_match(struct vfio_device *core_vdev, char *buf);

Otherwise LGTM:

author maintainer
#7 lore raw

PATCH v5 3/7 vfio/nvgrace-gpu: Add support for huge pfnmap

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

NVIDIA's Grace based systems have large device memory. The device memory is mapped as VM_PFNMAP in the VMM VMA. The nvgrace-gpu module could make use of the huge PFNMAP support added in mm [1].

To achieve this, nvgrace-gpu module is updated to implement huge_fault ops. The implementation establishes mapping according to the order request. Note that if the PFN or the VMA address is unaligned to the order, the mapping fallbacks to the PTE level.

cc Alex Williamson <[email protected]>
cc Jason Gunthorpe <[email protected]>
cc Vikram Sethi <[email protected]>
---
 drivers/vfio/pci/nvgrace-gpu/main.c | 43 +++++++++++++++++++++++------
 1 file changed, 35 insertions(+), 8 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +35 -8
index f74f3d8e1ebe..c84c01954c9e 100644
@@ -130,32 +130,58 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)
 	vfio_pci_core_close_device(core_vdev);
 }
 
-static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
+static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,
+						  unsigned int order)
 {
 	struct vm_area_struct *vma = vmf->vma;
 	struct nvgrace_gpu_pci_core_device *nvdev = vma->vm_private_data;
 	int index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
 	vm_fault_t ret = VM_FAULT_SIGBUS;
 	struct mem_region *memregion;
-	unsigned long pgoff, pfn;
+	unsigned long pgoff, pfn, addr;
 
 	memregion = nvgrace_gpu_memregion(index, nvdev);
 	if (!memregion)
 		return ret;
 
-	pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
+	addr = vmf->address & ~((PAGE_SIZE << order) - 1);
+	pgoff = (addr - vma->vm_start) >> PAGE_SHIFT;
 	pfn = PHYS_PFN(memregion->memphys) + pgoff;
 
+	if (order && (addr < vma->vm_start ||
+		      addr + (PAGE_SIZE << order) > vma->vm_end ||
+		      pfn & ((1 << order) - 1)))
+		return VM_FAULT_FALLBACK;
+
 	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock)
-		ret = vmf_insert_pfn(vmf->vma, vmf->address, pfn);
+		ret = vfio_pci_vmf_insert_pfn(vmf, pfn, order);
 
 	return ret;
 }
 
+static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
+{
+	return nvgrace_gpu_vfio_pci_huge_fault(vmf, 0);
+}
+
 static const struct vm_operations_struct nvgrace_gpu_vfio_pci_mmap_ops = {
 	.fault = nvgrace_gpu_vfio_pci_fault,
+#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
+	.huge_fault = nvgrace_gpu_vfio_pci_huge_fault,
+#endif
 };
 
+static size_t nvgrace_gpu_aligned_devmem_size(size_t memlength)
+{
+#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP
+	return ALIGN(memlength, PMD_SIZE);
+#endif
+#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP
+	return ALIGN(memlength, PUD_SIZE);
+#endif
+	return memlength;
+}
+
 static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
 			    struct vm_area_struct *vma)
 {
@@ -185,10 +211,10 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
 		return -EOVERFLOW;
 
 	/*
-	 * Check that the mapping request does not go beyond available device
-	 * memory size
+	 * Check that the mapping request does not go beyond the exposed
+	 * device memory size.
 	 */
-	if (end > memregion->memlength)
+	if (end > nvgrace_gpu_aligned_devmem_size(memregion->memlength))
 		return -EINVAL;
 
 	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
@@ -258,7 +284,8 @@ nvgrace_gpu_ioctl_get_region_info(struct vfio_device *core_vdev,
 
 	sparse->nr_areas = 1;
 	sparse->areas[0].offset = 0;
-	sparse->areas[0].size = memregion->memlength;
+	sparse->areas[0].size =
+		nvgrace_gpu_aligned_devmem_size(memregion->memlength);
 	sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
 	sparse->header.version = 1;
 
Signature · 1 line
2.34.1
maintainer
#8 lore raw

Re: [PATCH v5 3/7] vfio/nvgrace-gpu: Add support for huge pfnmap

Quoted 51 lines · On Mon, 24 Nov 2025 11:59:22 +0000 <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

NVIDIA's Grace based systems have large device memory. The device memory is mapped as VM_PFNMAP in the VMM VMA. The nvgrace-gpu module could make use of the huge PFNMAP support added in mm [1].

To achieve this, nvgrace-gpu module is updated to implement huge_fault ops. The implementation establishes mapping according to the order request. Note that if the PFN or the VMA address is unaligned to the order, the mapping fallbacks to the PTE level.

cc Alex Williamson <[email protected]>
cc Jason Gunthorpe <[email protected]>
cc Vikram Sethi <[email protected]>
---
 drivers/vfio/pci/nvgrace-gpu/main.c | 43 +++++++++++++++++++++++------
 1 file changed, 35 insertions(+), 8 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +6 -3
index f74f3d8e1ebe..c84c01954c9e 100644
@@ -130,32 +130,58 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)
 	vfio_pci_core_close_device(core_vdev);
 }
 
-static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
+static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,
+						  unsigned int order)
 {
 	struct vm_area_struct *vma = vmf->vma;
 	struct nvgrace_gpu_pci_core_device *nvdev = vma->vm_private_data;
 	int index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
 	vm_fault_t ret = VM_FAULT_SIGBUS;
 	struct mem_region *memregion;
-	unsigned long pgoff, pfn;
+	unsigned long pgoff, pfn, addr;
 
 	memregion = nvgrace_gpu_memregion(index, nvdev);
 	if (!memregion)
 		return ret;
 
-	pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
+	addr = vmf->address & ~((PAGE_SIZE << order) - 1);
+	pgoff = (addr - vma->vm_start) >> PAGE_SHIFT;
 	pfn = PHYS_PFN(memregion->memphys) + pgoff;
 
+	if (order && (addr < vma->vm_start ||

drivers/vfio/pci/nvgrace-gpu/main.c static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) · line 160

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c +5 -1
+		      addr + (PAGE_SIZE << order) > vma->vm_end ||
+		      pfn & ((1 << order) - 1)))
+		return VM_FAULT_FALLBACK;
+
 	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock)
-		ret = vmf_insert_pfn(vmf->vma, vmf->address, pfn);
+		ret = vfio_pci_vmf_insert_pfn(vmf, pfn, order);
 
 	return ret;
 }

It may be worth considering replicating the dev_dbg from vfio_pci_mmap_huge_fault(), it's been very useful in validating that we're getting the huge PFNMAPs we expect. Thanks,

Alex

Quoted 51 lines
drivers/vfio/pci/nvgrace-gpu/main.c +24 -4
+static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
+{
+	return nvgrace_gpu_vfio_pci_huge_fault(vmf, 0);
+}
+
 static const struct vm_operations_struct nvgrace_gpu_vfio_pci_mmap_ops = {
 	.fault = nvgrace_gpu_vfio_pci_fault,
+#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
+	.huge_fault = nvgrace_gpu_vfio_pci_huge_fault,
+#endif
 };
 
+static size_t nvgrace_gpu_aligned_devmem_size(size_t memlength)
+{
+#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP
+	return ALIGN(memlength, PMD_SIZE);
+#endif
+#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP
+	return ALIGN(memlength, PUD_SIZE);
+#endif
+	return memlength;
+}
+
 static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
 			    struct vm_area_struct *vma)
 {
@@ -185,10 +211,10 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
 		return -EOVERFLOW;
 
 	/*
-	 * Check that the mapping request does not go beyond available device
-	 * memory size
+	 * Check that the mapping request does not go beyond the exposed
+	 * device memory size.
 	 */
-	if (end > memregion->memlength)
+	if (end > nvgrace_gpu_aligned_devmem_size(memregion->memlength))
 		return -EINVAL;
 
 	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
@@ -258,7 +284,8 @@ nvgrace_gpu_ioctl_get_region_info(struct vfio_device *core_vdev,
 
 	sparse->nr_areas = 1;
 	sparse->areas[0].offset = 0;
-	sparse->areas[0].size = memregion->memlength;
+	sparse->areas[0].size =
+		nvgrace_gpu_aligned_devmem_size(memregion->memlength);
 	sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
 	sparse->header.version = 1;
 
author maintainer
#9 lore raw

Re: [PATCH v5 3/7] vfio/nvgrace-gpu: Add support for huge pfnmap

Quoted 10 lines
Quoted 6 lines

       scoped_guard(rwsem_read, &nvdev->core_device.memory_lock) -             ret = vmf_insert_pfn(vmf->vma, vmf->address, pfn); +             ret = vfio_pci_vmf_insert_pfn(vmf, pfn, order);

       return ret;
  }

It may be worth considering replicating the dev_dbg from vfio_pci_mmap_huge_fault(), it's been very useful in validating that we're getting the huge PFNMAPs we expect.  Thanks,

Sure, will add the log message. Thanks!

Quoted 2 lines

Alex

#10 lore raw

RE: [PATCH v5 3/7] vfio/nvgrace-gpu: Add support for huge pfnmap

In reply to [email protected]
Quoted 97 lines

-----Original Message-----
From: Ankit Agrawal <[email protected]>
Sent: 24 November 2025 11:59
To: Ankit Agrawal <[email protected]>; [email protected]; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; [email protected]; Aniket Agashe <[email protected]>; Vikram Sethi <[email protected]>; Matt Ochs <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Avihai Horon <[email protected]>; [email protected]; [email protected]; [email protected]; Alistair Popple <[email protected]>; [email protected]; linux- [email protected]; Neo Jia <[email protected]>; Kirti Wankhede <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Dan Williams <[email protected]>; Dheeraj Nigam <[email protected]>; Krishnakant Jaju <[email protected]>
Subject: [PATCH v5 3/7] vfio/nvgrace-gpu: Add support for huge pfnmap

From: Ankit Agrawal <[email protected]>

NVIDIA's Grace based systems have large device memory. The device memory is mapped as VM_PFNMAP in the VMM VMA. The nvgrace-gpu module could make use of the huge PFNMAP support added in mm [1].

To achieve this, nvgrace-gpu module is updated to implement huge_fault ops. The implementation establishes mapping according to the order request. Note that if the PFN or the VMA address is unaligned to the order, the mapping fallbacks to the PTE level.

[email protected]/ [1]

cc Alex Williamson <[email protected]>
cc Jason Gunthorpe <[email protected]>
cc Vikram Sethi <[email protected]>
---
 drivers/vfio/pci/nvgrace-gpu/main.c | 43 +++++++++++++++++++++++------
 1 file changed, 35 insertions(+), 8 deletions(-)
drivers/vfio/pci/nvgrace- renamed from drivers/vfio/pci/nvgrace-gpu/main.c +0 -0

gpu/main.c
index f74f3d8e1ebe..c84c01954c9e 100644

drivers/vfio/pci/nvgrace-gpu/main.c +19 -4
@@ -130,32 +130,58 @@ static void nvgrace_gpu_close_device(struct
 vfio_device *core_vdev)
 	vfio_pci_core_close_device(core_vdev);
 }
 
-static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
+static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,
+						  unsigned int order)
 {
 	struct vm_area_struct *vma = vmf->vma;
 	struct nvgrace_gpu_pci_core_device *nvdev = vma->vm_private_data;
 	int index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT -
 PAGE_SHIFT);
 	vm_fault_t ret = VM_FAULT_SIGBUS;
 	struct mem_region *memregion;
-	unsigned long pgoff, pfn;
+	unsigned long pgoff, pfn, addr;
 
 	memregion = nvgrace_gpu_memregion(index, nvdev);
 	if (!memregion)
 		return ret;
 
-	pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
+	addr = vmf->address & ~((PAGE_SIZE << order) - 1);
+	pgoff = (addr - vma->vm_start) >> PAGE_SHIFT;
 	pfn = PHYS_PFN(memregion->memphys) + pgoff;
 
+	if (order && (addr < vma->vm_start ||
+		      addr + (PAGE_SIZE << order) > vma->vm_end ||
+		      pfn & ((1 << order) - 1)))
+		return VM_FAULT_FALLBACK;
+
 	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock)
-		ret = vmf_insert_pfn(vmf->vma, vmf->address, pfn);
+		ret = vfio_pci_vmf_insert_pfn(vmf, pfn, order);
 
 	return ret;
 }
 
+static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
+{
+	return nvgrace_gpu_vfio_pci_huge_fault(vmf, 0);
+}
+
 static const struct vm_operations_struct nvgrace_gpu_vfio_pci_mmap_ops =
 {
 	.fault = nvgrace_gpu_vfio_pci_fault,
+#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
+	.huge_fault = nvgrace_gpu_vfio_pci_huge_fault,
+#endif

drivers/vfio/pci/nvgrace-gpu/main.c static void nvgrace_gpu_close_device(struct · line 184

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c +8 -0
 };
 
+static size_t nvgrace_gpu_aligned_devmem_size(size_t memlength)
+{
+#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP
+	return ALIGN(memlength, PMD_SIZE);
+#endif
+#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP
+	return ALIGN(memlength, PUD_SIZE);
+#endif

I think all this should be ALIGN_DOWN to be safe.

Thanks,
Shameer

Quoted 38 lines
drivers/vfio/pci/nvgrace-gpu/main.c +3 -0
+	return memlength;
+}
+

static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,

drivers/vfio/pci/nvgrace-gpu/main.c +0 -1
 			    struct vm_area_struct *vma)
 {
@@ -185,10 +211,10 @@ static int nvgrace_gpu_mmap(struct vfio_device
 *core_vdev,
 		return -EOVERFLOW;
 
 	/*
-	 * Check that the mapping request does not go beyond available

device

drivers/vfio/pci/nvgrace-gpu/main.c +3 -2
-	 * memory size
+	 * Check that the mapping request does not go beyond the exposed
+	 * device memory size.
 	 */
-	if (end > memregion->memlength)
+	if (end > nvgrace_gpu_aligned_devmem_size(memregion-
Quoted 1 line

memlength))

drivers/vfio/pci/nvgrace-gpu/main.c +0 -0
 		return -EINVAL;
 
 	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND |

VM_DONTDUMP);

drivers/vfio/pci/nvgrace-gpu/main.c +2 -1
@@ -258,7 +284,8 @@ nvgrace_gpu_ioctl_get_region_info(struct vfio_device
 *core_vdev,
 
 	sparse->nr_areas = 1;
 	sparse->areas[0].offset = 0;
-	sparse->areas[0].size = memregion->memlength;
+	sparse->areas[0].size =
+		nvgrace_gpu_aligned_devmem_size(memregion-
 >memlength);
 	sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;

sparse->header.version = 1;

--
2.34.1

#11 lore raw

Re: [PATCH v5 3/7] vfio/nvgrace-gpu: Add support for huge pfnmap

In reply to [email protected]

drivers/vfio/pci/nvgrace-gpu/main.c static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) · line 183

Quoted 10 lines · On Mon, Nov 24, 2025 at 11:59:22AM +0000, [email protected] wrote:
drivers/vfio/pci/nvgrace-gpu/main.c +10 -0
+static size_t nvgrace_gpu_aligned_devmem_size(size_t memlength)
+{
+#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP
+	return ALIGN(memlength, PMD_SIZE);
+#endif
+#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP
+	return ALIGN(memlength, PUD_SIZE);
+#endif
+	return memlength;
+}

This needs a comment why it is OK to change the size up? Seems really surprising (and wrong!). The commit message should explain it too.

Jason

author maintainer
#12 lore raw

PATCH v5 4/7 vfio: use vfio_pci_core_setup_barmap to map bar in mmap

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

Remove code duplication in vfio_pci_core_mmap by calling vfio_pci_core_setup_barmap to perform the bar mapping.

---
 drivers/vfio/pci/vfio_pci_core.c | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +3 -12
index ede410e0ae1c..3d21f6a8c279 100644
@@ -1761,18 +1761,9 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma
 	 * Even though we don't make use of the barmap for the mmap,
 	 * we need to request the region and the barmap tracks that.
 	 */
-	if (!vdev->barmap[index]) {
-		ret = pci_request_selected_regions(pdev,
-						   1 << index, "vfio-pci");
-		if (ret)
-			return ret;
-
-		vdev->barmap[index] = pci_iomap(pdev, index, 0);
-		if (!vdev->barmap[index]) {
-			pci_release_selected_regions(pdev, 1 << index);
-			return -ENOMEM;
-		}
-	}
+	ret = vfio_pci_core_setup_barmap(vdev, index);
+	if (ret)
+		return ret;
 
 	vma->vm_private_data = vdev;
 	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
Signature · 1 line
2.34.1
reviewer Reviewed-by on v5 4/7
#13 lore raw

RE: [PATCH v5 4/7] vfio: use vfio_pci_core_setup_barmap to map bar in mmap

Quoted 21 lines

-----Original Message-----
From: Ankit Agrawal <[email protected]>
Sent: 24 November 2025 11:59
To: Ankit Agrawal <[email protected]>; [email protected]; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; [email protected]; Aniket Agashe <[email protected]>; Vikram Sethi <[email protected]>; Matt Ochs <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Avihai Horon <[email protected]>; [email protected]; [email protected]; [email protected]; Alistair Popple <[email protected]>; [email protected]; linux- [email protected]; Neo Jia <[email protected]>; Kirti Wankhede <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Dan Williams <[email protected]>; Dheeraj Nigam <[email protected]>; Krishnakant Jaju <[email protected]>
Subject: [PATCH v5 4/7] vfio: use vfio_pci_core_setup_barmap to map bar in mmap

Quoted 8 lines

From: Ankit Agrawal <[email protected]>

Remove code duplication in vfio_pci_core_mmap by calling vfio_pci_core_setup_barmap to perform the bar mapping.

LGTM:

author maintainer
#14 lore raw

PATCH v5 5/7 vfio/nvgrace-gpu: split the code to wait for GPU ready

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

Split the function that check for the GPU device being ready on the probe.

Move the code to wait for the GPU to be ready through BAR0 register reads to a separate function. This would help reuse the code.

---
 drivers/vfio/pci/nvgrace-gpu/main.c | 33 ++++++++++++++++++-----------
 1 file changed, 21 insertions(+), 12 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +21 -12
index c84c01954c9e..3e45b8bd1a89 100644
@@ -130,6 +130,24 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)
 	vfio_pci_core_close_device(core_vdev);
 }
 
+static int nvgrace_gpu_wait_device_ready(void __iomem *io)
+{
+	unsigned long timeout = jiffies + msecs_to_jiffies(POLL_TIMEOUT_MS);
+	int ret = -ETIME;
+
+	do {
+		if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) &&
+		    (ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY)) {
+			ret = 0;
+			goto ready_check_exit;
+		}
+		msleep(POLL_QUANTUM_MS);
+	} while (!time_after(jiffies, timeout));
+
+ready_check_exit:
+	return ret;
+}
+
 static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,
 						  unsigned int order)
 {
@@ -930,9 +948,8 @@ static bool nvgrace_gpu_has_mig_hw_bug(struct pci_dev *pdev)
  * Ensure that the BAR0 region is enabled before accessing the
  * registers.
  */
-static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev)
+static int nvgrace_gpu_probe_check_device_ready(struct pci_dev *pdev)
 {
-	unsigned long timeout = jiffies + msecs_to_jiffies(POLL_TIMEOUT_MS);
 	void __iomem *io;
 	int ret = -ETIME;
 
@@ -950,16 +967,8 @@ static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev)
 		goto iomap_exit;
 	}
 
-	do {
-		if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) &&
-		    (ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY)) {
-			ret = 0;
-			goto reg_check_exit;
-		}
-		msleep(POLL_QUANTUM_MS);
-	} while (!time_after(jiffies, timeout));
+	ret = nvgrace_gpu_wait_device_ready(io);
 
-reg_check_exit:
 	pci_iounmap(pdev, io);
 iomap_exit:
 	pci_release_selected_regions(pdev, 1 << 0);
@@ -976,7 +985,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev,
 	u64 memphys, memlength;
 	int ret;
 
-	ret = nvgrace_gpu_wait_device_ready(pdev);
+	ret = nvgrace_gpu_probe_check_device_ready(pdev);
 	if (ret)
 		return ret;
 
Signature · 1 line
2.34.1
maintainer
#15 lore raw

Re: [PATCH v5 5/7] vfio/nvgrace-gpu: split the code to wait for GPU ready

Quoted 56 lines · On Mon, 24 Nov 2025 11:59:24 +0000 <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

Split the function that check for the GPU device being ready on the probe.

Move the code to wait for the GPU to be ready through BAR0 register reads to a separate function. This would help reuse the code.

---
 drivers/vfio/pci/nvgrace-gpu/main.c | 33 ++++++++++++++++++-----------
 1 file changed, 21 insertions(+), 12 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +19 -2
index c84c01954c9e..3e45b8bd1a89 100644
@@ -130,6 +130,24 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)
 	vfio_pci_core_close_device(core_vdev);
 }
 
+static int nvgrace_gpu_wait_device_ready(void __iomem *io)
+{
+	unsigned long timeout = jiffies + msecs_to_jiffies(POLL_TIMEOUT_MS);
+	int ret = -ETIME;
+
+	do {
+		if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) &&
+		    (ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY)) {
+			ret = 0;
+			goto ready_check_exit;
+		}
+		msleep(POLL_QUANTUM_MS);
+	} while (!time_after(jiffies, timeout));
+
+ready_check_exit:
+	return ret;
+}
+
 static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,
 						  unsigned int order)
 {
@@ -930,9 +948,8 @@ static bool nvgrace_gpu_has_mig_hw_bug(struct pci_dev *pdev)
  * Ensure that the BAR0 region is enabled before accessing the
  * registers.
  */
-static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev)
+static int nvgrace_gpu_probe_check_device_ready(struct pci_dev *pdev)
 {
-	unsigned long timeout = jiffies + msecs_to_jiffies(POLL_TIMEOUT_MS);
 	void __iomem *io;
 	int ret = -ETIME;
 
@@ -950,16 +967,8 @@ static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev)
 		goto iomap_exit;
 	}

drivers/vfio/pci/nvgrace-gpu/main.c static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev) · line 970

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c +1 -8
 
-	do {
-		if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) &&
-		    (ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY)) {
-			ret = 0;
-			goto reg_check_exit;
-		}
-		msleep(POLL_QUANTUM_MS);
-	} while (!time_after(jiffies, timeout));
+	ret = nvgrace_gpu_wait_device_ready(io);

I think you're inadvertently fixing a bug here too. The ret initialization to -ETIME is immediately clobbered by pci_enable_device(), so exceeding the timeout would never generate an error. Now it will:

Fixes d85f69d520e6 ("vfio/nvgrace-gpu: Check the HBM training and C2C link status")

Also we should remove the ret initialization. Otherwise the series LGTM. Thanks,

Alex

Quoted 14 lines
drivers/vfio/pci/nvgrace-gpu/main.c +1 -2
-reg_check_exit:
 	pci_iounmap(pdev, io);
 iomap_exit:
 	pci_release_selected_regions(pdev, 1 << 0);
@@ -976,7 +985,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev,
 	u64 memphys, memlength;
 	int ret;
 
-	ret = nvgrace_gpu_wait_device_ready(pdev);
+	ret = nvgrace_gpu_probe_check_device_ready(pdev);
 	if (ret)
 		return ret;
 
author maintainer
#16 lore raw

Re: [PATCH v5 5/7] vfio/nvgrace-gpu: split the code to wait for GPU ready

Quoted 10 lines
Quoted 9 lines

-     do {
-             if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) && -                 (ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY)) { -                     ret = 0;
-                     goto reg_check_exit;
-             }
-             msleep(POLL_QUANTUM_MS);
-     } while (!time_after(jiffies, timeout));
+     ret = nvgrace_gpu_wait_device_ready(io);

Quoted 6 lines

I think you're inadvertently fixing a bug here too.  The ret initialization to -ETIME is immediately clobbered by pci_enable_device(), so exceeding the timeout would never generate an error.  Now it will:

Fixes d85f69d520e6 ("vfio/nvgrace-gpu: Check the HBM training and C2C link status")

Oh yeah. Will add this line in the commit message.

Quoted 4 lines

Also we should remove the ret initialization.  Otherwise the series LGTM.  Thanks,

Alex

Thank you very much for the review, Alex!

reviewer Reviewed-by on v5 5/7
#17 lore raw

RE: [PATCH v5 5/7] vfio/nvgrace-gpu: split the code to wait for GPU ready

In reply to [email protected]
Quoted 22 lines

-----Original Message-----
From: Ankit Agrawal <[email protected]>
Sent: 24 November 2025 11:59
To: Ankit Agrawal <[email protected]>; [email protected]; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; [email protected]; Aniket Agashe <[email protected]>; Vikram Sethi <[email protected]>; Matt Ochs <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Avihai Horon <[email protected]>; [email protected]; [email protected]; [email protected]; Alistair Popple <[email protected]>; [email protected]; linux- [email protected]; Neo Jia <[email protected]>; Kirti Wankhede <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Dan Williams <[email protected]>; Dheeraj Nigam <[email protected]>; Krishnakant Jaju <[email protected]>
Subject: [PATCH v5 5/7] vfio/nvgrace-gpu: split the code to wait for GPU ready

From: Ankit Agrawal <[email protected]>

Quoted 7 lines

Split the function that check for the GPU device being ready on the probe.

Move the code to wait for the GPU to be ready through BAR0 register reads to a separate function. This would help reuse the code.

With a nit below:

Quoted 24 lines
 drivers/vfio/pci/nvgrace-gpu/main.c | 33 ++++++++++++++++++-----------
 1 file changed, 21 insertions(+), 12 deletions(-)
drivers/vfio/pci/nvgrace- renamed from drivers/vfio/pci/nvgrace-gpu/main.c +0 -0

gpu/main.c
index c84c01954c9e..3e45b8bd1a89 100644

drivers/vfio/pci/nvgrace-gpu/main.c +7 -0
@@ -130,6 +130,24 @@ static void nvgrace_gpu_close_device(struct
 vfio_device *core_vdev)
 	vfio_pci_core_close_device(core_vdev);
 }
 
+static int nvgrace_gpu_wait_device_ready(void __iomem *io)
+{
+	unsigned long timeout = jiffies +
 msecs_to_jiffies(POLL_TIMEOUT_MS);
+	int ret = -ETIME;
+
+	do {
+		if ((ioread32(io + C2C_LINK_BAR0_OFFSET) ==

STATUS_READY) &&

drivers/vfio/pci/nvgrace-gpu/main.c +1 -0
+		    (ioread32(io + HBM_TRAINING_BAR0_OFFSET) ==

STATUS_READY)) {

drivers/vfio/pci/nvgrace-gpu/main.c static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) · line 142

Quoted 2 lines
drivers/vfio/pci/nvgrace-gpu/main.c +2 -0
+			ret = 0;
+			goto ready_check_exit;

You could return directly here and avoid that goto.

Thanks,
Shameer

Quoted 56 lines
drivers/vfio/pci/nvgrace-gpu/main.c +9 -2
+		}
+		msleep(POLL_QUANTUM_MS);
+	} while (!time_after(jiffies, timeout));
+
+ready_check_exit:
+	return ret;
+}
+
 static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,
 						  unsigned int order)
 {
@@ -930,9 +948,8 @@ static bool nvgrace_gpu_has_mig_hw_bug(struct
 pci_dev *pdev)
  * Ensure that the BAR0 region is enabled before accessing the
  * registers.
  */
-static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev)
+static int nvgrace_gpu_probe_check_device_ready(struct pci_dev *pdev)
 {
-	unsigned long timeout = jiffies +
 msecs_to_jiffies(POLL_TIMEOUT_MS);
 	void __iomem *io;
drivers/vfio/pci/nvgrace-gpu/main.c +0 -2
 	int ret = -ETIME;
@@ -950,16 +967,8 @@ static int nvgrace_gpu_wait_device_ready(struct
 pci_dev *pdev)
 		goto iomap_exit;
 	}
 
-	do {
-		if ((ioread32(io + C2C_LINK_BAR0_OFFSET) ==

STATUS_READY) &&

drivers/vfio/pci/nvgrace-gpu/main.c +0 -1
-		    (ioread32(io + HBM_TRAINING_BAR0_OFFSET) ==

STATUS_READY)) {

drivers/vfio/pci/nvgrace-gpu/main.c +2 -8
-			ret = 0;
-			goto reg_check_exit;
-		}
-		msleep(POLL_QUANTUM_MS);
-	} while (!time_after(jiffies, timeout));
+	ret = nvgrace_gpu_wait_device_ready(io);
 
-reg_check_exit:
 	pci_iounmap(pdev, io);
 iomap_exit:
 	pci_release_selected_regions(pdev, 1 << 0);
@@ -976,7 +985,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev,
 	u64 memphys, memlength;
 	int ret;
 
-	ret = nvgrace_gpu_wait_device_ready(pdev);
+	ret = nvgrace_gpu_probe_check_device_ready(pdev);
 	if (ret)
 		return ret;
 
--

2.34.1

author maintainer
#18 lore raw

Re: [PATCH v5 5/7] vfio/nvgrace-gpu: split the code to wait for GPU ready

Thanks Shameer for the reviewed-by.

Quoted 9 lines
Quoted 7 lines

+     do {
+             if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) &&
+                 (ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY)) {
+                     ret = 0;
+                     goto ready_check_exit;

You could return directly here and avoid that goto.

Yeah, I have a bad habit of overusing goto. I'll update it to return.

author maintainer
#19 lore raw

PATCH v5 6/7 vfio/nvgrace-gpu: Inform devmem unmapped after reset

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

Introduce a new flag reset_done to notify that the GPU has just been reset and the mapping to the GPU memory is zapped.

Implement the reset_done handler to set this new variable. It will be used later in the patches to wait for the GPU memory to be ready before doing any mapping or access.

---
 drivers/vfio/pci/nvgrace-gpu/main.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
drivers/vfio/pci/nvgrace-gpu/main.c +20 -1
index 3e45b8bd1a89..bef9f25bf8f3 100644
@@ -58,6 +58,8 @@ struct nvgrace_gpu_pci_core_device {
 	/* Lock to control device memory kernel mapping */
 	struct mutex remap_lock;
 	bool has_mig_hw_bug;
+	/* GPU has just been reset */
+	bool reset_done;
 };
 
 static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device *core_vdev)
@@ -1048,12 +1050,29 @@ static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = {
 
 MODULE_DEVICE_TABLE(pci, nvgrace_gpu_vfio_pci_table);
 
+static void nvgrace_gpu_vfio_pci_reset_done(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev);
+	struct nvgrace_gpu_pci_core_device *nvdev =
+		container_of(core_device, struct nvgrace_gpu_pci_core_device,
+			     core_device);
+
+	lockdep_assert_held_write(&core_device->memory_lock);
+
+	nvdev->reset_done = true;
+}
+
+static const struct pci_error_handlers nvgrace_gpu_vfio_pci_err_handlers = {
+	.reset_done = nvgrace_gpu_vfio_pci_reset_done,
+	.error_detected = vfio_pci_core_aer_err_detected,
+};
+
 static struct pci_driver nvgrace_gpu_vfio_pci_driver = {
 	.name = KBUILD_MODNAME,
 	.id_table = nvgrace_gpu_vfio_pci_table,
 	.probe = nvgrace_gpu_probe,
 	.remove = nvgrace_gpu_remove,
-	.err_handler = &vfio_pci_core_err_handlers,
+	.err_handler = &nvgrace_gpu_vfio_pci_err_handlers,
 	.driver_managed_dma = true,
 };
 
Signature · 1 line
2.34.1
#20 lore raw

Re: [PATCH v5 6/7] vfio/nvgrace-gpu: Inform devmem unmapped after reset

Quoted 5 lines · On Mon, Nov 24, 2025 at 11:59:25AM +0000, [email protected] wrote:
drivers/vfio/pci/nvgrace-gpu/main.c +1 -0
@@ -1048,12 +1050,29 @@ static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = {
 
 MODULE_DEVICE_TABLE(pci, nvgrace_gpu_vfio_pci_table);
 
+static void nvgrace_gpu_vfio_pci_reset_done(struct pci_dev *pdev)

drivers/vfio/pci/nvgrace-gpu/main.c static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = { · line 1063

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c +10 -0
+{
+	struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev);
+	struct nvgrace_gpu_pci_core_device *nvdev =
+		container_of(core_device, struct nvgrace_gpu_pci_core_device,
+			     core_device);
+
+	lockdep_assert_held_write(&core_device->memory_lock);
+
+	nvdev->reset_done = true;
+}

Seems surprising/wrong at least needs a comment. What on earth is ensuring that lockdep within a PCI callback??

drivers/vfio/pci/nvgrace-gpu/main.c static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = { · line 1075

Quoted 7 lines
drivers/vfio/pci/nvgrace-gpu/main.c +1 -1
 static struct pci_driver nvgrace_gpu_vfio_pci_driver = {
 	.name = KBUILD_MODNAME,
 	.id_table = nvgrace_gpu_vfio_pci_table,
 	.probe = nvgrace_gpu_probe,
 	.remove = nvgrace_gpu_remove,
-	.err_handler = &vfio_pci_core_err_handlers,
+	.err_handler = &nvgrace_gpu_vfio_pci_err_handlers,

Jason

author maintainer
#21 lore raw

Re: [PATCH v5 6/7] vfio/nvgrace-gpu: Inform devmem unmapped after reset

Quoted 6 lines
Quoted 6 lines
drivers/vfio/pci/nvgrace-gpu/main.c +6 -0
+static void nvgrace_gpu_vfio_pci_reset_done(struct pci_dev *pdev)
+{
+     struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev);
+     struct nvgrace_gpu_pci_core_device *nvdev =
+             container_of(core_device, struct nvgrace_gpu_pci_core_device,
+                          core_device);
Quoted 8 lines

drivers/vfio/pci/nvgrace-gpu/main.c static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = { · line 1063

Quoted 5 lines
drivers/vfio/pci/nvgrace-gpu/main.c +5 -0
+
+     lockdep_assert_held_write(&core_device->memory_lock);
+
+     nvdev->reset_done = true;
+}

Seems surprising/wrong at least needs a comment. What on earth is ensuring that lockdep within a PCI callback??

Yeah, I noticed that even on vfio_pci_core_enable in the open, the lock isn't taken. I'll remove it.

author maintainer
#22 lore raw

PATCH v5 7/7 vfio/nvgrace-gpu: wait for the GPU mem to be ready

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

Speculative prefetches from CPU to GPU memory until the GPU is ready after reset can cause harmless corrected RAS events to be logged on Grace systems. It is thus preferred that the mapping not be re-established until the GPU is ready post reset.

The GPU readiness can be checked through BAR0 registers similar to the checking at the time of device probe.

It can take several seconds for the GPU to be ready. So it is desirable that the time overlaps as much of the VM startup as possible to reduce impact on the VM bootup time. The GPU readiness state is thus checked on the first fault/huge_fault request or read/write access which amortizes the GPU readiness time.

The first fault and read/write checks the GPU state when the reset_done flag - which denotes whether the GPU has just been reset. The memory_lock is taken across map/access to avoid races with GPU reset.

cc Alex Williamson <[email protected]>
cc Jason Gunthorpe <[email protected]>
cc Vikram Sethi <[email protected]>
---
 drivers/vfio/pci/nvgrace-gpu/main.c | 79 ++++++++++++++++++++++++++---
 1 file changed, 72 insertions(+), 7 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +72 -7
index bef9f25bf8f3..fbc19fe688ca 100644
@@ -104,6 +104,17 @@ static int nvgrace_gpu_open_device(struct vfio_device *core_vdev)
 		mutex_init(&nvdev->remap_lock);
 	}
 
+	/*
+	 * GPU readiness is checked by reading the BAR0 registers.
+	 *
+	 * ioremap BAR0 to ensure that the BAR0 mapping is present before
+	 * register reads on first fault before establishing any GPU
+	 * memory mapping.
+	 */
+	ret = vfio_pci_core_setup_barmap(vdev, 0);
+	if (ret)
+		return ret;
+
 	vfio_pci_core_finish_enable(vdev);
 
 	return 0;
@@ -150,6 +161,26 @@ static int nvgrace_gpu_wait_device_ready(void __iomem *io)
 	return ret;
 }
 
+static int
+nvgrace_gpu_check_device_ready(struct nvgrace_gpu_pci_core_device *nvdev)
+{
+	struct vfio_pci_core_device *vdev = &nvdev->core_device;
+	int ret;
+
+	lockdep_assert_held_read(&vdev->memory_lock);
+
+	if (!nvdev->reset_done)
+		return 0;
+
+	ret = nvgrace_gpu_wait_device_ready(vdev->barmap[0]);
+	if (ret)
+		return ret;
+
+	nvdev->reset_done = false;
+
+	return 0;
+}
+
 static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,
 						  unsigned int order)
 {
@@ -173,8 +204,18 @@ static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,
 		      pfn & ((1 << order) - 1)))
 		return VM_FAULT_FALLBACK;
 
-	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock)
+	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock) {
+		/*
+		 * If the GPU memory is accessed by the CPU while the GPU is
+		 * not ready after reset, it can cause harmless corrected RAS
+		 * events to be logged. Make sure the GPU is ready before
+		 * establishing the mappings.
+		 */
+		if (nvgrace_gpu_check_device_ready(nvdev))
+			return ret;
+
 		ret = vfio_pci_vmf_insert_pfn(vmf, pfn, order);
+	}
 
 	return ret;
 }
@@ -593,9 +634,21 @@ nvgrace_gpu_read_mem(struct nvgrace_gpu_pci_core_device *nvdev,
 	else
 		mem_count = min(count, memregion->memlength - (size_t)offset);
 
-	ret = nvgrace_gpu_map_and_read(nvdev, buf, mem_count, ppos);
-	if (ret)
-		return ret;
+	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock) {
+		/*
+		 * If the GPU memory is accessed by the CPU while the GPU is
+		 * not ready after reset, it can cause harmless corrected RAS
+		 * events to be logged. Make sure the GPU is ready before
+		 * establishing the mappings.
+		 */
+		ret = nvgrace_gpu_check_device_ready(nvdev);
+		if (ret)
+			return ret;
+
+		ret = nvgrace_gpu_map_and_read(nvdev, buf, mem_count, ppos);
+		if (ret)
+			return ret;
+	}
 
 	/*
 	 * Only the device memory present on the hardware is mapped, which may
@@ -713,9 +766,21 @@ nvgrace_gpu_write_mem(struct nvgrace_gpu_pci_core_device *nvdev,
 	 */
 	mem_count = min(count, memregion->memlength - (size_t)offset);
 
-	ret = nvgrace_gpu_map_and_write(nvdev, buf, mem_count, ppos);
-	if (ret)
-		return ret;
+	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock) {
+		/*
+		 * If the GPU memory is accessed by the CPU while the GPU is
+		 * not ready after reset, it can cause harmless corrected RAS
+		 * events to be logged. Make sure the GPU is ready before
+		 * establishing the mappings.
+		 */
+		ret = nvgrace_gpu_check_device_ready(nvdev);
+		if (ret)
+			return ret;
+
+		ret = nvgrace_gpu_map_and_write(nvdev, buf, mem_count, ppos);
+		if (ret)
+			return ret;
+	}
 
 exitfn:
 	*ppos += count;
Signature · 1 line
2.34.1
#23 lore raw

RE: [PATCH v5 7/7] vfio/nvgrace-gpu: wait for the GPU mem to be ready

Quoted 160 lines

-----Original Message-----
From: Ankit Agrawal <[email protected]>
Sent: 24 November 2025 11:59
To: Ankit Agrawal <[email protected]>; [email protected]; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; [email protected]; Aniket Agashe <[email protected]>; Vikram Sethi <[email protected]>; Matt Ochs <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Avihai Horon <[email protected]>; [email protected]; [email protected]; [email protected]; Alistair Popple <[email protected]>; [email protected]; linux- [email protected]; Neo Jia <[email protected]>; Kirti Wankhede <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Zhi Wang <[email protected]>; Dan Williams <[email protected]>; Dheeraj Nigam <[email protected]>; Krishnakant Jaju <[email protected]>
Subject: [PATCH v5 7/7] vfio/nvgrace-gpu: wait for the GPU mem to be ready

From: Ankit Agrawal <[email protected]>

Speculative prefetches from CPU to GPU memory until the GPU is ready after reset can cause harmless corrected RAS events to be logged on Grace systems. It is thus preferred that the mapping not be re-established until the GPU is ready post reset.

The GPU readiness can be checked through BAR0 registers similar to the checking at the time of device probe.

It can take several seconds for the GPU to be ready. So it is desirable that the time overlaps as much of the VM startup as possible to reduce impact on the VM bootup time. The GPU readiness state is thus checked on the first fault/huge_fault request or read/write access which amortizes the GPU readiness time.

The first fault and read/write checks the GPU state when the reset_done flag - which denotes whether the GPU has just been reset. The memory_lock is taken across map/access to avoid races with GPU reset.

cc Alex Williamson <[email protected]>
cc Jason Gunthorpe <[email protected]>
cc Vikram Sethi <[email protected]>
---
 drivers/vfio/pci/nvgrace-gpu/main.c | 79 ++++++++++++++++++++++++++-
--
 1 file changed, 72 insertions(+), 7 deletions(-)
drivers/vfio/pci/nvgrace- renamed from drivers/vfio/pci/nvgrace-gpu/main.c +0 -0

gpu/main.c
index bef9f25bf8f3..fbc19fe688ca 100644

drivers/vfio/pci/nvgrace-gpu/main.c +11 -0
@@ -104,6 +104,17 @@ static int nvgrace_gpu_open_device(struct
 vfio_device *core_vdev)
 		mutex_init(&nvdev->remap_lock);
 	}
 
+	/*
+	 * GPU readiness is checked by reading the BAR0 registers.
+	 *
+	 * ioremap BAR0 to ensure that the BAR0 mapping is present before
+	 * register reads on first fault before establishing any GPU
+	 * memory mapping.
+	 */
+	ret = vfio_pci_core_setup_barmap(vdev, 0);
+	if (ret)
+		return ret;
+
 	vfio_pci_core_finish_enable(vdev);
 
drivers/vfio/pci/nvgrace-gpu/main.c +20 -0
 	return 0;
@@ -150,6 +161,26 @@ static int nvgrace_gpu_wait_device_ready(void
 __iomem *io)
 	return ret;
 }
 
+static int
+nvgrace_gpu_check_device_ready(struct nvgrace_gpu_pci_core_device
 *nvdev)
+{
+	struct vfio_pci_core_device *vdev = &nvdev->core_device;
+	int ret;
+
+	lockdep_assert_held_read(&vdev->memory_lock);
+
+	if (!nvdev->reset_done)
+		return 0;
+
+	ret = nvgrace_gpu_wait_device_ready(vdev->barmap[0]);
+	if (ret)
+		return ret;
+
+	nvdev->reset_done = false;
+
+	return 0;
+}
+
 static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,
drivers/vfio/pci/nvgrace-gpu/main.c +11 -1
 						  unsigned int order)
 {
@@ -173,8 +204,18 @@ static vm_fault_t
 nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,
 		      pfn & ((1 << order) - 1)))
 		return VM_FAULT_FALLBACK;
 
-	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock)
+	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock) {
+		/*
+		 * If the GPU memory is accessed by the CPU while the GPU is
+		 * not ready after reset, it can cause harmless corrected RAS
+		 * events to be logged. Make sure the GPU is ready before
+		 * establishing the mappings.
+		 */
+		if (nvgrace_gpu_check_device_ready(nvdev))
+			return ret;
+
 		ret = vfio_pci_vmf_insert_pfn(vmf, pfn, order);
+	}
 
 	return ret;

}

drivers/vfio/pci/nvgrace-gpu/main.c +15 -3
@@ -593,9 +634,21 @@ nvgrace_gpu_read_mem(struct
 nvgrace_gpu_pci_core_device *nvdev,
 	else
 		mem_count = min(count, memregion->memlength -
 (size_t)offset);
 
-	ret = nvgrace_gpu_map_and_read(nvdev, buf, mem_count, ppos);
-	if (ret)
-		return ret;
+	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock) {
+		/*
+		 * If the GPU memory is accessed by the CPU while the GPU is
+		 * not ready after reset, it can cause harmless corrected RAS
+		 * events to be logged. Make sure the GPU is ready before
+		 * establishing the mappings.
+		 */
+		ret = nvgrace_gpu_check_device_ready(nvdev);
+		if (ret)
+			return ret;
+
+		ret = nvgrace_gpu_map_and_read(nvdev, buf, mem_count,
 ppos);
+		if (ret)
+			return ret;
+	}
drivers/vfio/pci/nvgrace-gpu/main.c +0 -0
 	/*
 	 * Only the device memory present on the hardware is mapped, which

may

drivers/vfio/pci/nvgrace-gpu/main.c +0 -0
@@ -713,9 +766,21 @@ nvgrace_gpu_write_mem(struct
 nvgrace_gpu_pci_core_device *nvdev,
 	 */
 	mem_count = min(count, memregion->memlength - (size_t)offset);
 

drivers/vfio/pci/nvgrace-gpu/main.c nvgrace_gpu_write_mem(struct · line 776

Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c +7 -3
-	ret = nvgrace_gpu_map_and_write(nvdev, buf, mem_count, ppos);
-	if (ret)
-		return ret;
+	scoped_guard(rwsem_read, &nvdev->core_device.memory_lock) {
+		/*
+		 * If the GPU memory is accessed by the CPU while the GPU is
+		 * not ready after reset, it can cause harmless corrected RAS
+		 * events to be logged. Make sure the GPU is ready before
+		 * establishing the mappings.
+		 */

The comment above is now repeated 3 times. Good to consolidate and add that comment above nvgrace_gpu_check_device_ready().

Thanks,
Shameer

Quoted 14 lines
drivers/vfio/pci/nvgrace-gpu/main.c +8 -0
+		ret = nvgrace_gpu_check_device_ready(nvdev);
+		if (ret)
+			return ret;
+
+		ret = nvgrace_gpu_map_and_write(nvdev, buf, mem_count,
 ppos);
+		if (ret)
+			return ret;
+	}
 

exitfn:
*ppos += count;
--
2.34.1