vfio/nvgrace-gpu: Support huge PFNMAP and wait for GPU ready post reset
[email protected]
maintainerv6 cover vfio/nvgrace-gpu: Support huge PFNMAP and wait for GPU ready post reset
From: Ankit Agrawal <[email protected]>
NVIDIA's Grace based system have large GPU device memory. The device memory is mapped as VM_PFNMAP in the VMM VMA. The nvgrace-gpu module could make use of the huge PFNMAP support added in mm [1].
To achieve this, nvgrace-gpu module is updated to implement huge_fault ops. The implementation establishes mapping according to the order request. Note that if the PFN or the VMA address is unaligned to the order, the mapping fallbacks to the PTE level.
Secondly, it is expected that the mapping not be re-established until the GPU is ready post reset. Presence of the mappings during that time could potentially leads to harmless corrected RAS events to be logged if the CPU attempts to do speculative reads on the GPU memory on the Grace systems.
It can take several seconds for the GPU to be ready. So it is desirable that the time overlaps as much of the VM startup as possible to reduce impact on the VM bootup time. The GPU readiness state is thus checked on the first fault/huge_fault request which amortizes the GPU readiness time. The GPU readiness is checked through BAR0 registers as is done at the device probe.
Patch 1 splits the code to map at the various levels.
Patch 2 implements support for huge pfnmap.
Patch 3 vfio_pci_core_mmap cleanup.
Patch 4 split the code to check the device readiness.
Patch 5 reset_done handler implementation
Patch 6 Ensures that the GPU is ready before re-establishing the mapping after reset.
Applied over 6.18-rc6.
Changelog:
v6:
- Updated the vfio_pci_vmf_insert_pfn function to add more common code. (Thanks Shameer Kolothum)
- Added missing userspace offset for pgoff calculation for huge pfnmap. (Thanks Shameer Kolothum)
- Removed alignment for GPU memory. (Thanks Jason Gunthorpe)
- Collected Reviewed-by. (Thanks Shameer Kolothum)
- Miscellaneous cleanup, log messages fixup
(Thanks Alex Williamson, Jason Gunthorpe, Shameer Kolothum)
- Updated gpu_mem_mapped with reset_done flag for clearer semantics. (6/7) (Thanks Alex Williamson)
- Renamed vfio_pci_map_pfn to vfio_pci_vmf_insert_pfn. (2/7) (Thanks Alex Williamson)
- Updated to hold memory_lock across the vmf_insert_pfn and the read/write access of the device. (7/7) (Thanks Alex Williamson)
- Used scoped_guard to simplify critical region. (1/7, 7/7) [v4]
- Implemented reset_done handler to set gpu_mem_mapped flag. Cleaned up FLR detection path (Thanks Alex Williamson)
- Moved the premap check of the device readiness to a new function. Added locking to avoid races. (Thanks Alex Williamson)
- vfio_pci_core_mmap cleanup.
- Added ioremap to BAR0 during open.
- Moved the code for BAR mapping to a separate function.
- Added BAR0 mapping during open. Ensures BAR0 is mapped when registers are checked. (Thanks Alex Williamson, Jason Gunthorpe for suggestion)
- Added check for GPU readiness on nvgrace_gpu_map_device_mem. (Thanks Alex Williamson for the suggestion.
- Fixed build kernel warning
- subject text changes
- Rebased to 6.18-rc6.
Ankit Agrawal (6):
vfio: export function to map the VMA vfio/nvgrace-gpu: Add support for huge pfnmap
vfio: use vfio_pci_core_setup_barmap to map bar in mmap vfio/nvgrace-gpu: split the code to wait for GPU ready vfio/nvgrace-gpu: Inform devmem unmapped after reset vfio/nvgrace-gpu: wait for the GPU mem to be ready
drivers/vfio/pci/nvgrace-gpu/main.c | 196 ++++++++++++++++++++++------ drivers/vfio/pci/vfio_pci_core.c | 65 ++++----- include/linux/vfio_pci_core.h | 3 + 3 files changed, 192 insertions(+), 72 deletions(-)
Signature · 1 line
2.34.1
[email protected]
maintainerPATCH v6 1/6 vfio: export function to map the VMA
From: Ankit Agrawal <[email protected]>
Take out the implementation to map the VMA to the PTE/PMD/PUD as a separate function.
Export the function to be used by nvgrace-gpu module.
--- drivers/vfio/pci/vfio_pci_core.c | 50 ++++++++++++++++++++------------ include/linux/vfio_pci_core.h | 3 ++ 2 files changed, 34 insertions(+), 19 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +31 -19
@@ -1640,31 +1640,21 @@ static unsigned long vma_to_pfn(struct vm_area_struct *vma)return (pci_resource_start (vdev->pdev, index) >>PAGE_SHIFT ) + pgoff; } -static vm_fault_t vfio_pci_mmap_huge_fault (struct vm_fault *vmf, -unsigned int order) +vm_fault_t vfio_pci_vmf_insert_pfn (struct vfio_pci_core_device *vdev, +struct vm_fault *vmf, +unsigned long pfn, +unsigned int order) { -struct vm_area_struct *vma = vmf->vma; -struct vfio_pci_core_device *vdev = vma->vm_private_data; -unsigned long addr = vmf->address & ~((PAGE_SIZE << order) -1 ); -unsigned long pgoff = (addr - vma->vm_start) >>PAGE_SHIFT ; -unsigned long pfn =vma_to_pfn (vma) + pgoff; -vm_fault_t ret =VM_FAULT_SIGBUS ; +vm_fault_t ret; -if (order && (addr < vma->vm_start || - addr + (PAGE_SIZE << order) > vma->vm_end || - pfn & ((1 << order) -1 ))) { - ret =VM_FAULT_FALLBACK ; - goto out; - } - -down_read (&vdev->memory_lock); +lockdep_assert_held_read (&vdev->memory_lock);if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled (vdev)) - goto out_unlock; +return VM_FAULT_SIGBUS ;switch (order) {case 0 : - ret =vmf_insert_pfn (vma, vmf->address, pfn); + ret =vmf_insert_pfn (vmf->vma, vmf->address, pfn);break ;#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP case PMD_ORDER :
@@ -1680,7 +1670,29 @@ static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, ret =VM_FAULT_FALLBACK ; } -out_unlock: +return ret; +} +EXPORT_SYMBOL_GPL (vfio_pci_vmf_insert_pfn); + +static vm_fault_t vfio_pci_mmap_huge_fault (struct vm_fault *vmf, +unsigned int order) +{ +struct vm_area_struct *vma = vmf->vma; +struct vfio_pci_core_device *vdev = vma->vm_private_data; +unsigned long addr = vmf->address & ~((PAGE_SIZE << order) -1 ); +unsigned long pgoff = (addr - vma->vm_start) >>PAGE_SHIFT ; +unsigned long pfn =vma_to_pfn (vma) + pgoff; +vm_fault_t ret =VM_FAULT_SIGBUS ; + +if (order && (addr < vma->vm_start || + addr + (PAGE_SIZE << order) > vma->vm_end || + pfn & ((1 << order) -1 ))) { + ret =VM_FAULT_FALLBACK ; + goto out; + } + +down_read (&vdev->memory_lock); + ret =vfio_pci_vmf_insert_pfn (vdev, vmf, pfn, order);up_read (&vdev->memory_lock); out: dev_dbg_ratelimited(&vdev->pdev->dev,
include/linux/vfio_pci_core.h +3 -0
@@ -119,6 +119,9 @@ ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf, size_t count, loff_t *ppos);ssize_t vfio_pci_core_write (struct vfio_device *core_vdev,const char __user *buf,size_t count,loff_t *ppos); +vm_fault_t vfio_pci_vmf_insert_pfn (struct vfio_pci_core_device *vdev, +struct vm_fault *vmf,unsigned long pfn, +unsigned int order);int vfio_pci_core_mmap (struct vfio_device *core_vdev,struct vm_area_struct *vma);void vfio_pci_core_request (struct vfio_device *core_vdev,unsigned int count);int vfio_pci_core_match (struct vfio_device *core_vdev,char *buf);
Signature · 1 line
2.34.1
Re: [PATCH v6 1/6] vfio: export function to map the VMA
Quoted 7 lines · On Tue, 25 Nov 2025 17:30:08 +0000 <[email protected]> wrote:
From: Ankit Agrawal <[email protected]>
Take out the implementation to map the VMA to the PTE/PMD/PUD as a separate function.
Export the function to be used by nvgrace-gpu module.
This looks more like a re-factor than a simple symbol export. Let's add:
No functional change is intended.
Quoted 3 lines
cc Shameer Kolothum <[email protected]>cc Alex Williamson <[email protected]>cc Jason Gunthorpe <[email protected]>
Nit: I saw "cc" tag is also used somewhere else in the git log. I was suprised that checkpatch.pl doesn't complain about it. I did test it.
In VFIO, people tend to use "Cc:" according to a search of the git log. Let's keep using "Cc:" in VFIO.
Quoted 101 lines
Reviewed-by Shameer Kolothum <[email protected]>Signed-off-by Ankit Agrawal <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 50++++++++++++++++++++------------
include/linux/vfio_pci_core.h|
3 ++ 2 files changed, 34 insertions(+), 19 deletions(-)(quoted diff) +0
-0
b/drivers/vfio/pci/vfio_pci_core.cindex7dcf5439dedc..c445a53ee12e100644 ---a/drivers/vfio/pci/vfio_pci_core.c
+++b/drivers/vfio/pci/vfio_pci_core.c(quoted diff) +0
-0@@ -1640,31 +1640,21 @@ static unsigned long vma_to_pfn(struct
vm_area_struct*vma) return (pci_resource_start(vdev->pdev, index)>>PAGE_SHIFT) + pgoff; }drivers/vfio/pci/vfio_pci_core.c +4
-5-static vm_fault_t vfio_pci_mmap_huge_fault (struct vm_fault *vmf, -unsigned int order) +vm_fault_t vfio_pci_vmf_insert_pfn (struct vfio_pci_core_device *vdev, +struct vm_fault *vmf, +unsigned long pfn, +unsigned int order) { -struct vm_area_struct *vma = vmf->vma; -struct vfio_pci_core_device *vdev = vma->vm_private_data; - unsigned long addr = vmf->address & ~((PAGE_SIZE << order) -1);
drivers/vfio/pci/vfio_pci_core.c +2
-11-unsigned long pgoff = (addr - vma->vm_start) >>PAGE_SHIFT ; -unsigned long pfn =vma_to_pfn (vma) + pgoff; -vm_fault_t ret =VM_FAULT_SIGBUS ; +vm_fault_t ret; -if (order && (addr < vma->vm_start || - addr + (PAGE_SIZE << order) > vma->vm_end || - pfn & ((1 << order) -1 ))) { - ret =VM_FAULT_FALLBACK ; - goto out; - } - -down_read (&vdev->memory_lock); +lockdep_assert_held_read (&vdev->memory_lock); if (vdev->pm_runtime_engaged ||!__vfio_pci_memory_enabled(vdev))
drivers/vfio/pci/vfio_pci_core.c +2
-2- goto out_unlock; +return VM_FAULT_SIGBUS ;switch (order) {case 0 : - ret =vmf_insert_pfn (vma, vmf->address, pfn); + ret =vmf_insert_pfn (vmf->vma, vmf->address, pfn);break ;#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP case PMD_ORDER :@@ -1680,7 +1670,29 @@ static vm_fault_tvfio_pci_mmap_huge_fault(struct
vm_fault*vmf, ret =VM_FAULT_FALLBACK; }drivers/vfio/pci/vfio_pci_core.c +10
-1-out_unlock: +return ret; +} +EXPORT_SYMBOL_GPL (vfio_pci_vmf_insert_pfn); + +static vm_fault_t vfio_pci_mmap_huge_fault (struct vm_fault *vmf, +unsigned int order) +{ +struct vm_area_struct *vma = vmf->vma; +struct vfio_pci_core_device *vdev = vma->vm_private_data; + unsigned long addr = vmf->address & ~((PAGE_SIZE << order) -1);
drivers/vfio/pci/vfio_pci_core.c +13
-0+unsigned long pgoff = (addr - vma->vm_start) >>PAGE_SHIFT ; +unsigned long pfn =vma_to_pfn (vma) + pgoff; +vm_fault_t ret =VM_FAULT_SIGBUS ; + +if (order && (addr < vma->vm_start || + addr + (PAGE_SIZE << order) > vma->vm_end || + pfn & ((1 << order) -1 ))) { + ret =VM_FAULT_FALLBACK ; + goto out; + } + +down_read (&vdev->memory_lock); + ret =vfio_pci_vmf_insert_pfn (vdev, vmf, pfn, order);up_read (&vdev->memory_lock); out: dev_dbg_ratelimited(&vdev->pdev->dev,(quoted diff) +0
-0
b/include/linux/vfio_pci_core.hindexf541044e42a2..6f7c6c0d4278100644 ---a/include/linux/vfio_pci_core.h
+++b/include/linux/vfio_pci_core.h(quoted diff) +3
-0@@ -119,6 +119,9 @@ ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf, size_t count, loff_t *ppos); ssize_t vfio_pci_core_write(struct vfio_device *core_vdev, const char __user *buf, size_t count, loff_t *ppos); +vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev, + struct vm_fault *vmf, unsigned long pfn, + unsigned int order); int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct
vm_area_struct*vma); voidvfio_pci_core_request(struct vfio_device *core_vdev, unsigned int count); intvfio_pci_core_match(struct vfio_device *core_vdev, char *buf);
Alex Williamson
maintainerRe: [PATCH v6 1/6] vfio: export function to map the VMA
In reply to [email protected]Quoted 62 lines · On Tue, 25 Nov 2025 17:30:08 +0000 <[email protected]> wrote:
From: Ankit Agrawal <[email protected]>
Take out the implementation to map the VMA to the PTE/PMD/PUD as a separate function.
Export the function to be used by nvgrace-gpu module.
cc Shameer Kolothum <[email protected]>cc Alex Williamson <[email protected]>cc Jason Gunthorpe <[email protected]>Reviewed-by Shameer Kolothum <[email protected]>Signed-off-by Ankit Agrawal <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 50 ++++++++++++++++++++------------ include/linux/vfio_pci_core.h | 3 ++ 2 files changed, 34 insertions(+), 19 deletions(-)drivers/vfio/pci/vfio_pci_core.c +8
-18@@ -1640,31 +1640,21 @@ static unsigned long vma_to_pfn(struct vm_area_struct *vma)return (pci_resource_start (vdev->pdev, index) >>PAGE_SHIFT ) + pgoff; } -static vm_fault_t vfio_pci_mmap_huge_fault (struct vm_fault *vmf, -unsigned int order) +vm_fault_t vfio_pci_vmf_insert_pfn (struct vfio_pci_core_device *vdev, +struct vm_fault *vmf, +unsigned long pfn, +unsigned int order) { -struct vm_area_struct *vma = vmf->vma; -struct vfio_pci_core_device *vdev = vma->vm_private_data; -unsigned long addr = vmf->address & ~((PAGE_SIZE << order) -1 ); -unsigned long pgoff = (addr - vma->vm_start) >>PAGE_SHIFT ; -unsigned long pfn =vma_to_pfn (vma) + pgoff; -vm_fault_t ret =VM_FAULT_SIGBUS ; +vm_fault_t ret; -if (order && (addr < vma->vm_start || - addr + (PAGE_SIZE << order) > vma->vm_end || - pfn & ((1 << order) -1 ))) { - ret =VM_FAULT_FALLBACK ; - goto out; - } - -down_read (&vdev->memory_lock); +lockdep_assert_held_read (&vdev->memory_lock);if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled (vdev)) - goto out_unlock; +return VM_FAULT_SIGBUS ;switch (order) {case 0 : - ret =vmf_insert_pfn (vma, vmf->address, pfn); + ret =vmf_insert_pfn (vmf->vma, vmf->address, pfn);break ;#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP case PMD_ORDER :
drivers/vfio/pci/vfio_pci_core.c static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, · line 1674
Quoted 6 lines
drivers/vfio/pci/vfio_pci_core.c +2
-1@@ -1680,7 +1670,29 @@ static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, ret =VM_FAULT_FALLBACK ; } -out_unlock: +return ret; +}
At this point we no longer need @ret, we can return directly in all cases.
drivers/vfio/pci/vfio_pci_core.c static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, · line 1685
Quoted 11 lines
drivers/vfio/pci/vfio_pci_core.c +11
-0+EXPORT_SYMBOL_GPL (vfio_pci_vmf_insert_pfn); + +static vm_fault_t vfio_pci_mmap_huge_fault (struct vm_fault *vmf, +unsigned int order) +{ +struct vm_area_struct *vma = vmf->vma; +struct vfio_pci_core_device *vdev = vma->vm_private_data; +unsigned long addr = vmf->address & ~((PAGE_SIZE << order) -1 ); +unsigned long pgoff = (addr - vma->vm_start) >>PAGE_SHIFT ; +unsigned long pfn =vma_to_pfn (vma) + pgoff; +vm_fault_t ret =VM_FAULT_SIGBUS ;
The only use case of this initialization is now in the new function.
drivers/vfio/pci/vfio_pci_core.c static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, · line 1692
Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c +7
-0+ +if (order && (addr < vma->vm_start || + addr + (PAGE_SIZE << order) > vma->vm_end || + pfn & ((1 << order) -1 ))) { + ret =VM_FAULT_FALLBACK ; + goto out; + }
Should we make a static inline in a vfio header for the above to avoid the duplicate implementation in the next patch? Also we might as well use an else branch rather than goto with the bulk of the code moved now. Maybe also just convert to a scoped_guard as well. Thanks,
Alex
Quoted 20 lines
drivers/vfio/pci/vfio_pci_core.c +3
-0+ +down_read (&vdev->memory_lock); + ret =vfio_pci_vmf_insert_pfn (vdev, vmf, pfn, order);up_read (&vdev->memory_lock); out: dev_dbg_ratelimited(&vdev->pdev->dev,include/linux/vfio_pci_core.h +3
-0@@ -119,6 +119,9 @@ ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf, size_t count, loff_t *ppos);ssize_t vfio_pci_core_write (struct vfio_device *core_vdev,const char __user *buf,size_t count,loff_t *ppos); +vm_fault_t vfio_pci_vmf_insert_pfn (struct vfio_pci_core_device *vdev, +struct vm_fault *vmf,unsigned long pfn, +unsigned int order);int vfio_pci_core_mmap (struct vfio_device *core_vdev,struct vm_area_struct *vma);void vfio_pci_core_request (struct vfio_device *core_vdev,unsigned int count);int vfio_pci_core_match (struct vfio_device *core_vdev,char *buf);
[email protected]
maintainerPATCH v6 2/6 vfio/nvgrace-gpu: Add support for huge pfnmap
In reply to [email protected]From: Ankit Agrawal <[email protected]>
NVIDIA's Grace based systems have large device memory. The device memory is mapped as VM_PFNMAP in the VMM VMA. The nvgrace-gpu module could make use of the huge PFNMAP support added in mm [1].
To make use of the huge pfnmap support, fault/huge_fault ops based mapping mechanism needs to be implemented. Currently nvgrace-gpu module relies on remap_pfn_range to do the mapping during VM bootup. Replace it to instead rely on fault and use vfio_pci_vmf_insert_pfn to setup the mapping.
Moreover to enable huge pfnmap, nvgrace-gpu module is updated by adding huge_fault ops implementation. The implementation establishes mapping according to the order request. Note that if the PFN or the VMA address is unaligned to the order, the mapping fallbacks to the PTE level.
--- drivers/vfio/pci/nvgrace-gpu/main.c | 84 +++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 22 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +62 -22
@@ -130,6 +130,62 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)vfio_pci_core_close_device (core_vdev); } +static unsigned long addr_to_pgoff (struct vm_area_struct *vma, +unsigned long addr) +{ +u64 pgoff = vma->vm_pgoff & + ((1U << (VFIO_PCI_OFFSET_SHIFT -PAGE_SHIFT )) -1 ); + +return ((addr - vma->vm_start) >>PAGE_SHIFT ) + pgoff; +} + +static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault (struct vm_fault *vmf, +unsigned int order) +{ +struct vm_area_struct *vma = vmf->vma; +struct nvgrace_gpu_pci_core_device *nvdev = vma->vm_private_data; +struct vfio_pci_core_device *vdev = &nvdev->core_device; +unsigned int index = + vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT -PAGE_SHIFT ); +vm_fault_t ret =VM_FAULT_SIGBUS ; +struct mem_region *memregion; +unsigned long pfn, addr; + + memregion =nvgrace_gpu_memregion (index, nvdev); +if (!memregion) +return ret; + + addr = vmf->address & ~((PAGE_SIZE << order) -1 ); + pfn =PHYS_PFN (memregion->memphys) +addr_to_pgoff (vma, addr); + +if (order && (addr < vma->vm_start || + addr + (PAGE_SIZE << order) > vma->vm_end || + pfn & ((1 << order) -1 ))) +return VM_FAULT_FALLBACK ; + +scoped_guard (rwsem_read, &vdev->memory_lock) + ret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order); + +dev_dbg_ratelimited (&vdev->pdev->dev, +"%s order = %d pfn 0x%lx: 0x%x\n" , + __func__, order, pfn, + (unsigned int )ret); + +return ret; +} + +static vm_fault_t nvgrace_gpu_vfio_pci_fault (struct vm_fault *vmf) +{ +return nvgrace_gpu_vfio_pci_huge_fault (vmf,0 ); +} + +static const struct vm_operations_struct nvgrace_gpu_vfio_pci_mmap_ops = { + .fault = nvgrace_gpu_vfio_pci_fault, +#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP + .huge_fault = nvgrace_gpu_vfio_pci_huge_fault, +#endif +}; +static int nvgrace_gpu_mmap (struct vfio_device *core_vdev,struct vm_area_struct *vma) {
@@ -137,10 +193,8 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev, container_of(core_vdev, struct nvgrace_gpu_pci_core_device, core_device.vdev);struct mem_region *memregion; -unsigned long start_pfn;u64 req_len, pgoff, end;unsigned int index; -int ret =0 ; index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT -PAGE_SHIFT );
@@ -157,17 +211,18 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev, ((1U << (VFIO_PCI_OFFSET_SHIFT -PAGE_SHIFT )) -1 );if (check_sub_overflow (vma->vm_end, vma->vm_start, &req_len) || -check_add_overflow (PHYS_PFN (memregion->memphys), pgoff, &start_pfn) ||check_add_overflow (PFN_PHYS (pgoff), req_len, &end))return -EOVERFLOW ;/* -* Check that the mapping request does not go beyond available device -* memory size +* Check that the mapping request does not go beyond the exposed +* device memory size. */ if (end > memregion->memlength)return -EINVAL ; +vm_flags_set (vma,VM_IO |VM_PFNMAP |VM_DONTEXPAND |VM_DONTDUMP ); +/* * The carved out region of the device memory needs the NORMAL_NC * property. Communicate as such to the hypervisor.
@@ -184,23 +239,8 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev, vma->vm_page_prot =pgprot_writecombine (vma->vm_page_prot); } -/* -* Perform a PFN map to the memory and back the device BAR by the -* GPU memory. -* -* The available GPU memory size may not be power-of-2 aligned. The -* remainder is only backed by vfio_device_ops read/write handlers. -* -* During device reset, the GPU is safely disconnected to the CPU -* and access to the BAR will be immediately returned preventing -* machine check. -*/ - ret =remap_pfn_range (vma, vma->vm_start, start_pfn, - req_len, vma->vm_page_prot); -if (ret) -return ret; - - vma->vm_pgoff = start_pfn; + vma->vm_ops = &nvgrace_gpu_vfio_pci_mmap_ops; + vma->vm_private_data = nvdev;return 0 ; }
Signature · 1 line
2.34.1
Re: [PATCH v6 2/6] vfio/nvgrace-gpu: Add support for huge pfnmap
Quoted 60 lines · On Tue, 25 Nov 2025 17:30:09 +0000 <[email protected]> wrote:
From: Ankit Agrawal <[email protected]>
NVIDIA's Grace based systems have large device memory. The device memory is mapped as
VM_PFNMAPin the VMM VMA. The nvgrace-gpu module could make use of the huge PFNMAP support added in mm [1].To make use of the huge pfnmap support, fault/huge_fault ops based mapping mechanism needs to be implemented. Currently nvgrace-gpu module relies on
remap_pfn_rangeto do the mapping during VM bootup. Replace it to instead rely on fault and usevfio_pci_vmf_insert_pfnto setup the mapping.Moreover to enable huge pfnmap, nvgrace-gpu module is updated by adding
huge_faultops implementation. The implementation establishes mapping according to the order request. Note that if the PFN or the VMA address is unaligned to the order, the mapping fallbacks to the PTE level.Link:
https://lore.kernel.org/all/[email protected]/
[1]cc Shameer Kolothum <[email protected]>cc Alex Williamson <[email protected]>cc Jason Gunthorpe <[email protected]>cc Vikram Sethi <[email protected]>Signed-off-by Ankit Agrawal <[email protected]>--- drivers/vfio/pci/nvgrace-gpu/main.c | 84+++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 22 deletions(-)
(quoted diff) +0
-0
b/drivers/vfio/pci/nvgrace-gpu/main.cindexe346392b72f6..8a982310b188100644 ---a/drivers/vfio/pci/nvgrace-gpu/main.c+++b/drivers/vfio/pci/nvgrace-gpu/main.c@@ -130,6 +130,62 @@ static voidnvgrace_gpu_close_device(struct vfio_device *core_vdev)vfio_pci_core_close_device(core_vdev); }drivers/vfio/pci/nvgrace-gpu/main.c +10
-0+static unsigned long addr_to_pgoff (struct vm_area_struct *vma, +unsigned long addr) +{ +u64 pgoff = vma->vm_pgoff & + ((1U << (VFIO_PCI_OFFSET_SHIFT -PAGE_SHIFT )) -1 ); + +return ((addr - vma->vm_start) >>PAGE_SHIFT ) + pgoff; +} + +static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault*vmf,
drivers/vfio/pci/nvgrace-gpu/main.c +4
-0+ unsigned int order) +{ +struct vm_area_struct *vma = vmf->vma; + struct nvgrace_gpu_pci_core_device *nvdev =
vma->vm_private_data;drivers/vfio/pci/nvgrace-gpu/main.c +3
-0+struct vfio_pci_core_device *vdev = &nvdev->core_device; +unsigned int index = + vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT -
PAGE_SHIFT);
drivers/vfio/pci/nvgrace-gpu/main.c static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) · line 158
Quoted 9 lines
drivers/vfio/pci/nvgrace-gpu/main.c +9
-0+vm_fault_t ret =VM_FAULT_SIGBUS ; +struct mem_region *memregion; +unsigned long pfn, addr; + + memregion =nvgrace_gpu_memregion (index, nvdev); +if (!memregion) +return ret; + + addr = vmf->address & ~((PAGE_SIZE << order) -1 );
ALIGN_DOWN(vmf->address, PAGE_SIZE << order).
drivers/vfio/pci/nvgrace-gpu/main.c static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) · line 163
Quoted 5 lines
(quoted diff) +1
-0+ pfn = PHYS_PFN(memregion->memphys) + addr_to_pgoff(vma,addr); +
drivers/vfio/pci/nvgrace-gpu/main.c +3
-0+if (order && (addr < vma->vm_start || + addr + (PAGE_SIZE << order) > vma->vm_end || + pfn & ((1 << order) -1 )))
!IS_ALIGNED(pfn, 1 << order).
Other parts looks good to me.
Alex Williamson
maintainerRe: [PATCH v6 2/6] vfio/nvgrace-gpu: Add support for huge pfnmap
In reply to [email protected]Quoted 59 lines · On Tue, 25 Nov 2025 17:30:09 +0000 <[email protected]> wrote:
From: Ankit Agrawal <[email protected]>
NVIDIA's Grace based systems have large device memory. The device memory is mapped as
VM_PFNMAPin the VMM VMA. The nvgrace-gpu module could make use of the huge PFNMAP support added in mm [1].To make use of the huge pfnmap support, fault/huge_fault ops based mapping mechanism needs to be implemented. Currently nvgrace-gpu module relies on
remap_pfn_rangeto do the mapping during VM bootup. Replace it to instead rely on fault and usevfio_pci_vmf_insert_pfnto setup the mapping.Moreover to enable huge pfnmap, nvgrace-gpu module is updated by adding
huge_faultops implementation. The implementation establishes mapping according to the order request. Note that if the PFN or the VMA address is unaligned to the order, the mapping fallbacks to the PTE level.cc Shameer Kolothum <[email protected]>cc Alex Williamson <[email protected]>cc Jason Gunthorpe <[email protected]>cc Vikram Sethi <[email protected]>Signed-off-by Ankit Agrawal <[email protected]>--- drivers/vfio/pci/nvgrace-gpu/main.c | 84 +++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 22 deletions(-)drivers/vfio/pci/nvgrace-gpu/main.c +22
-0@@ -130,6 +130,62 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)vfio_pci_core_close_device (core_vdev); } +static unsigned long addr_to_pgoff (struct vm_area_struct *vma, +unsigned long addr) +{ +u64 pgoff = vma->vm_pgoff & + ((1U << (VFIO_PCI_OFFSET_SHIFT -PAGE_SHIFT )) -1 ); + +return ((addr - vma->vm_start) >>PAGE_SHIFT ) + pgoff; +} + +static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault (struct vm_fault *vmf, +unsigned int order) +{ +struct vm_area_struct *vma = vmf->vma; +struct nvgrace_gpu_pci_core_device *nvdev = vma->vm_private_data; +struct vfio_pci_core_device *vdev = &nvdev->core_device; +unsigned int index = + vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT -PAGE_SHIFT ); +vm_fault_t ret =VM_FAULT_SIGBUS ; +struct mem_region *memregion; +unsigned long pfn, addr; + + memregion =nvgrace_gpu_memregion (index, nvdev);
drivers/vfio/pci/nvgrace-gpu/main.c static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) · line 164
Quoted 10 lines
drivers/vfio/pci/nvgrace-gpu/main.c +10
-0+if (!memregion) +return ret; + + addr = vmf->address & ~((PAGE_SIZE << order) -1 ); + pfn =PHYS_PFN (memregion->memphys) +addr_to_pgoff (vma, addr); + +if (order && (addr < vma->vm_start || + addr + (PAGE_SIZE << order) > vma->vm_end || + pfn & ((1 << order) -1 ))) +return VM_FAULT_FALLBACK ;
The dev_dbg misses this fallback this way. Thanks,
Alex
Quoted 86 lines
drivers/vfio/pci/nvgrace-gpu/main.c +30
-22+ +scoped_guard (rwsem_read, &vdev->memory_lock) + ret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order); + +dev_dbg_ratelimited (&vdev->pdev->dev, +"%s order = %d pfn 0x%lx: 0x%x\n" , + __func__, order, pfn, + (unsigned int )ret); + +return ret; +} + +static vm_fault_t nvgrace_gpu_vfio_pci_fault (struct vm_fault *vmf) +{ +return nvgrace_gpu_vfio_pci_huge_fault (vmf,0 ); +} + +static const struct vm_operations_struct nvgrace_gpu_vfio_pci_mmap_ops = { + .fault = nvgrace_gpu_vfio_pci_fault, +#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP + .huge_fault = nvgrace_gpu_vfio_pci_huge_fault, +#endif +}; +static int nvgrace_gpu_mmap (struct vfio_device *core_vdev,struct vm_area_struct *vma) {@@ -137,10 +193,8 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev, container_of(core_vdev, struct nvgrace_gpu_pci_core_device, core_device.vdev);struct mem_region *memregion; -unsigned long start_pfn;u64 req_len, pgoff, end;unsigned int index; -int ret =0 ; index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT -PAGE_SHIFT );@@ -157,17 +211,18 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev, ((1U << (VFIO_PCI_OFFSET_SHIFT -PAGE_SHIFT )) -1 );if (check_sub_overflow (vma->vm_end, vma->vm_start, &req_len) || -check_add_overflow (PHYS_PFN (memregion->memphys), pgoff, &start_pfn) ||check_add_overflow (PFN_PHYS (pgoff), req_len, &end))return -EOVERFLOW ;/* -* Check that the mapping request does not go beyond available device -* memory size +* Check that the mapping request does not go beyond the exposed +* device memory size. */ if (end > memregion->memlength)return -EINVAL ; +vm_flags_set (vma,VM_IO |VM_PFNMAP |VM_DONTEXPAND |VM_DONTDUMP ); +/* * The carved out region of the device memory needs the NORMAL_NC * property. Communicate as such to the hypervisor. @@ -184,23 +239,8 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev, vma->vm_page_prot =pgprot_writecombine (vma->vm_page_prot); } -/* -* Perform a PFN map to the memory and back the device BAR by the -* GPU memory. -* -* The available GPU memory size may not be power-of-2 aligned. The -* remainder is only backed by vfio_device_ops read/write handlers. -* -* During device reset, the GPU is safely disconnected to the CPU -* and access to the BAR will be immediately returned preventing -* machine check. -*/ - ret =remap_pfn_range (vma, vma->vm_start, start_pfn, - req_len, vma->vm_page_prot); -if (ret) -return ret; - - vma->vm_pgoff = start_pfn; + vma->vm_ops = &nvgrace_gpu_vfio_pci_mmap_ops; + vma->vm_private_data = nvdev;return 0 ; }
[email protected]
maintainerPATCH v6 3/6 vfio: use vfio_pci_core_setup_barmap to map bar in mmap
In reply to [email protected]From: Ankit Agrawal <[email protected]>
Remove code duplication in vfio_pci_core_mmap by calling vfio_pci_core_setup_barmap to perform the bar mapping.
--- drivers/vfio/pci/vfio_pci_core.c | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +3 -12
@@ -1761,18 +1761,9 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma* Even though we don't make use of the barmap for the mmap, * we need to request the region and the barmap tracks that. */ -if (!vdev->barmap[index]) { - ret =pci_request_selected_regions (pdev, -1 << index,"vfio-pci" ); -if (ret) -return ret; - - vdev->barmap[index] =pci_iomap (pdev, index,0 ); -if (!vdev->barmap[index]) { -pci_release_selected_regions (pdev,1 << index); -return -ENOMEM ; - } - } + ret =vfio_pci_core_setup_barmap (vdev, index); +if (ret) +return ret; vma->vm_private_data = vdev; vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot);
Signature · 1 line
2.34.1
Re: [PATCH v6 3/6] vfio: use vfio_pci_core_setup_barmap to map bar in mmap
On Tue, 25 Nov 2025 17:30:10 +0000
<[email protected]> wrote:
LGTM.
Quoted 42 lines
From: Ankit Agrawal <[email protected]>
Remove code duplication in
vfio_pci_core_mmapby callingvfio_pci_core_setup_barmapto perform the bar mapping.cc Donald Dutile <[email protected]>Reviewed-by Shameer Kolothum <[email protected]>Suggested-by Alex Williamson <[email protected]>Signed-off-by Ankit Agrawal <[email protected]>--- drivers/vfio/pci/vfio_pci_core.c | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-)(quoted diff) +0
-0
b/drivers/vfio/pci/vfio_pci_core.cindexc445a53ee12e..3cc799eb75ea100644 ---a/drivers/vfio/pci/vfio_pci_core.c
+++b/drivers/vfio/pci/vfio_pci_core.cdrivers/vfio/pci/vfio_pci_core.c +3
-12@@ -1761,18 +1761,9 @@ int vfio_pci_core_mmap(struct vfio_device*core_vdev, struct vm_area_struct *vma * Even though we don't make use of the barmap for the mmap, * we need to request the region and the barmap tracks that. */ -if (!vdev->barmap[index]) { - ret =pci_request_selected_regions (pdev, -1 << index, "vfio-pci"); -if (ret) -return ret; - - vdev->barmap[index] =pci_iomap (pdev, index,0 ); -if (!vdev->barmap[index]) { -pci_release_selected_regions (pdev,1 << index); -return -ENOMEM ; - } - } + ret =vfio_pci_core_setup_barmap (vdev, index); +if (ret) +return ret;drivers/vfio/pci/vfio_pci_core.c +0
-0vma->vm_private_data = vdev; vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot);
[email protected]
maintainerPATCH v6 4/6 vfio/nvgrace-gpu: split the code to wait for GPU ready
In reply to [email protected]From: Ankit Agrawal <[email protected]>
Split the function that check for the GPU device being ready on the probe.
Move the code to wait for the GPU to be ready through BAR0 register reads to a separate function. This would help reuse the code.
--- drivers/vfio/pci/nvgrace-gpu/main.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +17 -12
@@ -130,6 +130,20 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)vfio_pci_core_close_device (core_vdev); } +static int nvgrace_gpu_wait_device_ready (void __iomem *io) +{ +unsigned long timeout = jiffies +msecs_to_jiffies (POLL_TIMEOUT_MS ); + +do { +if ((ioread32 (io +C2C_LINK_BAR0_OFFSET ) ==STATUS_READY ) && + (ioread32 (io +HBM_TRAINING_BAR0_OFFSET ) ==STATUS_READY )) +return 0 ; +msleep (POLL_QUANTUM_MS ); + }while (!time_after (jiffies, timeout)); + +return -ETIME ; +} +static unsigned long addr_to_pgoff (struct vm_area_struct *vma,unsigned long addr) {
@@ -933,9 +947,8 @@ static bool nvgrace_gpu_has_mig_hw_bug(struct pci_dev *pdev)* Ensure that the BAR0 region is enabled before accessing the * registers. */ -static int nvgrace_gpu_wait_device_ready (struct pci_dev *pdev) +static int nvgrace_gpu_probe_check_device_ready (struct pci_dev *pdev) { -unsigned long timeout = jiffies +msecs_to_jiffies (POLL_TIMEOUT_MS );void __iomem *io;int ret = -ETIME ;
@@ -953,16 +966,8 @@ static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev) goto iomap_exit; } -do { -if ((ioread32 (io +C2C_LINK_BAR0_OFFSET ) ==STATUS_READY ) && - (ioread32 (io +HBM_TRAINING_BAR0_OFFSET ) ==STATUS_READY )) { - ret =0 ; - goto reg_check_exit; - } -msleep (POLL_QUANTUM_MS ); - }while (!time_after (jiffies, timeout)); + ret =nvgrace_gpu_wait_device_ready (io); -reg_check_exit:pci_iounmap (pdev, io); iomap_exit:pci_release_selected_regions (pdev,1 <<0 );
@@ -979,7 +984,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev,u64 memphys, memlength;int ret; - ret =nvgrace_gpu_wait_device_ready (pdev); + ret =nvgrace_gpu_probe_check_device_ready (pdev);if (ret)return ret;
Signature · 1 line
2.34.1
Re: [PATCH v6 4/6] vfio/nvgrace-gpu: split the code to wait for GPU ready
On Tue, 25 Nov 2025 17:30:11 +0000
<[email protected]> wrote:
Looking good to me.
Quoted 83 lines
From: Ankit Agrawal <[email protected]>
Split the function that check for the GPU device being ready on the probe.
Move the code to wait for the GPU to be ready through BAR0 register reads to a separate function. This would help reuse the code.
Reviewed-by Shameer Kolothum <[email protected]>Signed-off-by Ankit Agrawal <[email protected]>--- drivers/vfio/pci/nvgrace-gpu/main.c | 29+++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)
(quoted diff) +0
-0
b/drivers/vfio/pci/nvgrace-gpu/main.cindex8a982310b188..2b736cb82f38100644 ---a/drivers/vfio/pci/nvgrace-gpu/main.c+++b/drivers/vfio/pci/nvgrace-gpu/main.c@@ -130,6 +130,20 @@ static voidnvgrace_gpu_close_device(struct vfio_device *core_vdev)vfio_pci_core_close_device(core_vdev); }drivers/vfio/pci/nvgrace-gpu/main.c +3
-0+static int nvgrace_gpu_wait_device_ready (void __iomem *io) +{ + unsigned long timeout = jiffies +
msecs_to_jiffies(POLL_TIMEOUT_MS); +(quoted diff) +2
-0+ do { + if ((ioread32(io + C2C_LINK_BAR0_OFFSET) ==
STATUS_READY) &&(quoted diff) +1
-0+ (ioread32(io + HBM_TRAINING_BAR0_OFFSET) ==
STATUS_READY))drivers/vfio/pci/nvgrace-gpu/main.c +8
-4+return 0 ; +msleep (POLL_QUANTUM_MS ); + }while (!time_after (jiffies, timeout)); + +return -ETIME ; +} +static unsigned long addr_to_pgoff (struct vm_area_struct *vma,unsigned long addr) {@@ -933,9 +947,8 @@ static bool nvgrace_gpu_has_mig_hw_bug(structpci_dev *pdev) * Ensure that the BAR0 region is enabled before accessing the * registers. */ -static int nvgrace_gpu_wait_device_ready (struct pci_dev *pdev) +static int nvgrace_gpu_probe_check_device_ready (struct pci_dev *pdev) { -unsigned long timeout = jiffies +msecs_to_jiffies (POLL_TIMEOUT_MS );void __iomem *io;int ret = -ETIME ;@@ -953,16 +966,8 @@ static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev) goto iomap_exit; } - do { - if ((ioread32(io + C2C_LINK_BAR0_OFFSET) ==
STATUS_READY) &&(quoted diff) +0
-1- (ioread32(io + HBM_TRAINING_BAR0_OFFSET) ==
STATUS_READY)) {drivers/vfio/pci/nvgrace-gpu/main.c +2
-7- ret =0 ; - goto reg_check_exit; - } -msleep (POLL_QUANTUM_MS ); - }while (!time_after (jiffies, timeout)); + ret =nvgrace_gpu_wait_device_ready (io); -reg_check_exit:pci_iounmap (pdev, io); iomap_exit:pci_release_selected_regions (pdev,1 <<0 );@@ -979,7 +984,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev,u64 memphys, memlength;int ret; - ret =nvgrace_gpu_wait_device_ready (pdev); + ret =nvgrace_gpu_probe_check_device_ready (pdev);if (ret)return ret;
Alex Williamson
maintainerRe: [PATCH v6 4/6] vfio/nvgrace-gpu: split the code to wait for GPU ready
In reply to [email protected]Quoted 10 lines · On Tue, 25 Nov 2025 17:30:11 +0000 <[email protected]> wrote:
From: Ankit Agrawal <[email protected]>
Split the function that check for the GPU device being ready on the probe.
Move the code to wait for the GPU to be ready through BAR0 register reads to a separate function. This would help reuse the code.
Reviewed-by Shameer Kolothum <[email protected]>Signed-off-by Ankit Agrawal <[email protected]>
As noted last round:
And note the fix in the commit log.
Quoted 30 lines
--- drivers/vfio/pci/nvgrace-gpu/main.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)drivers/vfio/pci/nvgrace-gpu/main.c +14
-0@@ -130,6 +130,20 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev)vfio_pci_core_close_device (core_vdev); } +static int nvgrace_gpu_wait_device_ready (void __iomem *io) +{ +unsigned long timeout = jiffies +msecs_to_jiffies (POLL_TIMEOUT_MS ); + +do { +if ((ioread32 (io +C2C_LINK_BAR0_OFFSET ) ==STATUS_READY ) && + (ioread32 (io +HBM_TRAINING_BAR0_OFFSET ) ==STATUS_READY )) +return 0 ; +msleep (POLL_QUANTUM_MS ); + }while (!time_after (jiffies, timeout)); + +return -ETIME ; +} +static unsigned long addr_to_pgoff (struct vm_area_struct *vma,unsigned long addr) {
drivers/vfio/pci/nvgrace-gpu/main.c static bool nvgrace_gpu_has_mig_hw_bug(struct pci_dev *pdev) · line 953
Quoted 9 lines
drivers/vfio/pci/nvgrace-gpu/main.c +1
-2@@ -933,9 +947,8 @@ static bool nvgrace_gpu_has_mig_hw_bug(struct pci_dev *pdev)* Ensure that the BAR0 region is enabled before accessing the * registers. */ -static int nvgrace_gpu_wait_device_ready (struct pci_dev *pdev) +static int nvgrace_gpu_probe_check_device_ready (struct pci_dev *pdev) { -unsigned long timeout = jiffies +msecs_to_jiffies (POLL_TIMEOUT_MS );void __iomem *io;int ret = -ETIME ;
And this initialization is unnecessary. Thanks,
Alex
Quoted 28 lines
drivers/vfio/pci/nvgrace-gpu/main.c +2
-10@@ -953,16 +966,8 @@ static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev) goto iomap_exit; } -do { -if ((ioread32 (io +C2C_LINK_BAR0_OFFSET ) ==STATUS_READY ) && - (ioread32 (io +HBM_TRAINING_BAR0_OFFSET ) ==STATUS_READY )) { - ret =0 ; - goto reg_check_exit; - } -msleep (POLL_QUANTUM_MS ); - }while (!time_after (jiffies, timeout)); + ret =nvgrace_gpu_wait_device_ready (io); -reg_check_exit:pci_iounmap (pdev, io); iomap_exit:pci_release_selected_regions (pdev,1 <<0 );@@ -979,7 +984,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev,u64 memphys, memlength;int ret; - ret =nvgrace_gpu_wait_device_ready (pdev); + ret =nvgrace_gpu_probe_check_device_ready (pdev);if (ret)return ret;
[email protected]
maintainerPATCH v6 5/6 vfio/nvgrace-gpu: Inform devmem unmapped after reset
In reply to [email protected]From: Ankit Agrawal <[email protected]>
Introduce a new flag reset_done to notify that the GPU has just been reset and the mapping to the GPU memory is zapped.
Implement the reset_done handler to set this new variable. It will be used later in the patches to wait for the GPU memory to be ready before doing any mapping or access.
--- drivers/vfio/pci/nvgrace-gpu/main.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)
drivers/vfio/pci/nvgrace-gpu/main.c +18 -1
@@ -58,6 +58,8 @@ struct nvgrace_gpu_pci_core_device {/* Lock to control device memory kernel mapping */ struct mutex remap_lock;bool has_mig_hw_bug; +/* GPU has just been reset */ +bool reset_done; };static void nvgrace_gpu_init_fake_bar_emu_regs (struct vfio_device *core_vdev)
@@ -1047,12 +1049,27 @@ static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = {MODULE_DEVICE_TABLE (pci, nvgrace_gpu_vfio_pci_table); +static void nvgrace_gpu_vfio_pci_reset_done (struct pci_dev *pdev) +{ +struct vfio_pci_core_device *core_device =dev_get_drvdata (&pdev->dev); +struct nvgrace_gpu_pci_core_device *nvdev = + container_of(core_device, struct nvgrace_gpu_pci_core_device, + core_device); + + nvdev->reset_done = true; +} + +static const struct pci_error_handlers nvgrace_gpu_vfio_pci_err_handlers = { + .reset_done = nvgrace_gpu_vfio_pci_reset_done, + .error_detected = vfio_pci_core_aer_err_detected, +}; +static struct pci_driver nvgrace_gpu_vfio_pci_driver = { .name =KBUILD_MODNAME , .id_table = nvgrace_gpu_vfio_pci_table, .probe = nvgrace_gpu_probe, .remove = nvgrace_gpu_remove, - .err_handler = &vfio_pci_core_err_handlers, + .err_handler = &nvgrace_gpu_vfio_pci_err_handlers, .driver_managed_dma = true, };
Signature · 1 line
2.34.1
Alex Williamson
maintainerRe: [PATCH v6 5/6] vfio/nvgrace-gpu: Inform devmem unmapped after reset
Quoted 30 lines · On Tue, 25 Nov 2025 17:30:12 +0000 <[email protected]> wrote:
From: Ankit Agrawal <[email protected]>
Introduce a new flag
reset_doneto notify that the GPU has just been reset and the mapping to the GPU memory is zapped.Implement the
reset_donehandler to set this new variable. It will be used later in the patches to wait for the GPU memory to be ready before doing any mapping or access.cc Jason Gunthorpe <[email protected]>Suggested-by Alex Williamson <[email protected]>Signed-off-by Ankit Agrawal <[email protected]>--- drivers/vfio/pci/nvgrace-gpu/main.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)drivers/vfio/pci/nvgrace-gpu/main.c +2
-0@@ -58,6 +58,8 @@ struct nvgrace_gpu_pci_core_device {/* Lock to control device memory kernel mapping */ struct mutex remap_lock;bool has_mig_hw_bug; +/* GPU has just been reset */ +bool reset_done; };static void nvgrace_gpu_init_fake_bar_emu_regs (struct vfio_device *core_vdev)
drivers/vfio/pci/nvgrace-gpu/main.c static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = { · line 1051
Quoted 3 lines
drivers/vfio/pci/nvgrace-gpu/main.c +0
-0@@ -1047,12 +1049,27 @@ static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = {MODULE_DEVICE_TABLE (pci, nvgrace_gpu_vfio_pci_table);
/*
* Comment explaining why this can't use lockdep_assert_held_write but
* in vfio use cases relies on this for serialization against faults and
* read/write.
*/
Thanks,
Alex
Quoted 25 lines
drivers/vfio/pci/nvgrace-gpu/main.c +16
-1+static void nvgrace_gpu_vfio_pci_reset_done (struct pci_dev *pdev) +{ +struct vfio_pci_core_device *core_device =dev_get_drvdata (&pdev->dev); +struct nvgrace_gpu_pci_core_device *nvdev = + container_of(core_device, struct nvgrace_gpu_pci_core_device, + core_device); + + nvdev->reset_done = true; +} + +static const struct pci_error_handlers nvgrace_gpu_vfio_pci_err_handlers = { + .reset_done = nvgrace_gpu_vfio_pci_reset_done, + .error_detected = vfio_pci_core_aer_err_detected, +}; +static struct pci_driver nvgrace_gpu_vfio_pci_driver = { .name =KBUILD_MODNAME , .id_table = nvgrace_gpu_vfio_pci_table, .probe = nvgrace_gpu_probe, .remove = nvgrace_gpu_remove, - .err_handler = &vfio_pci_core_err_handlers, + .err_handler = &nvgrace_gpu_vfio_pci_err_handlers, .driver_managed_dma = true, };
Ankit Agrawal
maintainerRe: [PATCH v6 5/6] vfio/nvgrace-gpu: Inform devmem unmapped after reset
Quoted 8 lines
Quoted 2 lines
MODULE_DEVICE_TABLE(pci, nvgrace_gpu_vfio_pci_table);/*
* Comment explaining why this can't uselockdep_assert_held_writebut
* in vfio use cases relies on this for serialization against faults and
* read/write.
*/
In this patch or the next where we actually do the serialization with memory_lock?
Quoted 2 lines
Thanks,
Alex
Alex Williamson
maintainerRe: [PATCH v6 5/6] vfio/nvgrace-gpu: Inform devmem unmapped after reset
Quoted 11 lines · On Wed, 26 Nov 2025 03:26:54 +0000 Ankit Agrawal <[email protected]> wrote:
Quoted 8 lines
Quoted 2 lines
MODULE_DEVICE_TABLE(pci, nvgrace_gpu_vfio_pci_table);/*
* Comment explaining why this can't uselockdep_assert_held_writebut
* in vfio use cases relies on this for serialization against faults and
* read/write.
*/In this patch or the next where we actually do the serialization with
memory_lock?
When the interaction is added would make more sense. Thanks,
Alex
[email protected]
maintainerPATCH v6 6/6 vfio/nvgrace-gpu: wait for the GPU mem to be ready
In reply to [email protected]From: Ankit Agrawal <[email protected]>
Speculative prefetches from CPU to GPU memory until the GPU is ready after reset can cause harmless corrected RAS events to be logged on Grace systems. It is thus preferred that the mapping not be re-established until the GPU is ready post reset.
The GPU readiness can be checked through BAR0 registers similar to the checking at the time of device probe.
It can take several seconds for the GPU to be ready. So it is desirable that the time overlaps as much of the VM startup as possible to reduce impact on the VM bootup time. The GPU readiness state is thus checked on the first fault/huge_fault request or read/write access which amortizes the GPU readiness time.
The first fault and read/write checks the GPU state when the reset_done flag - which denotes whether the GPU has just been reset. The memory_lock is taken across map/access to avoid races with GPU reset.
--- drivers/vfio/pci/nvgrace-gpu/main.c | 66 ++++++++++++++++++++++++++--- 1 file changed, 59 insertions(+), 7 deletions(-)
drivers/vfio/pci/nvgrace-gpu/main.c +59 -7
@@ -104,6 +104,17 @@ static int nvgrace_gpu_open_device(struct vfio_device *core_vdev)mutex_init (&nvdev->remap_lock); } +/* +* GPU readiness is checked by reading the BAR0 registers. +* +* ioremap BAR0 to ensure that the BAR0 mapping is present before +* register reads on first fault before establishing any GPU +* memory mapping. +*/ + ret =vfio_pci_core_setup_barmap (vdev,0 ); +if (ret) +return ret; +vfio_pci_core_finish_enable (vdev);return 0 ;
@@ -146,6 +157,31 @@ static int nvgrace_gpu_wait_device_ready(void __iomem *io)return -ETIME ; } +/* +* If the GPU memory is accessed by the CPU while the GPU is not ready +* after reset, it can cause harmless corrected RAS events to be logged. +* Make sure the GPU is ready before establishing the mappings. +*/ +static int +nvgrace_gpu_check_device_ready (struct nvgrace_gpu_pci_core_device *nvdev) +{ +struct vfio_pci_core_device *vdev = &nvdev->core_device; +int ret; + +lockdep_assert_held_read (&vdev->memory_lock); + +if (!nvdev->reset_done) +return 0 ; + + ret =nvgrace_gpu_wait_device_ready (vdev->barmap[0 ]); +if (ret) +return ret; + + nvdev->reset_done = false; + +return 0 ; +} +static unsigned long addr_to_pgoff (struct vm_area_struct *vma,unsigned long addr) {
@@ -179,8 +215,12 @@ static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf, pfn & ((1 << order) - 1))) return VM_FAULT_FALLBACK; - scoped_guard(rwsem_read, &vdev->memory_lock) + scoped_guard(rwsem_read, &vdev->memory_lock) { +if (nvgrace_gpu_check_device_ready (nvdev)) +return ret; + ret =vfio_pci_vmf_insert_pfn (vdev, vmf, pfn, order); + } dev_dbg_ratelimited(&vdev->pdev->dev, "%s order = %d pfn 0x%lx: 0x%x\n",
@@ -592,9 +632,15 @@ nvgrace_gpu_read_mem(struct nvgrace_gpu_pci_core_device *nvdev,else mem_count =min (count, memregion->memlength - (size_t )offset); - ret =nvgrace_gpu_map_and_read (nvdev, buf, mem_count, ppos); -if (ret) -return ret; + scoped_guard(rwsem_read, &nvdev->core_device.memory_lock) { + ret =nvgrace_gpu_check_device_ready (nvdev); +if (ret) +return ret; + + ret =nvgrace_gpu_map_and_read (nvdev, buf, mem_count, ppos); +if (ret) +return ret; + }/* * Only the device memory present on the hardware is mapped, which may
@@ -712,9 +758,15 @@ nvgrace_gpu_write_mem(struct nvgrace_gpu_pci_core_device *nvdev,*/ mem_count =min (count, memregion->memlength - (size_t )offset); - ret =nvgrace_gpu_map_and_write (nvdev, buf, mem_count, ppos); -if (ret) -return ret; + scoped_guard(rwsem_read, &nvdev->core_device.memory_lock) { + ret =nvgrace_gpu_check_device_ready (nvdev); +if (ret) +return ret; + + ret =nvgrace_gpu_map_and_write (nvdev, buf, mem_count, ppos); +if (ret) +return ret; + } exitfn: *ppos += count;
Signature · 1 line
2.34.1
Re: [PATCH v6 6/6] vfio/nvgrace-gpu: wait for the GPU mem to be ready
Quoted 17 lines · On Tue, 25 Nov 2025 17:30:13 +0000 <[email protected]> wrote:
From: Ankit Agrawal <[email protected]>
Speculative prefetches from CPU to GPU memory until the GPU is ready after reset can cause harmless corrected RAS events to be logged on Grace systems. It is thus preferred that the mapping not be re-established until the GPU is ready post reset.
The GPU readiness can be checked through BAR0 registers similar to the checking at the time of device probe.
It can take several seconds for the GPU to be ready. So it is desirable that the time overlaps as much of the VM startup as possible to reduce impact on the VM bootup time. The GPU readiness state is thus checked on the first fault/huge_fault request or read/write access which amortizes the GPU readiness time.
snip
drivers/vfio/pci/nvgrace-gpu/main.c static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf, · line 221
Quoted 9 lines
(quoted diff) +0
-0@@ -179,8 +215,12 @@ static vm_fault_t
nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf, pfn & ((1 << order) - 1))) returnVM_FAULT_FALLBACK;drivers/vfio/pci/nvgrace-gpu/main.c +4
-1-scoped_guard (rwsem_read, &vdev->memory_lock) + scoped_guard(rwsem_read, &vdev->memory_lock) { +if (nvgrace_gpu_check_device_ready (nvdev)) +return ret; +
I would suggest opening the error code if we don't have a "bailing out without touching the ret" similar to vfio_pci_mmap_huge_fault(), since this looks unnecessarily confusing.
Please also fix the same in PATCH 2.
Quoted 6 lines
drivers/vfio/pci/nvgrace-gpu/main.c +1
-0ret =vfio_pci_vmf_insert_pfn (vdev, vmf, pfn, order); + } dev_dbg_ratelimited(&vdev->pdev->dev, "%s order = %d pfn 0x%lx: 0x%x\n",@@ -592,9 +632,15 @@ nvgrace_gpu_read_mem(struct