Skip to contentSkip to conversation

Expose GPU memory as coherently CPU accessible

PATCH In review [email protected] 11 participants 31 messages Apr 5, 2023 – Jul 25, 2023

Subsystems (12)
Memory management KVM ACPI MM core +8 more subsystems
lore
Share & tools
lore
#0 lore raw

v3 cover Expose GPU memory as coherently CPU accessible

From: Ankit Agrawal <[email protected]>

NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device for the on-chip GPU that is the logical OS representation of the internal propritary cache coherent interconnect.

This representation has a number of limitations compared to a real PCI device, in particular, it does not model the coherent GPU memory aperture as a PCI config space BAR, and PCI doesn't know anything about cacheable memory types.

Provide a VFIO PCI variant driver that adapts the unique PCI representation into a more standard PCI representation facing userspace. The GPU memory aperture is obtained from ACPI, according to the FW specification, and exported to userspace as the VFIO_REGION that covers the first PCI BAR. qemu will naturally generate a PCI device in the VM where the cacheable aperture is reported in BAR1.

Since this memory region is actually cache coherent with the CPU, the VFIO variant driver will mmap it into VMA using a cacheable mapping.

As this is the first time an ARM environment has placed cacheable non-struct page backed memory (eg from remap_pfn_range) into a KVM page table, fix a bug in ARM KVM where it does not copy the cacheable memory attributes from non-struct page backed PTEs to ensure the guest also gets a cacheable mapping.

Finally, the cacheable memory can participate in memory failure handling. ECC failures on this memory will trigger the normal ARM mechanism to get into memory-failure.c. Since this memory is not backed by struct page create a mechanism to route the memory-failure's physical address to the VMA owner so that a SIGBUS can be generated toward the correct process. This works with the existing KVM/qemu handling for memory failure reporting toward a guest.

This goes along with a qemu series to provides the necessary implementation of the Grace Hopper Superchip firmware specification so that the guest operating system can see the correct ACPI modeling for the coherent GPU device.
https://github.com/qemu/qemu/compare/master...ankita-nv:qemu:dev-ankit/cohmem-0330

Applied and tested over v6.3-rc4.

Ankit Agrawal (6):
kvm: determine memory type from VMA
vfio/nvgpu: expose GPU device memory as BAR1
mm: handle poisoning of pfn without struct pages
mm: Add poison error check in fixup_user_fault() for mapped PFN
mm: Change ghes code to allow poison of non-struct PFN vfio/nvgpu: register device memory for poison handling

 MAINTAINERS                          |   6 +
 arch/arm64/include/asm/kvm_pgtable.h |   8 +-
 arch/arm64/include/asm/memory.h      |   6 +-
 arch/arm64/kvm/hyp/pgtable.c         |  16 +-
 arch/arm64/kvm/mmu.c                 |  27 +-
 drivers/acpi/apei/ghes.c             |  12 +-
 drivers/vfio/pci/Kconfig             |   2 +
 drivers/vfio/pci/Makefile            |   2 +
 drivers/vfio/pci/nvgpu/Kconfig       |  10 +
 drivers/vfio/pci/nvgpu/Makefile      |   3 +
 drivers/vfio/pci/nvgpu/main.c        | 359 +++++++++++++++++++++++++++
 include/linux/memory-failure.h       |  22 ++
 include/linux/mm.h                   |   1 +
 include/ras/ras_event.h              |   1 +
 mm/gup.c                             |   2 +-
 mm/memory-failure.c                  | 148 +++++++++--
 virt/kvm/kvm_main.c                  |   6 +
 17 files changed, 586 insertions(+), 45 deletions(-)
 create mode 100644 drivers/vfio/pci/nvgpu/Kconfig
 create mode 100644 drivers/vfio/pci/nvgpu/Makefile
 create mode 100644 drivers/vfio/pci/nvgpu/main.c
 create mode 100644 include/linux/memory-failure.h
Signature · 1 line
2.17.1
#1 lore raw

PATCH v3 1/6 kvm: determine memory type from VMA

From: Ankit Agrawal <[email protected]>

Each VM stores the requires pgprots for its mappings in the vma->pgprot. Based on this we can determine the desired MT_DEVICE_* for the VMA directly, and do not have to guess based on heuristics based on pfn_is_map_memory().

There are the following kinds of pgprot available to userspace and their corresponding type:
pgprot_noncached -> MT_DEVICE_nGnRnE
pgprot_writecombine -> MT_NORMAL_NC
pgprot_device -> MT_DEVICE_nGnRE
pgprot_tagged -> MT_NORMAL_TAGGED

Decode the relevant MT_* types in use and translate them into the corresponding KVM_PGTABLEPROT_*:

- MT_DEVICE_nGnRE -> KVM_PGTABLE_PROT_DEVICE_nGnRE (device)
- MT_DEVICE_nGnRnE -> KVM_PGTABLE_PROT_DEVICE_nGnRnE (noncached)
- MT_NORMAL/_TAGGED/_NC -> 0

The selection of 0 for the S2 KVM_PGTABLE_PROT_DEVICE_nGnRnE is based on [2].

Also worth noting is the result of the stage-1 and stage-2. Ref [3] If FWB not set, then the combination is the one that is more restrictive. The sequence from lowest restriction to the highest:
DEVICE_nGnRnE -> DEVICE_nGnRE -> NORMAL/_TAGGED/_NC
If FWB is set, then stage-2 mapping type overrides the stage-1 [1].

This solves a problem where KVM cannot preserve the MT_NORMAL memory type for non-struct page backed memory into the S2 mapping. Instead the VMA creator determines the MT type and the S2 will follow it.

[1] https://developer.arm.com/documentation/102376/0100/Combining-Stage-1-and-Stage-2-attributes
[2] ARMv8 reference manual: https://developer.arm.com/documentation/ddi0487/gb/ Section D5.5.3, Table D5-38
[3] ARMv8 reference manual: https://developer.arm.com/documentation/ddi0487/gb/ Table G5-20 on page G5-6330

---
 arch/arm64/include/asm/kvm_pgtable.h |  8 +++++---
 arch/arm64/include/asm/memory.h      |  6 ++++--
 arch/arm64/kvm/hyp/pgtable.c         | 16 +++++++++++-----
 arch/arm64/kvm/mmu.c                 | 27 ++++++++++++++++++++++-----
 4 files changed, 42 insertions(+), 15 deletions(-)
arch/arm64/include/asm/kvm_pgtable.h +5 -3
index 4cd6762bda80..d3166b6e6329 100644
@@ -150,7 +150,8 @@ enum kvm_pgtable_stage2_flags {
  * @KVM_PGTABLE_PROT_X:		Execute permission.
  * @KVM_PGTABLE_PROT_W:		Write permission.
  * @KVM_PGTABLE_PROT_R:		Read permission.
- * @KVM_PGTABLE_PROT_DEVICE:	Device attributes.
+ * @KVM_PGTABLE_PROT_DEVICE_nGnRE:	Device nGnRE attributes.
+ * @KVM_PGTABLE_PROT_DEVICE_nGnRnE:	Device nGnRnE attributes.
  * @KVM_PGTABLE_PROT_SW0:	Software bit 0.
  * @KVM_PGTABLE_PROT_SW1:	Software bit 1.
  * @KVM_PGTABLE_PROT_SW2:	Software bit 2.
@@ -161,7 +162,8 @@ enum kvm_pgtable_prot {
 	KVM_PGTABLE_PROT_W			= BIT(1),
 	KVM_PGTABLE_PROT_R			= BIT(2),
 
-	KVM_PGTABLE_PROT_DEVICE			= BIT(3),
+	KVM_PGTABLE_PROT_DEVICE_nGnRE		= BIT(3),
+	KVM_PGTABLE_PROT_DEVICE_nGnRnE		= BIT(4),
 
 	KVM_PGTABLE_PROT_SW0			= BIT(55),
 	KVM_PGTABLE_PROT_SW1			= BIT(56),
@@ -178,7 +180,7 @@ enum kvm_pgtable_prot {
 #define PAGE_HYP		KVM_PGTABLE_PROT_RW
 #define PAGE_HYP_EXEC		(KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_X)
 #define PAGE_HYP_RO		(KVM_PGTABLE_PROT_R)
-#define PAGE_HYP_DEVICE		(PAGE_HYP | KVM_PGTABLE_PROT_DEVICE)
+#define PAGE_HYP_DEVICE		(PAGE_HYP | KVM_PGTABLE_PROT_DEVICE_nGnRE)
 
 typedef bool (*kvm_pgtable_force_pte_cb_t)(u64 addr, u64 end,
 					   enum kvm_pgtable_prot prot);
arch/arm64/include/asm/memory.h +4 -2
index 78e5163836a0..4ebbc4b1ba4d 100644
@@ -147,14 +147,16 @@
  * Memory types for Stage-2 translation
  */
 #define MT_S2_NORMAL		0xf
+#define MT_S2_DEVICE_nGnRnE 0x0
 #define MT_S2_DEVICE_nGnRE	0x1
 
 /*
  * Memory types for Stage-2 translation when ID_AA64MMFR2_EL1.FWB is 0001
  * Stage-2 enforces Normal-WB and Device-nGnRE
  */
-#define MT_S2_FWB_NORMAL	6
-#define MT_S2_FWB_DEVICE_nGnRE	1
+#define MT_S2_FWB_NORMAL	0x6
+#define MT_S2_FWB_DEVICE_nGnRnE 0x0
+#define MT_S2_FWB_DEVICE_nGnRE	0x1
 
 #ifdef CONFIG_ARM64_4K_PAGES
 #define IOREMAP_MAX_ORDER	(PUD_SHIFT)
arch/arm64/kvm/hyp/pgtable.c +11 -5
index 3d61bd3e591d..7a8238b41590 100644
@@ -355,7 +355,7 @@ struct hyp_map_data {
 
 static int hyp_set_prot_attr(enum kvm_pgtable_prot prot, kvm_pte_t *ptep)
 {
-	bool device = prot & KVM_PGTABLE_PROT_DEVICE;
+	bool device = prot & KVM_PGTABLE_PROT_DEVICE_nGnRE;
 	u32 mtype = device ? MT_DEVICE_nGnRE : MT_NORMAL;
 	kvm_pte_t attr = FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S1_ATTRIDX, mtype);
 	u32 sh = KVM_PTE_LEAF_ATTR_LO_S1_SH_IS;
@@ -636,14 +636,20 @@ static bool stage2_has_fwb(struct kvm_pgtable *pgt)
 static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot prot,
 				kvm_pte_t *ptep)
 {
-	bool device = prot & KVM_PGTABLE_PROT_DEVICE;
-	kvm_pte_t attr = device ? KVM_S2_MEMATTR(pgt, DEVICE_nGnRE) :
-			    KVM_S2_MEMATTR(pgt, NORMAL);
 	u32 sh = KVM_PTE_LEAF_ATTR_LO_S2_SH_IS;
+	kvm_pte_t attr;
+
+	if (prot & KVM_PGTABLE_PROT_DEVICE_nGnRE)
+		attr = KVM_S2_MEMATTR(pgt, DEVICE_nGnRE);
+	else if (prot & KVM_PGTABLE_PROT_DEVICE_nGnRnE)
+		attr = KVM_S2_MEMATTR(pgt, DEVICE_nGnRnE);
+	else
+		attr = KVM_S2_MEMATTR(pgt, NORMAL);
 
 	if (!(prot & KVM_PGTABLE_PROT_X))
 		attr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
-	else if (device)
+	else if (prot & KVM_PGTABLE_PROT_DEVICE_nGnRE ||
+		 prot & KVM_PGTABLE_PROT_DEVICE_nGnRnE)
 		return -EINVAL;
 
 	if (prot & KVM_PGTABLE_PROT_R)
arch/arm64/kvm/mmu.c +22 -5
index 7113587222ff..8d63aa951c33 100644
@@ -897,7 +897,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
 	int ret = 0;
 	struct kvm_mmu_memory_cache cache = { .gfp_zero = __GFP_ZERO };
 	struct kvm_pgtable *pgt = kvm->arch.mmu.pgt;
-	enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_DEVICE |
+	enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_DEVICE_nGnRE |
 				     KVM_PGTABLE_PROT_R |
 				     (writable ? KVM_PGTABLE_PROT_W : 0);
 
@@ -1186,6 +1186,15 @@ static bool kvm_vma_mte_allowed(struct vm_area_struct *vma)
 	return vma->vm_flags & VM_MTE_ALLOWED;
 }
 
+/*
+ * Determine the memory region cacheability from VMA's pgprot. This
+ * is used to set the stage 2 PTEs.
+ */
+static unsigned long mapping_type(pgprot_t page_prot)
+{
+	return ((pgprot_val(page_prot) & PTE_ATTRINDX_MASK) >> 2);
+}
+
 static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 			  struct kvm_memory_slot *memslot, unsigned long hva,
 			  unsigned long fault_status)
@@ -1368,10 +1377,18 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	if (exec_fault)
 		prot |= KVM_PGTABLE_PROT_X;
 
-	if (device)
-		prot |= KVM_PGTABLE_PROT_DEVICE;
-	else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC))
-		prot |= KVM_PGTABLE_PROT_X;
+	switch (mapping_type(vma->vm_page_prot)) {
+	case MT_DEVICE_nGnRE:
+		prot |= KVM_PGTABLE_PROT_DEVICE_nGnRE;
+		break;
+	case MT_DEVICE_nGnRnE:
+		prot |= KVM_PGTABLE_PROT_DEVICE_nGnRnE;
+		break;
+		/* MT_NORMAL/_TAGGED/_NC */
+	default:
+		if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC))
+			prot |= KVM_PGTABLE_PROT_X;
+	}
 
 	/*
 	 * Under the premise of getting a FSC_PERM fault, we just need to relax
Signature · 1 line
2.17.1
maintainer
#2 lore raw

Re: [PATCH v3 1/6] kvm: determine memory type from VMA

Quoted 4 lines · On Wed, 05 Apr 2023 19:01:29 +0100, <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

Each VM stores the requires pgprots for its mappings in the

The VM stores *nothing*. Did you mean VMA? And even then, I can't parse this.

Quoted 18 lines

vma->pgprot. Based on this we can determine the desired MT_DEVICE_* for the VMA directly, and do not have to guess based on heuristics based on pfn_is_map_memory().

There are the following kinds of pgprot available to userspace and their corresponding type:
pgprot_noncached -> MT_DEVICE_nGnRnE
pgprot_writecombine -> MT_NORMAL_NC
pgprot_device -> MT_DEVICE_nGnRE
pgprot_tagged -> MT_NORMAL_TAGGED

Decode the relevant MT_* types in use and translate them into the corresponding KVM_PGTABLEPROT_*:

- MT_DEVICE_nGnRE -> KVM_PGTABLE_PROT_DEVICE_nGnRE (device)
- MT_DEVICE_nGnRnE -> KVM_PGTABLE_PROT_DEVICE_nGnRnE (noncached)
- MT_NORMAL/_TAGGED/_NC -> 0

Quoted 7 lines

The selection of 0 for the S2 KVM_PGTABLE_PROT_DEVICE_nGnRnE is based on [2].

Also worth noting is the result of the stage-1 and stage-2. Ref [3] If FWB not set, then the combination is the one that is more restrictive. The sequence from lowest restriction to the highest:
DEVICE_nGnRnE -> DEVICE_nGnRE -> NORMAL/_TAGGED/_NC

Sorry, but I can't parse this either. If you mean that the strongest memory type wins, then I agree. But what does it bring to the discussion?

Quoted 5 lines

If FWB is set, then stage-2 mapping type overrides the stage-1 [1].

This solves a problem where KVM cannot preserve the MT_NORMAL memory type for non-struct page backed memory into the S2 mapping. Instead the VMA creator determines the MT type and the S2 will follow it.

What makes it safe? How does VFIO ensures that the memory type used is correct in all circumstances? This has to hold for *ANY* device, not just your favourite toy of the day. Nothing in this patch is horribly wrong, but the above question must be answered before we can consider any of this.

Quoted 4 lines

[1] https://developer.arm.com/documentation/102376/0100/Combining-Stage-1-and-Stage-2-attributes
[2] ARMv8 reference manual: https://developer.arm.com/documentation/ddi0487/gb/ Section D5.5.3, Table D5-38
[3] ARMv8 reference manual: https://developer.arm.com/documentation/ddi0487/gb/ Table G5-20 on page G5-6330

Please quote references that are current at the time of posting. Revision DDI0487I.a is the most recent version, so use that.

Quoted 52 lines
---
 arch/arm64/include/asm/kvm_pgtable.h |  8 +++++---
 arch/arm64/include/asm/memory.h      |  6 ++++--
 arch/arm64/kvm/hyp/pgtable.c         | 16 +++++++++++-----
 arch/arm64/kvm/mmu.c                 | 27 ++++++++++++++++++++++-----
 4 files changed, 42 insertions(+), 15 deletions(-)
arch/arm64/include/asm/kvm_pgtable.h +5 -3
index 4cd6762bda80..d3166b6e6329 100644
@@ -150,7 +150,8 @@ enum kvm_pgtable_stage2_flags {
  * @KVM_PGTABLE_PROT_X:		Execute permission.
  * @KVM_PGTABLE_PROT_W:		Write permission.
  * @KVM_PGTABLE_PROT_R:		Read permission.
- * @KVM_PGTABLE_PROT_DEVICE:	Device attributes.
+ * @KVM_PGTABLE_PROT_DEVICE_nGnRE:	Device nGnRE attributes.
+ * @KVM_PGTABLE_PROT_DEVICE_nGnRnE:	Device nGnRnE attributes.
  * @KVM_PGTABLE_PROT_SW0:	Software bit 0.
  * @KVM_PGTABLE_PROT_SW1:	Software bit 1.
  * @KVM_PGTABLE_PROT_SW2:	Software bit 2.
@@ -161,7 +162,8 @@ enum kvm_pgtable_prot {
 	KVM_PGTABLE_PROT_W			= BIT(1),
 	KVM_PGTABLE_PROT_R			= BIT(2),
 
-	KVM_PGTABLE_PROT_DEVICE			= BIT(3),
+	KVM_PGTABLE_PROT_DEVICE_nGnRE		= BIT(3),
+	KVM_PGTABLE_PROT_DEVICE_nGnRnE		= BIT(4),
 
 	KVM_PGTABLE_PROT_SW0			= BIT(55),
 	KVM_PGTABLE_PROT_SW1			= BIT(56),
@@ -178,7 +180,7 @@ enum kvm_pgtable_prot {
 #define PAGE_HYP		KVM_PGTABLE_PROT_RW
 #define PAGE_HYP_EXEC		(KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_X)
 #define PAGE_HYP_RO		(KVM_PGTABLE_PROT_R)
-#define PAGE_HYP_DEVICE		(PAGE_HYP | KVM_PGTABLE_PROT_DEVICE)
+#define PAGE_HYP_DEVICE		(PAGE_HYP | KVM_PGTABLE_PROT_DEVICE_nGnRE)
 
 typedef bool (*kvm_pgtable_force_pte_cb_t)(u64 addr, u64 end,
 					   enum kvm_pgtable_prot prot);
arch/arm64/include/asm/memory.h +1 -0
index 78e5163836a0..4ebbc4b1ba4d 100644
@@ -147,14 +147,16 @@
  * Memory types for Stage-2 translation
  */
 #define MT_S2_NORMAL		0xf
+#define MT_S2_DEVICE_nGnRnE 0x0
 #define MT_S2_DEVICE_nGnRE	0x1

arch/arm64/include/asm/memory.h line 159

Quoted 10 lines
arch/arm64/include/asm/memory.h +3 -2
 
 /*
  * Memory types for Stage-2 translation when ID_AA64MMFR2_EL1.FWB is 0001
  * Stage-2 enforces Normal-WB and Device-nGnRE
  */
-#define MT_S2_FWB_NORMAL	6
-#define MT_S2_FWB_DEVICE_nGnRE	1
+#define MT_S2_FWB_NORMAL	0x6
+#define MT_S2_FWB_DEVICE_nGnRnE 0x0
+#define MT_S2_FWB_DEVICE_nGnRE	0x1

Why the repainting of perfectly valid constants? What does the 0x prefix bring? Does the comment above need updating?

Quoted 28 lines
arch/arm64/include/asm/memory.h +0 -0
 #ifdef CONFIG_ARM64_4K_PAGES
 #define IOREMAP_MAX_ORDER	(PUD_SHIFT)
arch/arm64/kvm/hyp/pgtable.c +5 -4
index 3d61bd3e591d..7a8238b41590 100644
@@ -355,7 +355,7 @@ struct hyp_map_data {
 
 static int hyp_set_prot_attr(enum kvm_pgtable_prot prot, kvm_pte_t *ptep)
 {
-	bool device = prot & KVM_PGTABLE_PROT_DEVICE;
+	bool device = prot & KVM_PGTABLE_PROT_DEVICE_nGnRE;
 	u32 mtype = device ? MT_DEVICE_nGnRE : MT_NORMAL;
 	kvm_pte_t attr = FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S1_ATTRIDX, mtype);
 	u32 sh = KVM_PTE_LEAF_ATTR_LO_S1_SH_IS;
@@ -636,14 +636,20 @@ static bool stage2_has_fwb(struct kvm_pgtable *pgt)
 static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot prot,
 				kvm_pte_t *ptep)
 {
-	bool device = prot & KVM_PGTABLE_PROT_DEVICE;
-	kvm_pte_t attr = device ? KVM_S2_MEMATTR(pgt, DEVICE_nGnRE) :
-			    KVM_S2_MEMATTR(pgt, NORMAL);
 	u32 sh = KVM_PTE_LEAF_ATTR_LO_S2_SH_IS;
+	kvm_pte_t attr;
+
+	if (prot & KVM_PGTABLE_PROT_DEVICE_nGnRE)
+		attr = KVM_S2_MEMATTR(pgt, DEVICE_nGnRE);

arch/arm64/kvm/hyp/pgtable.c static bool stage2_has_fwb(struct kvm_pgtable *pgt) · line 652

Quoted 10 lines
arch/arm64/kvm/hyp/pgtable.c +6 -1
+	else if (prot & KVM_PGTABLE_PROT_DEVICE_nGnRnE)
+		attr = KVM_S2_MEMATTR(pgt, DEVICE_nGnRnE);
+	else
+		attr = KVM_S2_MEMATTR(pgt, NORMAL);
 
 	if (!(prot & KVM_PGTABLE_PROT_X))
 		attr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
-	else if (device)
+	else if (prot & KVM_PGTABLE_PROT_DEVICE_nGnRE ||
+		 prot & KVM_PGTABLE_PROT_DEVICE_nGnRnE)

Why don't you keep 'device', which is concise and readable, and simply change the way it is assigned?

Quoted 17 lines
arch/arm64/kvm/hyp/pgtable.c +0 -0
 		return -EINVAL;
 
 	if (prot & KVM_PGTABLE_PROT_R)
arch/arm64/kvm/mmu.c +1 -1
index 7113587222ff..8d63aa951c33 100644
@@ -897,7 +897,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
 	int ret = 0;
 	struct kvm_mmu_memory_cache cache = { .gfp_zero = __GFP_ZERO };
 	struct kvm_pgtable *pgt = kvm->arch.mmu.pgt;
-	enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_DEVICE |
+	enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_DEVICE_nGnRE |
 				     KVM_PGTABLE_PROT_R |
 				     (writable ? KVM_PGTABLE_PROT_W : 0);
 

arch/arm64/kvm/mmu.c static bool kvm_vma_mte_allowed(struct vm_area_struct *vma) · line 1195

Quoted 10 lines
arch/arm64/kvm/mmu.c +7 -0
@@ -1186,6 +1186,15 @@ static bool kvm_vma_mte_allowed(struct vm_area_struct *vma)
 	return vma->vm_flags & VM_MTE_ALLOWED;
 }
 
+/*
+ * Determine the memory region cacheability from VMA's pgprot. This
+ * is used to set the stage 2 PTEs.
+ */
+static unsigned long mapping_type(pgprot_t page_prot)
+{
+	return ((pgprot_val(page_prot) & PTE_ATTRINDX_MASK) >> 2);

Please use FIELD_GET() for field extraction.

Quoted 10 lines
arch/arm64/kvm/mmu.c +2 -1
+}
+
 static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 			  struct kvm_memory_slot *memslot, unsigned long hva,
 			  unsigned long fault_status)
@@ -1368,10 +1377,18 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	if (exec_fault)
 		prot |= KVM_PGTABLE_PROT_X;
 
-	if (device)

arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1386

Quoted 10 lines
arch/arm64/kvm/mmu.c +7 -3
-		prot |= KVM_PGTABLE_PROT_DEVICE;
-	else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC))
-		prot |= KVM_PGTABLE_PROT_X;
+	switch (mapping_type(vma->vm_page_prot)) {
+	case MT_DEVICE_nGnRE:
+		prot |= KVM_PGTABLE_PROT_DEVICE_nGnRE;
+		break;
+	case MT_DEVICE_nGnRnE:
+		prot |= KVM_PGTABLE_PROT_DEVICE_nGnRnE;
+		break;

Please keep the 'device' guard. It makes it easy to find the code paths that are relevant to this case.

arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1394

Quoted 8 lines
arch/arm64/kvm/mmu.c +5 -0
+		/* MT_NORMAL/_TAGGED/_NC */
+	default:
+		if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC))
+			prot |= KVM_PGTABLE_PROT_X;
+	}
 
 	/*
 	 * Under the premise of getting a FSC_PERM fault, we just need to relax

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#3 lore raw

Re: [PATCH v3 1/6] kvm: determine memory type from VMA

Quoted 5 lines · Marc Zyngier

What makes it safe? How does VFIO ensures that the memory type used is correct in all circumstances? This has to hold for *ANY* device, not just your favourite toy of the day. Nothing in this patch is horribly wrong, but the above question must be answered before we can consider any of this.

In VFIO we now have the concept of "variant drivers" which work with specific PCI IDs. The variant drivers can inject device specific knowledge into VFIO. In this series the driver injects the cachable pgprot when it creates some of the VMAs because it knows the PCI IDs it supports, parses the ACPI description, and knows for sure that the memory it puts in the cachable VMA is linked with a cache coherent interconnect.

The generic vfio-pci path is not changed, so 'any device' is not relevant here.

Jason

maintainer
#4 lore raw

Re: [PATCH v3 1/6] kvm: determine memory type from VMA

Quoted 15 lines · Jason Gunthorpe
Quoted 5 lines · Marc Zyngier

What makes it safe? How does VFIO ensures that the memory type used is correct in all circumstances? This has to hold for *ANY* device, not just your favourite toy of the day. Nothing in this patch is horribly wrong, but the above question must be answered before we can consider any of this.

In VFIO we now have the concept of "variant drivers" which work with specific PCI IDs. The variant drivers can inject device specific knowledge into VFIO. In this series the driver injects the cachable pgprot when it creates some of the VMAs because it knows the PCI IDs it supports, parses the ACPI description, and knows for sure that the memory it puts in the cachable VMA is linked with a cache coherent interconnect.

Quoted 2 lines

The generic vfio-pci path is not changed, so 'any device' is not relevant here.

There were several off-list discussions, I'm trying to summarise my understanding here. This series aims to relax the VFIO mapping to cacheable and have KVM map it into the guest with the same attributes. Somewhat related past threads also tried to relax the KVM device pass-through mapping from Device_nGnRnE (pgprot_noncached) to Normal_NC (pgprot_writecombine). Those were initially using the PCIe prefetchable BAR attribute but that's not a reliable means to infer whether Normal vs Device is safe. Anyway, I think we'd need to unify these threads and come up with some common handling that can cater for various attributes required by devices/drivers. Therefore replying in this thread.

Current status on arm64: vfio-pci user mapping (Qemu) of PCIe BARs is Device_nGnRnE. KVM also maps it at Stage 2 with the same attributes. The user mapping is fine since the VMM may not have detailed knowledge about how to safely map the BAR. KVM doesn't have such detailed knowledge either in the device pass-through case but for safety reasons it follows the same attributes as the user mapping.

From a safety perspective, relaxing the KVM stage 2 mapping has some implications:

1. It creates multiple memory aliases with different attributes.

2. Speculative loads, unaligned accesses.

3. Potentially new uncontained errors introduced by Normal NC vs Device mappings.

From the private discussions I had regarding point (3), Device doesn't really make any difference and it can be even worse. For example, Normal mappings would allow a contained error while Device would trigger an uncontained SError. So the only safe bet here is for the platform, root complex to behave properly when device pass-through is involved (e.g. PCIe to ignore writes, return 0xff on reads if there are errors). That's something Arm is working on to require in the BSA specs (base system architecture). Presumably cloud vendors allowing device pass-through already know their system well enough, no need for further policing in KVM (which it can't do properly anyway).

As long as the errors are contained, point (2) becomes the responsibility of the guest, given its detailed knowledge of the device, using the appropriate attributes (whether x86 WC maps exactly onto arm64 Normal NC is the subject of a separate discussion; so far that's the assumption we took in the arm64 kernel). Regarding vfio-pci, the user mapping must remain Device_nGnRnE on arm64 to avoid unexpected speculative loads.

This leaves us with (1) and since the vfio-pci user mapping must remain Device, there's a potential mismatched alias if the guest maps the corresponding BAR as Normal NC. Luckily the Arm ARM constrains the hardware behaviour here to basically allowing the Device mapping in the VMM to behave somewhat the Normal NC mapping in the guest. IOW, the VMM loses the Device guarantees (non-speculation, ordering). That's not a problem for the device since the guest already deemed such mapping to be safe.

While I think it's less likely for the VMM to access the same BAR that was mapped into the guest, if we want to be on the safe side from an ABI perspective (the returned mmap() now has different memory guarantees), we could make this an opt-in. That's pretty much the VMM stating that it is ok with losing some of the Device guarantees for the vfio-pci mapping. A question here is whether we want to this to be done at the vfio-pci level, the KVM memory slot or a big knob per VMM. Or whether we need to bother with this at all (if it's just theoretical, maybe we can have a global opt-out).

Going back to this series, allowing Cacheable mapping in KVM has similar implications as above. (2) and (3) would be assumed safe by the platform vendors. Regarding (1), to avoid confusion, I would only allow it if FWB (force write-back) is present so that KVM can enforce a cacheable mapping at Stage 2 if the vfio variant driver also maps it as cacheable in user space.

There were some other thoughts on only allowing different attributes in KVM if the user counterpart does not have any mapping (e.g. fd-based KVM memory slots). However, this won't work well with CXL-attached memory for example where the VMM may want access to it (e.g. virtio). So I wouldn't spend more thoughts on this.

The TL;DR summary of what I think we should do:

a) Relax the KVM Stage 2 mapping for vfio-pci to Normal NC (pgprot_writecombine). TBD whether we need a VMM opt-in is at the vfio-pci level, KVM slot or per-VMM level. Another option is to do this by default and have a global opt-out (cmdline). I don't think the latter is such a bad idea since I find it unlikely for the VMM to also touch the same PCIe BAR _and_ expect different memory ordering guarantees than the guest device driver.

b) Map the KVM stage 2 mapping as cacheable+FWB iff the VMM has a corresponding cacheable mapping.

Thoughts?

Signature · 1 line
Catalin
#5 lore raw

Re: [PATCH v3 1/6] kvm: determine memory type from VMA

Quoted 6 lines · Catalin Marinas

Current status on arm64: vfio-pci user mapping (Qemu) of PCIe BARs is Device_nGnRnE. KVM also maps it at Stage 2 with the same attributes. The user mapping is fine since the VMM may not have detailed knowledge about how to safely map the BAR. KVM doesn't have such detailed knowledge either in the device pass-through case but for safety reasons it follows the same attributes as the user mapping.

To put a fine point on this - it means the KVM VM does not emulate the same behaviors as a bare metal ARM system and this results in significant performance degradation (loss of WC) or malfunction (loss of cachable) when working with some VFIO device scenarios.

Quoted 3 lines

architecture). Presumably cloud vendors allowing device pass-through already know their system well enough, no need for further policing in KVM (which it can't do properly anyway).

This also matches x86 pretty well. There is alot of platform and BIOS magic at work to make sure that VFIO is safe in general. As this is not discoverable by the OS we have no choice in the kernel but to assume the operator knows what they are doing.

Quoted 6 lines

While I think it's less likely for the VMM to access the same BAR that was mapped into the guest, if we want to be on the safe side from an ABI perspective (the returned mmap() now has different memory guarantees), we could make this an opt-in. That's pretty much the VMM stating that it is ok with losing some of the Device guarantees for the vfio-pci mapping.

I don't know of any use case where this matters. The VMM is already basically not allowed to touch the MMIO spaces for security reasons. The only reason it is mmap'd into the VMM at all is because this is the only way to get it into KVM.

So an opt-in seems like overkill to me.

Quoted 6 lines

Going back to this series, allowing Cacheable mapping in KVM has similar implications as above. (2) and (3) would be assumed safe by the platform vendors. Regarding (1), to avoid confusion, I would only allow it if FWB (force write-back) is present so that KVM can enforce a cacheable mapping at Stage 2 if the vfio variant driver also maps it as cacheable in user space.

Yes, currently if FWB is not available this patch makes KVM crash :\
FWB needs to be enforced as well in this patch.

Quoted 5 lines

There were some other thoughts on only allowing different attributes in KVM if the user counterpart does not have any mapping (e.g. fd-based KVM memory slots). However, this won't work well with CXL-attached memory for example where the VMM may want access to it (e.g. virtio). So I wouldn't spend more thoughts on this.

CXL is a different beast - struct page backed CXL memory should be force-cachable everywhere just like normal system memory (it is normal system memory). This is the kind of memory that might be mixed with virtio.

However, some future CXL VFIO device memory should ideally allow the VM full flexibility to use cachable or not to properly emulate bare-metal. This approach where we pre-select some of the CXL memory as cachable is kind of a half job..

I think providing a way for KVM to take in a FD instead of a VMA for mapping memory could be interesting in general. We waste a fair amount of PTEs creating a 40GB linear map in the CPU just for KVM. Avoiding the alias mapping is a nice side effect.

We'd have to do something else for the SIGBUS stuff though..

Quoted 2 lines

a) Relax the KVM Stage 2 mapping for vfio-pci to Normal NC (pgprot_writecombine). TBD whether we need a VMM opt-in is at the

We are working on a tested patch for this, it isn't working yet for some reason, still debugging.

Quoted 2 lines

b) Map the KVM stage 2 mapping as cacheable+FWB iff the VMM has a corresponding cacheable mapping.

That is this patch, with some improvements to check FWB/etc.

Thanks,
Jason

#6 lore raw

Re: [PATCH v3 1/6] kvm: determine memory type from VMA

In reply to Catalin Marinas
Quoted 10 lines · Catalin Marinas

There were several off-list discussions, I'm trying to summarise my understanding here. This series aims to relax the VFIO mapping to cacheable and have KVM map it into the guest with the same attributes. Somewhat related past threads also tried to relax the KVM device pass-through mapping from Device_nGnRnE (pgprot_noncached) to Normal_NC (pgprot_writecombine). Those were initially using the PCIe prefetchable BAR attribute but that's not a reliable means to infer whether Normal vs Device is safe. Anyway, I think we'd need to unify these threads and come up with some common handling that can cater for various attributes required by devices/drivers. Therefore replying in this thread.

So picking up on this as I was just trying to start a separate discussion
on the subject for write combine :-)

In this case, not so much for KVM as much as for VFIO to userspace though.

The rough idea is that the "userspace driver" (ie DPDK or equivalent) for the device is the one to "know" wether a BAR or portion of a BAR can/should be mapped write-combine, and is expected to also "know" what to do to enforce ordering when necessary.

So the userspace component needs to be responsible for selecting the mapping, the same way using the PCI sysfs resource files today allows to do that by selecting the _wc variant.

I posted a separate message that Lorenzo CCed back to some of you, but let's recap here to keep the discussion localized.

I don't know how much of this makes sense for KVM, but I think what we really want is for userspace to be able to specify some "attributes" (which we can initially limit to writecombine, full cachability probably requires a device specific kernel driver providing adequate authority, separate discussion in any case), for all or a portion of a BAR mapping.

The easy way is an ioctl to affect the attributes of the next mmap but it's a rather gross interface.

A better approach (still requires some coordination but not nearly as bad) would be to have an ioctl to create "subregions", ie, dynamically add new "struct vfio_pci_region" (using the existing dynamic index API), which are children of existing regions (including real BARs) and provide different attributes, which mmap can then honor.

This is particularly suited for the case (which used to exist, I don't know if it still does) where the buffer that wants write combining reside in the same BAR as registers that otherwise don't.

A simpler compromise if that latter case is deemed irrelevant would be an ioctl to selectively set a region index (including BARs) to be WC prior to mmap.

I don't know if that fits in the ideas you have for KVM, I think it could by having the userspace component require mappings using a "special" attribute which we could define as being the most relaxed allowed to pass to a VM, which can then be architecture defined. The guest can then enforce specifics. Does this make sense ?

Cheers
Ben.

maintainer
#7 lore raw

Re: [PATCH v3 1/6] kvm: determine memory type from VMA

Hi Ben,

Quoted 12 lines · Benjamin Herrenschmidt
Quoted 10 lines · Catalin Marinas

There were several off-list discussions, I'm trying to summarise my understanding here. This series aims to relax the VFIO mapping to cacheable and have KVM map it into the guest with the same attributes. Somewhat related past threads also tried to relax the KVM device pass-through mapping from Device_nGnRnE (pgprot_noncached) to Normal_NC (pgprot_writecombine). Those were initially using the PCIe prefetchable BAR attribute but that's not a reliable means to infer whether Normal vs Device is safe. Anyway, I think we'd need to unify these threads and come up with some common handling that can cater for various attributes required by devices/drivers. Therefore replying in this thread.

Quoted 2 lines

So picking up on this as I was just trying to start a separate discussion on the subject for write combine :-)

Basically this thread started as a fix/improvement for KVM by mimicking the VFIO user mapping attributes at the guest but the conclusion we came to is that the VFIO PCIe driver cannot reliably tell when WC is possible.

Quoted 7 lines

In this case, not so much for KVM as much as for VFIO to userspace though.

The rough idea is that the "userspace driver" (ie DPDK or equivalent) for the device is the one to "know" wether a BAR or portion of a BAR can/should be mapped write-combine, and is expected to also "know" what to do to enforce ordering when necessary.

I agree in principle. On the KVM side we concluded that it's the guest driver that knows the attributes, so the hypervisor should not restrict them. In the DPDK case, it would be the user driver that knows the device it is mapping and the required attributes.

In terms of security for arm64 at least, Device vs Normal NC (or nc vs wc in Linux terminology) doesn't make much difference with the former occasionally being worse. The kernel would probably trust the DPDK code if it allows direct device access.

Quoted 3 lines

So the userspace component needs to be responsible for selecting the mapping, the same way using the PCI sysfs resource files today allows to do that by selecting the _wc variant.

I guess the sysfs interface is just trying to work around the VFIO limitations.

Quoted 6 lines

I don't know how much of this makes sense for KVM, but I think what we really want is for userspace to be able to specify some "attributes" (which we can initially limit to writecombine, full cachability probably requires a device specific kernel driver providing adequate authority, separate discussion in any case), for all or a portion of a BAR mapping.

For KVM, at least the WC case, user-space doesn't need to be involved as it normally should not access the same BAR concurrently with the guest. But at some point, for CXL-attached memory for example, it may need to be able to map it as cacheable so that it has the same attributes as the guest.

Quoted 9 lines

The easy way is an ioctl to affect the attributes of the next mmap but it's a rather gross interface.

A better approach (still requires some coordination but not nearly as bad) would be to have an ioctl to create "subregions", ie, dynamically add new "struct vfio_pci_region" (using the existing dynamic index API), which are children of existing regions (including real BARs) and provide different attributes, which mmap can then honor.

Quoted 3 lines

This is particularly suited for the case (which used to exist, I don't know if it still does) where the buffer that wants write combining reside in the same BAR as registers that otherwise don't.

IIUC that's still the case for some devices (I think Jason mentioned some Mellanox cards).

Quoted 9 lines

A simpler compromise if that latter case is deemed irrelevant would be an ioctl to selectively set a region index (including BARs) to be WC prior to mmap.

I don't know if that fits in the ideas you have for KVM, I think it could by having the userspace component require mappings using a "special" attribute which we could define as being the most relaxed allowed to pass to a VM, which can then be architecture defined. The guest can then enforce specifics. Does this make sense ?

I think this interface would help KVM when we'll need a cacheable mapping. For WC, we are ok without any VFIO changes.

Signature · 1 line
Catalin
#8 lore raw

Re: [PATCH v3 1/6] kvm: determine memory type from VMA

Quoted 4 lines · Catalin Marinas

In terms of security for arm64 at least, Device vs Normal NC (or nc vs wc in Linux terminology) doesn't make much difference with the former occasionally being worse. The kernel would probably trust the DPDK code if it allows direct device access.

RDMA and DRM already allow device drivers to map WC to userspace on demand, we expect the platform to support this.

Quoted 6 lines
Quoted 3 lines

So the userspace component needs to be responsible for selecting the mapping, the same way using the PCI sysfs resource files today allows to do that by selecting the _wc variant.

I guess the sysfs interface is just trying to work around the VFIO limitations.

I think just nobody has ever asked for VFIO WC support. The main non-VM user is DPDK and none of the NIC drivers have wanted this (DPDK applications areis more of throughput than latency focused typically)

Quoted 6 lines
Quoted 3 lines

This is particularly suited for the case (which used to exist, I don't know if it still does) where the buffer that wants write combining reside in the same BAR as registers that otherwise don't.

IIUC that's still the case for some devices (I think Jason mentioned some Mellanox cards).

Right, VFIO will have to allow it page-by-page

Quoted 2 lines

I think this interface would help KVM when we'll need a cacheable mapping. For WC, we are ok without any VFIO changes.

Yes, it may be interesting to map cachable CXL memory as NORMAL_NC into userspace for similar reasons.

Jason

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#9 lore raw

Re: [PATCH v3 1/6] kvm: determine memory type from VMA

Quoted 14 lines · On Sun, 16 Jul 2023 19:30:23 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 4 lines · Catalin Marinas

In terms of security for arm64 at least, Device vs Normal NC (or nc vs wc in Linux terminology) doesn't make much difference with the former occasionally being worse. The kernel would probably trust the DPDK code if it allows direct device access.

RDMA and DRM already allow device drivers to map WC to userspace on demand, we expect the platform to support this.

Quoted 4 lines
Quoted 3 lines

So the userspace component needs to be responsible for selecting the mapping, the same way using the PCI sysfs resource files today allows to do that by selecting the _wc variant.

Quoted 6 lines
Quoted 2 lines

I guess the sysfs interface is just trying to work around the VFIO limitations.

I think just nobody has ever asked for VFIO WC support. The main non-VM user is DPDK and none of the NIC drivers have wanted this (DPDK applications areis more of throughput than latency focused typically)

Yes, QEMU can't know whether the device or driver want a WC BAR mapping, so we've left it for KVM manipulation relative to VM use cases. Nobody has followed through with a complete proposal to enable it otherwise for direct userspace driver access, but I don't think there's opposition to providing such a thing. Thanks,

Alex

Quoted 17 lines
Quoted 6 lines
Quoted 3 lines

This is particularly suited for the case (which used to exist, I don't know if it still does) where the buffer that wants write combining reside in the same BAR as registers that otherwise don't.

IIUC that's still the case for some devices (I think Jason mentioned some Mellanox cards).

Right, VFIO will have to allow it page-by-page

Quoted 2 lines

I think this interface would help KVM when we'll need a cacheable mapping. For WC, we are ok without any VFIO changes.

Yes, it may be interesting to map cachable CXL memory as NORMAL_NC into userspace for similar reasons.

Jason

#10 lore raw

Re: [PATCH v3 1/6] kvm: determine memory type from VMA

Quoted 24 lines · Alex Williamson
Quoted 14 lines · On Sun, 16 Jul 2023 19:30:23 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 4 lines · Catalin Marinas

In terms of security for arm64 at least, Device vs Normal NC (or nc vs wc in Linux terminology) doesn't make much difference with the former occasionally being worse. The kernel would probably trust the DPDK code if it allows direct device access.

RDMA and DRM already allow device drivers to map WC to userspace on demand, we expect the platform to support this.

Quoted 4 lines
Quoted 3 lines

So the userspace component needs to be responsible for selecting the mapping, the same way using the PCI sysfs resource files today allows to do that by selecting the _wc variant.

Quoted 6 lines
Quoted 2 lines

I guess the sysfs interface is just trying to work around the VFIO limitations.

I think just nobody has ever asked for VFIO WC support. The main non-VM user is DPDK and none of the NIC drivers have wanted this (DPDK applications areis more of throughput than latency focused typically)

Quoted 5 lines

Yes, QEMU can't know whether the device or driver want a WC BAR mapping, so we've left it for KVM manipulation relative to VM use cases.  Nobody has followed through with a complete proposal to enable it otherwise for direct userspace driver access, but I don't think there's opposition to providing such a thing.  Thanks,

Ok, this is really backburner work for me but I'll try to cook up a POC patch in the near (hopefully) future along the lines of the subregions I proposed and we can discuss from there.

Cheers,
Ben.

#11 lore raw

PATCH v3 2/6 vfio/nvgpu: expose GPU device memory as BAR1

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

The NVIDIA Grace Hopper superchip does not model the coherent GPU memory aperture as a PCI config space BAR.

Introduce an in-tree VFIO PCI variant module (nvgpu-vfio-pci) to expose the GPU memory as BAR1 to the userspace. The GPU memory size and physical address are obtained from ACPI using device_property_read_u64() and exported to userspace as the VFIO_REGION. QEMU will naturally generate a PCI device in the VM where the cachable aperture is reported in BAR1.

QEMU can fetch the region information and perform mapping on it. The subsequent mmap call is handled by mmap() function pointer for the nvgpu-vfio-pci module and mapping to the GPU memory is established using the remap_pfn_range() API.

---
 MAINTAINERS                     |   6 +
 drivers/vfio/pci/Kconfig        |   2 +
 drivers/vfio/pci/Makefile       |   2 +
 drivers/vfio/pci/nvgpu/Kconfig  |  10 ++
 drivers/vfio/pci/nvgpu/Makefile |   3 +
 drivers/vfio/pci/nvgpu/main.c   | 255 ++++++++++++++++++++++++++++++++
 6 files changed, 278 insertions(+)
 create mode 100644 drivers/vfio/pci/nvgpu/Kconfig
 create mode 100644 drivers/vfio/pci/nvgpu/Makefile
 create mode 100644 drivers/vfio/pci/nvgpu/main.c
MAINTAINERS +6 -0
index 1dc8bd26b6cf..6b48756c30d3 100644
@@ -21954,6 +21954,12 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/vfio/pci/mlx5/
 
+VFIO NVIDIA PCI DRIVER
+M:	Ankit Agrawal <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	drivers/vfio/pci/nvgpu/
+
 VGA_SWITCHEROO
 R:	Lukas Wunner <[email protected]>
 S:	Maintained
drivers/vfio/pci/Kconfig +2 -0
index f9d0c908e738..ade18b0ffb7b 100644
@@ -59,4 +59,6 @@ source "drivers/vfio/pci/mlx5/Kconfig"
 
 source "drivers/vfio/pci/hisilicon/Kconfig"
 
+source "drivers/vfio/pci/nvgpu/Kconfig"
+
 endif
drivers/vfio/pci/Makefile +2 -0
index 24c524224da5..0c93d452d0da 100644
@@ -11,3 +11,5 @@ obj-$(CONFIG_VFIO_PCI) += vfio-pci.o
 obj-$(CONFIG_MLX5_VFIO_PCI)           += mlx5/
 
 obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/
+
+obj-$(CONFIG_NVGPU_VFIO_PCI) += nvgpu/
drivers/vfio/pci/nvgpu/Kconfig new file +10 -0
new file mode 100644
index 000000000000..066f764f7c5f
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config NVGPU_VFIO_PCI
+	tristate "VFIO support for the GPU in the NVIDIA Grace Hopper Superchip"
+	depends on ARM64 || (COMPILE_TEST && 64BIT)
+	select VFIO_PCI_CORE
+	help
+	  VFIO support for the GPU in the NVIDIA Grace Hopper Superchip is
+	  required to assign the GPU device to a VM using KVM/qemu/etc.
+
+	  If you don't know what to do here, say N.
drivers/vfio/pci/nvgpu/Makefile new file +3 -0
new file mode 100644
index 000000000000..00fd3a078218
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_NVGPU_VFIO_PCI) += nvgpu-vfio-pci.o
+nvgpu-vfio-pci-y := main.o
drivers/vfio/pci/nvgpu/main.c new file +255 -0

255 lines not on this page · raw message

Signature · 1 line
2.17.1
#12 lore raw

Re: [PATCH v3 2/6] vfio/nvgpu: expose GPU device memory as BAR1

kernel test robot noticed the following build warnings:

Hi,

kernel test robot noticed the following build warnings:

[auto build test WARNING on awilliam-vfio/for-linus]
[also build test WARNING on kvmarm/next akpm-mm/mm-everything linus/master v6.3-rc5 next-20230405] [cannot apply to awilliam-vfio/next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/ankita-nvidia-com/kvm-determine-memory-type-from-VMA/20230406-020404
base:   https://github.com/awilliam/linux-vfio.git for-linus
patch link:    https://lore.kernel.org/r/20230405180134.16932-3-ankita%40nvidia.com
patch subject: [PATCH v3 2/6] vfio/nvgpu: expose GPU device memory as BAR1
config: sparc-allyesconfig (https://download.01.org/0day-ci/archive/20230406/[email protected]/config)
compiler: sparc64-linux-gcc (GCC) 12.1.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/09ea30fcd2fb02d13a38cab4bf3d903f902408f4
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review ankita-nvidia-com/kvm-determine-memory-type-from-VMA/20230406-020404
git checkout 09ea30fcd2fb02d13a38cab4bf3d903f902408f4
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=sparc olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=sparc SHELL=/bin/bash drivers/vfio/pci/nvgpu/

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> drivers/vfio/pci/nvgpu/main.c:57:5: warning: no previous prototype for 'nvgpu_vfio_pci_mmap' [-Wmissing-prototypes]
      57 | int nvgpu_vfio_pci_mmap(struct vfio_device *core_vdev,
         |     ^~~~~~~~~~~~~~~~~~~
>> drivers/vfio/pci/nvgpu/main.c:100:6: warning: no previous prototype for 'nvgpu_vfio_pci_ioctl' [-Wmissing-prototypes]

100 | long nvgpu_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
| ^~~~~~~~~~~~~~~~~~~~

vim +/nvgpu_vfio_pci_mmap +57 drivers/vfio/pci/nvgpu/main.c

    56	
  > 57	int nvgpu_vfio_pci_mmap(struct vfio_device *core_vdev,
    58				struct vm_area_struct *vma)
    59	{
    60		struct nvgpu_vfio_pci_core_device *nvdev = container_of(
    61			core_vdev, struct nvgpu_vfio_pci_core_device, core_device.vdev);
    62	
    63		unsigned long start_pfn;
    64		unsigned int index;
    65		u64 req_len, pgoff;
    66		int ret = 0;
    67	
    68		index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
    69		if (index != nvdev->mem_prop.bar1_start_offset)
    70			return vfio_pci_core_mmap(core_vdev, vma);
    71	
    72		/*
    73		 * Request to mmap the BAR1. Map to the CPU accessible memory on the
    74		 * GPU using the memory information gathered from the system ACPI
    75		 * tables.
    76		 */
    77		start_pfn = nvdev->mem_prop.hpa >> PAGE_SHIFT;
    78		req_len = vma->vm_end - vma->vm_start;
    79		pgoff = vma->vm_pgoff &
    80			((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
    81		if (pgoff >= (nvdev->mem_prop.mem_length >> PAGE_SHIFT))
    82			return -EINVAL;
    83	
    84		/*
    85		 * Perform a PFN map to the memory. The device BAR1 is backed by the
    86		 * GPU memory now. Check that the mapping does not overflow out of
    87		 * the GPU memory size.
    88		 */
    89		ret = remap_pfn_range(vma, vma->vm_start, start_pfn + pgoff,
    90				      min(req_len, nvdev->mem_prop.mem_length - pgoff),
    91				      vma->vm_page_prot);
    92		if (ret)
    93			return ret;
    94	
    95		vma->vm_pgoff = start_pfn + pgoff;
    96	
    97		return 0;
    98	}
    99	
 > 100	long nvgpu_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int cmd,

101 unsigned long arg)
102 {
103 struct nvgpu_vfio_pci_core_device *nvdev = container_of(
104 core_vdev, struct nvgpu_vfio_pci_core_device, core_device.vdev);
105
106 unsigned long minsz = offsetofend(struct vfio_region_info, offset);
107 struct vfio_region_info info;
108
109 switch (cmd) {
110 case VFIO_DEVICE_GET_REGION_INFO:
111 if (copy_from_user(&info, (void __user *)arg, minsz))
112 return -EFAULT;
113
114 if (info.argsz < minsz)
115 return -EINVAL;
116
117 if (info.index == nvdev->mem_prop.bar1_start_offset) {
118 /*
119 * Request to determine the BAR1 region information. Send the
120 * GPU memory information.
121 */
122 info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
123 info.size = nvdev->mem_prop.mem_length;
124 info.flags = VFIO_REGION_INFO_FLAG_READ |
125 VFIO_REGION_INFO_FLAG_WRITE |
126 VFIO_REGION_INFO_FLAG_MMAP;
127 return copy_to_user((void __user *)arg, &info, minsz) ?
128 -EFAULT : 0;
129 }
130
131 if (info.index == nvdev->mem_prop.bar1_start_offset + 1) {
132 /*
133 * The BAR1 region is 64b. Ignore this access.
134 */
135 info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
136 info.size = 0;
137 info.flags = 0;
138 return copy_to_user((void __user *)arg, &info, minsz) ?
139 -EFAULT : 0;
140 }
141
142 return vfio_pci_core_ioctl(core_vdev, cmd, arg);
143
144 default:
145 return vfio_pci_core_ioctl(core_vdev, cmd, arg);
146 }
147 }
148

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
#13 lore raw

PATCH v3 3/6 mm: handle poisoning of pfn without struct pages

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

The kernel MM does not currently handle ECC errors / poison on a memory region that is not backed by struct pages. In this series, mapping request from QEMU to the device memory is executed using remap_pfn_range(). Hence added a new mechanism to handle memory failure on such memory.

Make kernel MM expose a function to allow modules managing the device memory to register a failure function and the address space that is associated with the device memory. MM maintains this information as interval tree. The registered memory failure function is used by MM to notify the module of the PFN, so that the module may take any required action. The module for example may use the information to track the poisoned pages.

In this implementation, kernel MM follows the following sequence (mostly) similar to the memory_failure() handler for struct page backed memory:
1. memory_failure() is triggered on reception of a poison error. An absence of struct page is detected and consequently memory_failure_pfn is executed.
2. memory_failure_pfn() call the newly introduced failure handler exposed by the module managing the poisoned memory to notify it of the problematic PFN.
3. memory_failure_pfn() unmaps the stage-2 mapping to the PFN.
4. memory_failure_pfn() collects the processes mapped to the PFN.
5. memory_failure_pfn() sends SIGBUS (BUS_MCEERR_AO) to all the processes mapping the faulty PFN using kill_procs().
6. An access to the faulty PFN by an operation in VM at a later point of time is trapped and user_mem_abort() is called.
7. user_mem_abort() calls __gfn_to_pfn_memslot() on the PFN, and the following execution path is followed: __gfn_to_pfn_memslot() -> hva_to_pfn() -> hva_to_pfn_remapped() -> fixup_user_fault() -> handle_mm_fault() -> handle_pte_fault() -> do_fault(). do_fault() is expected to return VM_FAULT_HWPOISON on the PFN (it currently does not and is fixed as part of another patch in the series).
8. __gfn_to_pfn_memslot() then returns KVM_PFN_ERR_HWPOISON, which cause the poison with SIGBUS (BUS_MCEERR_AR) to be sent to the QEMU process through kvm_send_hwpoison_signal().

---
 include/linux/memory-failure.h |  22 +++++
 include/linux/mm.h             |   1 +
 include/ras/ras_event.h        |   1 +
 mm/memory-failure.c            | 148 +++++++++++++++++++++++++++++----
 4 files changed, 154 insertions(+), 18 deletions(-)
 create mode 100644 include/linux/memory-failure.h
include/linux/memory-failure.h new file +22 -0
new file mode 100644
index 000000000000..9a579960972a
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_FAILURE_H
+#define _LINUX_MEMORY_FAILURE_H
+
+#include <linux/interval_tree.h>
+
+struct pfn_address_space;
+
+struct pfn_address_space_ops {
+	void (*failure)(struct pfn_address_space *pfn_space, unsigned long pfn);
+};
+
+struct pfn_address_space {
+	struct interval_tree_node node;
+	const struct pfn_address_space_ops *ops;
+	struct address_space *mapping;
+};
+
+int register_pfn_address_space(struct pfn_address_space *pfn_space);
+void unregister_pfn_address_space(struct pfn_address_space *pfn_space);
+
+#endif /* _LINUX_MEMORY_FAILURE_H */
include/linux/mm.h +1 -0
index 1f79667824eb..e3ef52d3d45a 100644
@@ -3530,6 +3530,7 @@ enum mf_action_page_type {
 	MF_MSG_BUDDY,
 	MF_MSG_DAX,
 	MF_MSG_UNSPLIT_THP,
+	MF_MSG_PFN,
 	MF_MSG_UNKNOWN,
 };
 
include/ras/ras_event.h +1 -0
index cbd3ddd7c33d..5c62a4d17172 100644
@@ -373,6 +373,7 @@ TRACE_EVENT(aer_event,
 	EM ( MF_MSG_BUDDY, "free buddy page" )				\
 	EM ( MF_MSG_DAX, "dax page" )					\
 	EM ( MF_MSG_UNSPLIT_THP, "unsplit thp" )			\
+	EM ( MF_MSG_PFN, "non struct page pfn" )					\
 	EMe ( MF_MSG_UNKNOWN, "unknown page" )
 
 /*
mm/memory-failure.c +130 -18

227 lines not on this page · raw message

Signature · 1 line
2.17.1
#14 lore raw

Re: [PATCH v3 3/6] mm: handle poisoning of pfn without struct pages

kernel test robot noticed the following build errors:

Hi,

kernel test robot noticed the following build errors:

[auto build test ERROR on awilliam-vfio/for-linus]
[also build test ERROR on kvmarm/next akpm-mm/mm-everything linus/master v6.3-rc5] [cannot apply to awilliam-vfio/next next-20230405]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/ankita-nvidia-com/kvm-determine-memory-type-from-VMA/20230406-020404
base:   https://github.com/awilliam/linux-vfio.git for-linus
patch link:    https://lore.kernel.org/r/20230405180134.16932-4-ankita%40nvidia.com
patch subject: [PATCH v3 3/6] mm: handle poisoning of pfn without struct pages
config: x86_64-randconfig-a015-20230403 (https://download.01.org/0day-ci/archive/20230406/[email protected]/config)
compiler: gcc-11 (Debian 11.3.0-8) 11.3.0
reproduce (this is a W=1 build):
        # https://github.com/intel-lab-lkp/linux/commit/25466c8c2fa22d39a08721a24f0cf3bc3059417b
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review ankita-nvidia-com/kvm-determine-memory-type-from-VMA/20230406-020404
        git checkout 25466c8c2fa22d39a08721a24f0cf3bc3059417b
        # save the config file
        mkdir build_dir && cp config build_dir/.config
        make W=1 O=build_dir ARCH=x86_64 olddefconfig
        make W=1 O=build_dir ARCH=x86_64 SHELL=/bin/bash

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

ld: vmlinux.o: in function `memory_failure_pfn':

>> mm/memory-failure.c:2124: undefined reference to `interval_tree_iter_first'
>> ld: mm/memory-failure.c:2125: undefined reference to `interval_tree_iter_next'
   ld: vmlinux.o: in function `register_pfn_address_space':
>> mm/memory-failure.c:2087: undefined reference to `interval_tree_insert'
   ld: vmlinux.o: in function `unregister_pfn_address_space':
>> mm/memory-failure.c:2105: undefined reference to `interval_tree_remove'

vim +2124 mm/memory-failure.c

2065
2066 /**
2067 * register_pfn_address_space - Register PA region for poison notification.
2068 * @pfn_space: structure containing region range and callback function on
2069 * poison detection.
2070 *
2071 * This function is called by a kernel module to register a PA region and
2072 * a callback function with the kernel. On detection of poison, the
2073 * kernel code will go through all registered regions and call the
2074 * appropriate callback function associated with the range. The kernel
2075 * module is responsible for tracking the poisoned pages.
2076 *
2077 * Return: 0 if successfully registered,
2078 * -EBUSY if the region is already registered
2079 */
2080 int register_pfn_address_space(struct pfn_address_space *pfn_space)
2081 {
2082 if (!request_mem_region(pfn_space->node.start << PAGE_SHIFT,
2083 (pfn_space->node.last - pfn_space->node.start + 1) << PAGE_SHIFT, ""))
2084 return -EBUSY;
2085
2086 mutex_lock(&pfn_space_lock);

> 2087		interval_tree_insert(&pfn_space->node, &pfn_space_itree);
  2088		mutex_unlock(&pfn_space_lock);
  2089	
  2090		return 0;
  2091	}
  2092	EXPORT_SYMBOL_GPL(register_pfn_address_space);
  2093	
  2094	/**
  2095	 * unregister_pfn_address_space - Unregister a PA region from poison
  2096	 * notification.
  2097	 * @pfn_space: structure containing region range to be unregistered.
  2098	 *
  2099	 * This function is called by a kernel module to unregister the PA region
  2100	 * from the kernel from poison tracking.
  2101	 */
  2102	void unregister_pfn_address_space(struct pfn_address_space *pfn_space)
  2103	{
  2104		mutex_lock(&pfn_space_lock);
> 2105		interval_tree_remove(&pfn_space->node, &pfn_space_itree);
  2106		mutex_unlock(&pfn_space_lock);
  2107		release_mem_region(pfn_space->node.start << PAGE_SHIFT,
  2108			(pfn_space->node.last - pfn_space->node.start + 1) << PAGE_SHIFT);
  2109	}
  2110	EXPORT_SYMBOL_GPL(unregister_pfn_address_space);
  2111	
  2112	static int memory_failure_pfn(unsigned long pfn, int flags)
  2113	{
  2114		struct interval_tree_node *node;
  2115		int rc = -EBUSY;
  2116		LIST_HEAD(tokill);
  2117	
  2118		mutex_lock(&pfn_space_lock);
  2119		/*
  2120		 * Modules registers with MM the address space mapping to the device memory they
  2121		 * manage. Iterate to identify exactly which address space has mapped to this
  2122		 * failing PFN.
  2123		 */
> 2124		for (node = interval_tree_iter_first(&pfn_space_itree, pfn, pfn); node;
> 2125		     node = interval_tree_iter_next(node, pfn, pfn)) {

2126 struct pfn_address_space *pfn_space =
2127 container_of(node, struct pfn_address_space, node);
2128 rc = 0;
2129
2130 /*
2131 * Modules managing the device memory needs to be conveyed about the
2132 * memory failure so that the poisoned PFN can be tracked.
2133 */
2134 pfn_space->ops->failure(pfn_space, pfn);
2135
2136 collect_procs_pgoff(NULL, pfn_space->mapping, pfn, &tokill);
2137
2138 unmap_mapping_range(pfn_space->mapping, pfn << PAGE_SHIFT,
2139 PAGE_SIZE, 0);
2140 }
2141 mutex_unlock(&pfn_space_lock);
2142
2143 /*
2144 * Unlike System-RAM there is no possibility to swap in a different
2145 * physical page at a given virtual address, so all userspace
2146 * consumption of direct PFN memory necessitates SIGBUS (i.e.
2147 * MF_MUST_KILL)
2148 */
2149 flags |= MF_ACTION_REQUIRED | MF_MUST_KILL;
2150 kill_procs(&tokill, true, false, pfn, flags);
2151
2152 pr_err("%#lx: recovery action for %s: %s\n",
2153 pfn, action_page_types[MF_MSG_PFN],
2154 action_name[rc ? MF_FAILED : MF_RECOVERED]);
2155
2156 return rc;
2157 }
2158

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
#15 lore raw

Re: [PATCH v3 3/6] mm: handle poisoning of pfn without struct pages

In reply to [email protected]
Quoted 81 lines · On Wed, Apr 05, 2023 at 11:01:31AM -0700, [email protected] wrote:

From: Ankit Agrawal <[email protected]>

The kernel MM does not currently handle ECC errors / poison on a memory region that is not backed by struct pages. In this series, mapping request from QEMU to the device memory is executed using remap_pfn_range(). Hence added a new mechanism to handle memory failure on such memory.

Make kernel MM expose a function to allow modules managing the device memory to register a failure function and the address space that is associated with the device memory. MM maintains this information as interval tree. The registered memory failure function is used by MM to notify the module of the PFN, so that the module may take any required action. The module for example may use the information to track the poisoned pages.

In this implementation, kernel MM follows the following sequence (mostly) similar to the memory_failure() handler for struct page backed memory:
1. memory_failure() is triggered on reception of a poison error. An absence of struct page is detected and consequently memory_failure_pfn is executed.
2. memory_failure_pfn() call the newly introduced failure handler exposed by the module managing the poisoned memory to notify it of the problematic PFN.
3. memory_failure_pfn() unmaps the stage-2 mapping to the PFN.
4. memory_failure_pfn() collects the processes mapped to the PFN.
5. memory_failure_pfn() sends SIGBUS (BUS_MCEERR_AO) to all the processes mapping the faulty PFN using kill_procs().
6. An access to the faulty PFN by an operation in VM at a later point of time is trapped and user_mem_abort() is called.
7. user_mem_abort() calls __gfn_to_pfn_memslot() on the PFN, and the following execution path is followed: __gfn_to_pfn_memslot() -> hva_to_pfn() -> hva_to_pfn_remapped() -> fixup_user_fault() -> handle_mm_fault() -> handle_pte_fault() -> do_fault(). do_fault() is expected to return VM_FAULT_HWPOISON on the PFN (it currently does not and is fixed as part of another patch in the series).
8. __gfn_to_pfn_memslot() then returns KVM_PFN_ERR_HWPOISON, which cause the poison with SIGBUS (BUS_MCEERR_AR) to be sent to the QEMU process through kvm_send_hwpoison_signal().

---
 include/linux/memory-failure.h |  22 +++++
 include/linux/mm.h             |   1 +
 include/ras/ras_event.h        |   1 +
 mm/memory-failure.c            | 148 +++++++++++++++++++++++++++++----
 4 files changed, 154 insertions(+), 18 deletions(-)
 create mode 100644 include/linux/memory-failure.h
include/linux/memory-failure.h new file +22 -0
new file mode 100644
index 000000000000..9a579960972a
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MEMORY_FAILURE_H
+#define _LINUX_MEMORY_FAILURE_H
+
+#include <linux/interval_tree.h>
+
+struct pfn_address_space;
+
+struct pfn_address_space_ops {
+	void (*failure)(struct pfn_address_space *pfn_space, unsigned long pfn);
+};
+
+struct pfn_address_space {
+	struct interval_tree_node node;
+	const struct pfn_address_space_ops *ops;
+	struct address_space *mapping;
+};
+
+int register_pfn_address_space(struct pfn_address_space *pfn_space);
+void unregister_pfn_address_space(struct pfn_address_space *pfn_space);
+
+#endif /* _LINUX_MEMORY_FAILURE_H */
include/linux/mm.h +0 -0
index 1f79667824eb..e3ef52d3d45a 100644

include/linux/mm.h enum mf_action_page_type { · line 3533

Quoted 4 lines
include/linux/mm.h +1 -0
@@ -3530,6 +3530,7 @@ enum mf_action_page_type {
 	MF_MSG_BUDDY,
 	MF_MSG_DAX,
 	MF_MSG_UNSPLIT_THP,
+	MF_MSG_PFN,

Personally, the keyword "PFN" looks to me a little too generic, so I prefer "PFNMAP" or "PFN_MAP" because memory_failure() is anyway called with pfn regardless of being backed by struct page.

Quoted 47 lines
include/linux/mm.h +0 -0
 	MF_MSG_UNKNOWN,
 };
 
include/ras/ras_event.h +1 -0
index cbd3ddd7c33d..5c62a4d17172 100644
@@ -373,6 +373,7 @@ TRACE_EVENT(aer_event,
 	EM ( MF_MSG_BUDDY, "free buddy page" )				\
 	EM ( MF_MSG_DAX, "dax page" )					\
 	EM ( MF_MSG_UNSPLIT_THP, "unsplit thp" )			\
+	EM ( MF_MSG_PFN, "non struct page pfn" )					\
 	EMe ( MF_MSG_UNKNOWN, "unknown page" )
 
 /*
mm/memory-failure.c +6 -0
index fae9baf3be16..2c1a2ec42f7b 100644
@@ -38,6 +38,7 @@
 
 #include <linux/kernel.h>
 #include <linux/mm.h>
+#include <linux/memory-failure.h>
 #include <linux/page-flags.h>
 #include <linux/kernel-page-flags.h>
 #include <linux/sched/signal.h>
@@ -62,6 +63,7 @@
 #include <linux/page-isolation.h>
 #include <linux/pagewalk.h>
 #include <linux/shmem_fs.h>
+#include <linux/pfn_t.h>
 #include "swap.h"
 #include "internal.h"
 #include "ras/ras_event.h"
@@ -122,6 +124,10 @@ const struct attribute_group memory_failure_attr_group = {
 	.attrs = memory_failure_attr,
 };
 
+static struct rb_root_cached pfn_space_itree = RB_ROOT_CACHED;
+
+static DEFINE_MUTEX(pfn_space_lock);
+
 /*
  * Return values:
  *   1:   the page is dissolved (if needed) and taken off from buddy,

mm/memory-failure.c static unsigned long dev_pagemap_mapping_shift(struct vm_area_struct *vma, · line 410

Quoted 10 lines
mm/memory-failure.c +3 -4
@@ -399,15 +405,14 @@ static unsigned long dev_pagemap_mapping_shift(struct vm_area_struct *vma,
  * Schedule a process for later kill.
  * Uses GFP_ATOMIC allocations to avoid potential recursions in the VM.
  *
- * Note: @fsdax_pgoff is used only when @p is a fsdax page and a
- * filesystem with a memory failure handler has claimed the
- * memory_failure event. In all other cases, page->index and
- * page->mapping are sufficient for mapping the page back to its
+ * Notice: @pgoff is used either when @p is a fsdax page or a PFN is not
+ * backed by struct page and a filesystem with a memory failure handler
+ * has claimed the memory_failure event. In all other cases, page->index

You add a new case using @pgoff, and now we have 3 such cases, so could you update the comment to itemize them (which makes it easier to read and update)?

Quoted 38 lines
mm/memory-failure.c +15 -8
+ * and page->mapping are sufficient for mapping the page back to its
  * corresponding user virtual address.
  */
-static void add_to_kill(struct task_struct *tsk, struct page *p,
-			pgoff_t fsdax_pgoff, struct vm_area_struct *vma,
-			struct list_head *to_kill)
+static void add_to_kill(struct task_struct *tsk, struct page *p, pgoff_t pgoff,
+			struct vm_area_struct *vma, struct list_head *to_kill)
 {
 	struct to_kill *tk;
 
@@ -417,13 +422,20 @@ static void add_to_kill(struct task_struct *tsk, struct page *p,
 		return;
 	}
 
-	tk->addr = page_address_in_vma(p, vma);
-	if (is_zone_device_page(p)) {
-		if (fsdax_pgoff != FSDAX_INVALID_PGOFF)
-			tk->addr = vma_pgoff_address(fsdax_pgoff, 1, vma);
+	if (vma->vm_flags | PFN_MAP) {
+		tk->addr =
+			vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
+		tk->size_shift = PAGE_SHIFT;
+	} else if (is_zone_device_page(p)) {
+		if (pgoff != FSDAX_INVALID_PGOFF)
+			tk->addr = vma_pgoff_address(pgoff, 1, vma);
+		else
+			tk->addr = page_address_in_vma(p, vma);
 		tk->size_shift = dev_pagemap_mapping_shift(vma, tk->addr);
-	} else
+	} else {
+		tk->addr = page_address_in_vma(p, vma);
 		tk->size_shift = page_shift(compound_head(p));
+	}
 
 	/*
 	 * Send SIGKILL if "tk->addr == -EFAULT". Also, as

mm/memory-failure.c static void collect_procs_file(struct page *page, struct list_head *to_kill, · line 620

Quoted 4 lines
mm/memory-failure.c +0 -1
@@ -617,13 +629,12 @@ static void collect_procs_file(struct page *page, struct list_head *to_kill,
 	i_mmap_unlock_read(mapping);
 }
 
-#ifdef CONFIG_FS_DAX

It seems that your new driver is built only in limited configuration/architecture, so loosening the condition instead of simply removing might be better.

Quoted 58 lines
mm/memory-failure.c +22 -5
 /*
  * Collect processes when the error hit a fsdax page.
  */
-static void collect_procs_fsdax(struct page *page,
-		struct address_space *mapping, pgoff_t pgoff,
-		struct list_head *to_kill)
+static void collect_procs_pgoff(struct page *page,
+				struct address_space *mapping, pgoff_t pgoff,
+				struct list_head *to_kill)
 {
 	struct vm_area_struct *vma;
 	struct task_struct *tsk;
@@ -643,7 +654,6 @@ static void collect_procs_fsdax(struct page *page,
 	read_unlock(&tasklist_lock);
 	i_mmap_unlock_read(mapping);
 }
-#endif /* CONFIG_FS_DAX */
 
 /*
  * Collect the processes who have the corrupted page mapped to kill.
@@ -835,6 +845,7 @@ static const char * const action_page_types[] = {
 	[MF_MSG_BUDDY]			= "free buddy page",
 	[MF_MSG_DAX]			= "dax page",
 	[MF_MSG_UNSPLIT_THP]		= "unsplit thp",
+	[MF_MSG_PFN]			= "non struct page pfn",
 	[MF_MSG_UNKNOWN]		= "unknown page",
 };
 
@@ -1745,7 +1756,7 @@ int mf_dax_kill_procs(struct address_space *mapping, pgoff_t index,
 
 		SetPageHWPoison(page);
 
-		collect_procs_fsdax(page, mapping, index, &to_kill);
+		collect_procs_pgoff(page, mapping, index, &to_kill);
 		unmap_and_kill(&to_kill, page_to_pfn(page), mapping,
 				index, mf_flags);
 unlock:
@@ -2052,6 +2063,99 @@ static int memory_failure_dev_pagemap(unsigned long pfn, int flags,
 	return rc;
 }
 
+/**
+ * register_pfn_address_space - Register PA region for poison notification.
+ * @pfn_space: structure containing region range and callback function on
+ *             poison detection.
+ *
+ * This function is called by a kernel module to register a PA region and
+ * a callback function with the kernel. On detection of poison, the
+ * kernel code will go through all registered regions and call the
+ * appropriate callback function associated with the range. The kernel
+ * module is responsible for tracking the poisoned pages.
+ *
+ * Return: 0 if successfully registered,
+ *         -EBUSY if the region is already registered
+ */
+int register_pfn_address_space(struct pfn_address_space *pfn_space)
+{
+	if (!request_mem_region(pfn_space->node.start << PAGE_SHIFT,

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags, · line 2092

Quoted 10 lines
mm/memory-failure.c +10 -0
+		(pfn_space->node.last - pfn_space->node.start + 1) << PAGE_SHIFT, ""))
+		return -EBUSY;
+
+	mutex_lock(&pfn_space_lock);
+	interval_tree_insert(&pfn_space->node, &pfn_space_itree);
+	mutex_unlock(&pfn_space_lock);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(register_pfn_address_space);

register_pfn_address_space and unregister_pfn_address_space are not compiled if CONFIG_MEMORY_FAILURE is not set, so maybe your new driver might need to depend on this config.

Quoted 39 lines
mm/memory-failure.c +39 -0
+
+/**
+ * unregister_pfn_address_space - Unregister a PA region from poison
+ * notification.
+ * @pfn_space: structure containing region range to be unregistered.
+ *
+ * This function is called by a kernel module to unregister the PA region
+ * from the kernel from poison tracking.
+ */
+void unregister_pfn_address_space(struct pfn_address_space *pfn_space)
+{
+	mutex_lock(&pfn_space_lock);
+	interval_tree_remove(&pfn_space->node, &pfn_space_itree);
+	mutex_unlock(&pfn_space_lock);
+	release_mem_region(pfn_space->node.start << PAGE_SHIFT,
+		(pfn_space->node.last - pfn_space->node.start + 1) << PAGE_SHIFT);
+}
+EXPORT_SYMBOL_GPL(unregister_pfn_address_space);
+
+static int memory_failure_pfn(unsigned long pfn, int flags)
+{
+	struct interval_tree_node *node;
+	int rc = -EBUSY;
+	LIST_HEAD(tokill);
+
+	mutex_lock(&pfn_space_lock);
+	/*
+	 * Modules registers with MM the address space mapping to the device memory they
+	 * manage. Iterate to identify exactly which address space has mapped to this
+	 * failing PFN.
+	 */
+	for (node = interval_tree_iter_first(&pfn_space_itree, pfn, pfn); node;
+	     node = interval_tree_iter_next(node, pfn, pfn)) {
+		struct pfn_address_space *pfn_space =
+			container_of(node, struct pfn_address_space, node);
+		rc = 0;
+
+		/*
+		 * Modules managing the device memory needs to be conveyed about the

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags, · line 2141

Quoted 10 lines
mm/memory-failure.c +10 -0
+		 * memory failure so that the poisoned PFN can be tracked.
+		 */
+		pfn_space->ops->failure(pfn_space, pfn);
+
+		collect_procs_pgoff(NULL, pfn_space->mapping, pfn, &tokill);
+
+		unmap_mapping_range(pfn_space->mapping, pfn << PAGE_SHIFT,
+				    PAGE_SIZE, 0);
+	}
+	mutex_unlock(&pfn_space_lock);

If rc == 0 at this point, the given pfn seems to be outside the GPU memory, so that should be considered as "Invalid address" case whose check is removed by patch 5/6. So it might be better to sperate the case from "do handling for non struct page pfn" case.

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags, · line 2154

Quoted 13 lines
mm/memory-failure.c +13 -0
+
+	/*
+	 * Unlike System-RAM there is no possibility to swap in a different
+	 * physical page at a given virtual address, so all userspace
+	 * consumption of direct PFN memory necessitates SIGBUS (i.e.
+	 * MF_MUST_KILL)
+	 */
+	flags |= MF_ACTION_REQUIRED | MF_MUST_KILL;
+	kill_procs(&tokill, true, false, pfn, flags);
+
+	pr_err("%#lx: recovery action for %s: %s\n",
+			pfn, action_page_types[MF_MSG_PFN],
+			action_name[rc ? MF_FAILED : MF_RECOVERED]);

Could you call action_result() to print out the summary line. It has some other common things like accounting and tracepoint.

Quoted 8 lines
mm/memory-failure.c +4 -0
+
+	return rc;
+}
+
 static DEFINE_MUTEX(mf_mutex);
 
 /**

mm/memory-failure.c int memory_failure(unsigned long pfn, int flags) · line 2203

Quoted 7 lines
mm/memory-failure.c +4 -0
@@ -2093,6 +2197,11 @@ int memory_failure(unsigned long pfn, int flags)
 	if (!(flags & MF_SW_SIMULATED))
 		hw_memory_failure = true;
 
+	if (!pfn_valid(pfn) && !arch_is_platform_page(PFN_PHYS(pfn))) {
+		res = memory_failure_pfn(pfn, flags);
+		goto unlock_mutex;
+	}

This might break exisiting corner case about DAX device, so maybe this should be checked after confirming that pfn_to_online_page returns NULL.

mm/memory-failure.c int memory_failure(unsigned long pfn, int flags) · line 2220

Quoted 11 lines
mm/memory-failure.c +4 -0
+
 	p = pfn_to_online_page(pfn);
 	if (!p) {
 		res = arch_memory_failure(pfn, flags);
@@ -2106,6 +2215,9 @@ int memory_failure(unsigned long pfn, int flags)
 								 pgmap);
 				goto unlock_mutex;
 			}
+
+			res = memory_failure_pfn(pfn, flags);
+			goto unlock_mutex;

This path is chosen when pfn_valid returns true, which cannot happen for GPU memory's case?

Thanks,
Naoya Horiguchi

Quoted 5 lines
mm/memory-failure.c +0 -0
 		}
 		pr_err("%#lx: memory outside kernel control\n", pfn);
 		res = -ENXIO;
Signature · 1 line
2.17.1
#16 lore raw

RE: [PATCH v3 3/6] mm: handle poisoning of pfn without struct pages

Thanks, Naoya for reviewing the patch. Comments inline.

Quoted 103 lines

-----Original Message-----
From: HORIGUCHI NAOYA(堀口 直也) <[email protected]>
Sent: Tuesday, May 9, 2023 2:51 AM
To: Ankit Agrawal <[email protected]>
Cc: Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Aniket Agashe <[email protected]>; Neo Jia <[email protected]>; Kirti Wankhede <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Vikram Sethi <[email protected]>; Andy Currid <[email protected]>; Alistair Popple <[email protected]>; John Hubbard <[email protected]>; Dan Williams <[email protected]>; [email protected]; [email protected]; linux-arm- [email protected]; [email protected]
Subject: Re: [PATCH v3 3/6] mm: handle poisoning of pfn without struct pages

External email: Use caution opening links or attachments

Quoted 5 lines · On Wed, Apr 05, 2023 at 11:01:31AM -0700, [email protected] wrote:

From: Ankit Agrawal <[email protected]>

The kernel MM does not currently handle ECC errors / poison on a memory region that is not backed by struct pages. In this series, mapping request from QEMU to the device memory is executed using

remap_pfn_range().

Quoted 10 lines

Hence added a new mechanism to handle memory failure on such memory.

Make kernel MM expose a function to allow modules managing the device memory to register a failure function and the address space that is associated with the device memory. MM maintains this information as interval tree. The registered memory failure function is used by MM to notify the module of the PFN, so that the module may take any required action. The module for example may use the information to track the poisoned pages.

Quoted 2 lines

In this implementation, kernel MM follows the following sequence (mostly) similar to the memory_failure() handler for struct page backed

memory:

Quoted 11 lines

1. memory_failure() is triggered on reception of a poison error. An absence of struct page is detected and consequently memory_failure_pfn is executed.
2. memory_failure_pfn() call the newly introduced failure handler exposed by the module managing the poisoned memory to notify it of the problematic PFN.
3. memory_failure_pfn() unmaps the stage-2 mapping to the PFN.
4. memory_failure_pfn() collects the processes mapped to the PFN.
5. memory_failure_pfn() sends SIGBUS (BUS_MCEERR_AO) to all the processes mapping the faulty PFN using kill_procs().
6. An access to the faulty PFN by an operation in VM at a later point

Quoted 8 lines

of time is trapped and user_mem_abort() is called.
7. user_mem_abort() calls __gfn_to_pfn_memslot() on the PFN, and the following execution path is followed: __gfn_to_pfn_memslot() -> hva_to_pfn() -> hva_to_pfn_remapped() -> fixup_user_fault() -> handle_mm_fault() -> handle_pte_fault() -> do_fault(). do_fault() is expected to return VM_FAULT_HWPOISON on the PFN (it currently does not and is fixed as part of another patch in the series).
8. __gfn_to_pfn_memslot() then returns KVM_PFN_ERR_HWPOISON,

which

Quoted 45 lines

cause the poison with SIGBUS (BUS_MCEERR_AR) to be sent to the QEMU process through kvm_send_hwpoison_signal().

---
 include/linux/memory-failure.h |  22 +++++
 include/linux/mm.h             |   1 +
 include/ras/ras_event.h        |   1 +
 mm/memory-failure.c            | 148 +++++++++++++++++++++++++++++----
 4 files changed, 154 insertions(+), 18 deletions(-)  create mode

100644 include/linux/memory-failure.h

(quoted diff) +0 -0

b/include/linux/memory-failure.h new file mode 100644 index 000000000000..9a579960972a

include/linux/memory-failure.h new file +22 -0
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */ #ifndef
+_LINUX_MEMORY_FAILURE_H #define _LINUX_MEMORY_FAILURE_H
+
+#include <linux/interval_tree.h>
+
+struct pfn_address_space;
+
+struct pfn_address_space_ops {
+     void (*failure)(struct pfn_address_space *pfn_space, unsigned
+long pfn); };
+
+struct pfn_address_space {
+     struct interval_tree_node node;
+     const struct pfn_address_space_ops *ops;
+     struct address_space *mapping;
+};
+
+int register_pfn_address_space(struct pfn_address_space *pfn_space);
+void unregister_pfn_address_space(struct pfn_address_space
+*pfn_space);
+
+#endif /* _LINUX_MEMORY_FAILURE_H */
include/linux/mm.h index renamed from include/linux/mm.h +0 -0

1f79667824eb..e3ef52d3d45a 100644

include/linux/mm.h +0 -0
@@ -3530,6 +3530,7 @@ enum mf_action_page_type {
Quoted 9 lines

include/linux/mm.h enum mf_action_page_type { · line 3533

Quoted 4 lines
include/linux/mm.h +1 -0
      MF_MSG_BUDDY,
      MF_MSG_DAX,
      MF_MSG_UNSPLIT_THP,
+     MF_MSG_PFN,

Personally, the keyword "PFN" looks to me a little too generic, so I prefer "PFNMAP" or "PFN_MAP" because memory_failure() is anyway called with pfn regardless of being backed by struct page.

Makes sense. Will change to PFNMAP.

Quoted 63 lines

mm/memory-failure.c const struct attribute_group

Quoted 36 lines
include/linux/mm.h +0 -0
      MF_MSG_UNKNOWN,
 };
 
include/ras/ras_event.h index renamed from include/ras/ras_event.h +0 -0

cbd3ddd7c33d..5c62a4d17172 100644

include/ras/ras_event.h +1 -0
@@ -373,6 +373,7 @@ TRACE_EVENT(aer_event,
      EM ( MF_MSG_BUDDY, "free buddy page" )                          \
      EM ( MF_MSG_DAX, "dax page" )                                   \
      EM ( MF_MSG_UNSPLIT_THP, "unsplit thp" )                        \
+     EM ( MF_MSG_PFN, "non struct page pfn" )                                        \
      EMe ( MF_MSG_UNKNOWN, "unknown page" )
 
 /*
mm/memory-failure.c index renamed from mm/memory-failure.c +0 -0

fae9baf3be16..2c1a2ec42f7b 100644

mm/memory-failure.c +2 -0
@@ -38,6 +38,7 @@
 
 #include <linux/kernel.h>
 #include <linux/mm.h>
+#include <linux/memory-failure.h>
 #include <linux/page-flags.h>
 #include <linux/kernel-page-flags.h>
 #include <linux/sched/signal.h>
@@ -62,6 +63,7 @@
 #include <linux/page-isolation.h>
 #include <linux/pagewalk.h>
 #include <linux/shmem_fs.h>
+#include <linux/pfn_t.h>
 #include "swap.h"
 #include "internal.h"
 #include "ras/ras_event.h"
@@ -122,6 +124,10 @@ const struct attribute_group

memory_failure_attr_group = {

mm/memory-failure.c static unsigned long

Quoted 11 lines
mm/memory-failure.c +4 -0
      .attrs = memory_failure_attr,
 };
 
+static struct rb_root_cached pfn_space_itree = RB_ROOT_CACHED;
+
+static DEFINE_MUTEX(pfn_space_lock);
+
 /*
  * Return values:
  *   1:   the page is dissolved (if needed) and taken off from buddy,
@@ -399,15 +405,14 @@ static unsigned long

dev_pagemap_mapping_shift(struct vm_area_struct *vma,

mm/memory-failure.c static unsigned long dev_pagemap_mapping_shift(struct vm_area_struct *vma,

Quoted 13 lines
mm/memory-failure.c +6 -4
  * Schedule a process for later kill.
  * Uses GFP_ATOMIC allocations to avoid potential recursions in the VM.
  *
- * Note: @fsdax_pgoff is used only when @p is a fsdax page and a
- * filesystem with a memory failure handler has claimed the
- * memory_failure event. In all other cases, page->index and
- * page->mapping are sufficient for mapping the page back to its
+ * Notice: @pgoff is used either when @p is a fsdax page or a PFN is
+ not
+ * backed by struct page and a filesystem with a memory failure
+ handler
+ * has claimed the memory_failure event. In all other cases,
+ page->index
Quoted 4 lines

You add a new case using @pgoff, and now we have 3 such cases, so could you update the comment to itemize them (which makes it easier to read and update)?

Sure, will do in the next version.

Quoted 113 lines

mm/memory-failure.c static void add_to_kill(struct task_struct *tsk,

Quoted 13 lines
mm/memory-failure.c +4 -3
+ * and page->mapping are sufficient for mapping the page back to its
  * corresponding user virtual address.
  */
-static void add_to_kill(struct task_struct *tsk, struct page *p,
-                     pgoff_t fsdax_pgoff, struct vm_area_struct *vma,
-                     struct list_head *to_kill)
+static void add_to_kill(struct task_struct *tsk, struct page *p, pgoff_t pgoff,
+                     struct vm_area_struct *vma, struct list_head
+*to_kill)
 {
      struct to_kill *tk;
@@ -417,13 +422,20 @@ static void add_to_kill(struct task_struct *tsk,

struct page *p,

Quoted 16 lines
mm/memory-failure.c +9 -4
              return;
      }
 
-     tk->addr = page_address_in_vma(p, vma);
-     if (is_zone_device_page(p)) {
-             if (fsdax_pgoff != FSDAX_INVALID_PGOFF)
-                     tk->addr = vma_pgoff_address(fsdax_pgoff, 1, vma);
+     if (vma->vm_flags | PFN_MAP) {
+             tk->addr =
+                     vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
+             tk->size_shift = PAGE_SHIFT;
+     } else if (is_zone_device_page(p)) {
+             if (pgoff != FSDAX_INVALID_PGOFF)
+                     tk->addr = vma_pgoff_address(pgoff, 1, vma);
+             else
+                     tk->addr = page_address_in_vma(p, vma);

mm/memory-failure.c static void add_to_kill(struct task_struct *tsk, struct page *p,

Quoted 10 lines
mm/memory-failure.c +4 -1
              tk->size_shift = dev_pagemap_mapping_shift(vma, tk->addr);
-     } else
+     } else {
+             tk->addr = page_address_in_vma(p, vma);
              tk->size_shift = page_shift(compound_head(p));
+     }
 
      /*
       * Send SIGKILL if "tk->addr == -EFAULT". Also, as @@ -617,13
+629,12 @@ static void collect_procs_file(struct page *page, struct list_head

*to_kill,

Quoted 4 lines
      i_mmap_unlock_read(mapping);

}

-#ifdef CONFIG_FS_DAX

It seems that your new driver is built only in limited configuration/architecture, so loosening the condition instead of simply removing might be better.

mm/memory-failure.c static int

Quoted 39 lines

/*
* Collect processes when the error hit a fsdax page. */

mm/memory-failure.c +5 -5
-static void collect_procs_fsdax(struct page *page,
-             struct address_space *mapping, pgoff_t pgoff,
-             struct list_head *to_kill)
+static void collect_procs_pgoff(struct page *page,
+                             struct address_space *mapping, pgoff_t pgoff,
+                             struct list_head *to_kill)
 {
      struct vm_area_struct *vma;
      struct task_struct *tsk;
@@ -643,7 +654,6 @@ static void collect_procs_fsdax(struct page *page,
      read_unlock(&tasklist_lock);
      i_mmap_unlock_read(mapping);
 }
-#endif /* CONFIG_FS_DAX */
 
 /*
  * Collect the processes who have the corrupted page mapped to kill.
@@ -835,6 +845,7 @@ static const char * const action_page_types[] = {
      [MF_MSG_BUDDY]                  = "free buddy page",
      [MF_MSG_DAX]                    = "dax page",
      [MF_MSG_UNSPLIT_THP]            = "unsplit thp",
+     [MF_MSG_PFN]                    = "non struct page pfn",
      [MF_MSG_UNKNOWN]                = "unknown page",
 };
 
@@ -1745,7 +1756,7 @@ int mf_dax_kill_procs(struct address_space
 *mapping, pgoff_t index,
 
              SetPageHWPoison(page);
 
-             collect_procs_fsdax(page, mapping, index, &to_kill);
+             collect_procs_pgoff(page, mapping, index, &to_kill);
              unmap_and_kill(&to_kill, page_to_pfn(page), mapping,
                              index, mf_flags);

unlock:

mm/memory-failure.c +0 -0
@@ -2052,6 +2063,99 @@ static int

memory_failure_dev_pagemap(unsigned long pfn, int flags,

Quoted 12 lines
mm/memory-failure.c +9 -0
      return rc;
 }
 
+/**
+ * register_pfn_address_space - Register PA region for poison notification.
+ * @pfn_space: structure containing region range and callback function on
+ *             poison detection.
+ *
+ * This function is called by a kernel module to register a PA region
+and
+ * a callback function with the kernel. On detection of poison, the
+ * kernel code will go through all registered regions and call the

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags,

Quoted 10 lines
mm/memory-failure.c +10 -0
+ * appropriate callback function associated with the range. The
+kernel
+ * module is responsible for tracking the poisoned pages.
+ *
+ * Return: 0 if successfully registered,
+ *         -EBUSY if the region is already registered
+ */
+int register_pfn_address_space(struct pfn_address_space *pfn_space) {
+     if (!request_mem_region(pfn_space->node.start << PAGE_SHIFT,
+             (pfn_space->node.last - pfn_space->node.start + 1) << PAGE_SHIFT,

""))

Quoted 14 lines

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags, · line 2092

Quoted 9 lines
mm/memory-failure.c +9 -0
+             return -EBUSY;
+
+     mutex_lock(&pfn_space_lock);
+     interval_tree_insert(&pfn_space->node, &pfn_space_itree);
+     mutex_unlock(&pfn_space_lock);
+
+     return 0;
+}
+EXPORT_SYMBOL_GPL(register_pfn_address_space);

register_pfn_address_space and unregister_pfn_address_space are not compiled if CONFIG_MEMORY_FAILURE is not set, so maybe your new driver might need to depend on this config.

Yeah, that's right. Shall I put parts of code in the vfio-pci variant driver related to poison handling as part of #ifdef CONFIG_MEMORY_FAILURE. Or would you rather prefer I make NVGPU_VFIO_PCI config dependent on CONFIG_MEMORY_FAILURE?

Quoted 41 lines
Quoted 17 lines
mm/memory-failure.c +17 -0
+
+/**
+ * unregister_pfn_address_space - Unregister a PA region from poison
+ * notification.
+ * @pfn_space: structure containing region range to be unregistered.
+ *
+ * This function is called by a kernel module to unregister the PA
+region
+ * from the kernel from poison tracking.
+ */
+void unregister_pfn_address_space(struct pfn_address_space
+*pfn_space) {
+     mutex_lock(&pfn_space_lock);
+     interval_tree_remove(&pfn_space->node, &pfn_space_itree);
+     mutex_unlock(&pfn_space_lock);
+     release_mem_region(pfn_space->node.start << PAGE_SHIFT,
+             (pfn_space->node.last - pfn_space->node.start + 1) <<

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags,

Quoted 10 lines
mm/memory-failure.c +10 -0
+PAGE_SHIFT); } EXPORT_SYMBOL_GPL(unregister_pfn_address_space);
+
+static int memory_failure_pfn(unsigned long pfn, int flags) {
+     struct interval_tree_node *node;
+     int rc = -EBUSY;
+     LIST_HEAD(tokill);
+
+     mutex_lock(&pfn_space_lock);
+     /*
+      * Modules registers with MM the address space mapping to the device

memory they

mm/memory-failure.c static int · line 2121

Quoted 1 line
mm/memory-failure.c +1 -0
+      * manage. Iterate to identify exactly which address space has mapped to

this

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags,

Quoted 10 lines
mm/memory-failure.c +10 -0
+      * failing PFN.
+      */
+     for (node = interval_tree_iter_first(&pfn_space_itree, pfn, pfn); node;
+          node = interval_tree_iter_next(node, pfn, pfn)) {
+             struct pfn_address_space *pfn_space =
+                     container_of(node, struct pfn_address_space, node);
+             rc = 0;
+
+             /*
+              * Modules managing the device memory needs to be conveyed

about the

Quoted 17 lines

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags, · line 2141

Quoted 11 lines
mm/memory-failure.c +11 -0
+              * memory failure so that the poisoned PFN can be tracked.
+              */
+             pfn_space->ops->failure(pfn_space, pfn);
+
+             collect_procs_pgoff(NULL, pfn_space->mapping, pfn,
+ &tokill);
+
+             unmap_mapping_range(pfn_space->mapping, pfn << PAGE_SHIFT,
+                                 PAGE_SIZE, 0);
+     }
+     mutex_unlock(&pfn_space_lock);

If rc == 0 at this point, the given pfn seems to be outside the GPU memory, so that should be considered as "Invalid address" case whose check is removed by patch 5/6. So it might be better to sperate the case from "do handling for non struct page pfn" case.

Sorry did you mean rc !=0 here? But yeah, you are right that I should add the case for check in case a region with the desired PFN isn't found above.

Quoted 8 lines
Quoted 8 lines
mm/memory-failure.c +8 -0
+
+     /*
+      * Unlike System-RAM there is no possibility to swap in a different
+      * physical page at a given virtual address, so all userspace
+      * consumption of direct PFN memory necessitates SIGBUS (i.e.
+      * MF_MUST_KILL)
+      */
+     flags |= MF_ACTION_REQUIRED | MF_MUST_KILL;
Quoted 9 lines

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags, · line 2154

Quoted 5 lines
mm/memory-failure.c +5 -0
+     kill_procs(&tokill, true, false, pfn, flags);
+
+     pr_err("%#lx: recovery action for %s: %s\n",
+                     pfn, action_page_types[MF_MSG_PFN],
+                     action_name[rc ? MF_FAILED : MF_RECOVERED]);

Could you call action_result() to print out the summary line. It has some other common things like accounting and tracepoint.

Ack.

Quoted 11 lines
Quoted 11 lines
mm/memory-failure.c +4 -0
+
+     return rc;
+}
+
 static DEFINE_MUTEX(mf_mutex);
 
 /**
@@ -2093,6 +2197,11 @@ int memory_failure(unsigned long pfn, int flags)
      if (!(flags & MF_SW_SIMULATED))
              hw_memory_failure = true;
 
Quoted 8 lines

mm/memory-failure.c int memory_failure(unsigned long pfn, int flags) · line 2203

Quoted 4 lines
mm/memory-failure.c +4 -0
+     if (!pfn_valid(pfn) && !arch_is_platform_page(PFN_PHYS(pfn))) {
+             res = memory_failure_pfn(pfn, flags);
+             goto unlock_mutex;
+     }

This might break exisiting corner case about DAX device, so maybe this should be checked after confirming that pfn_to_online_page returns NULL.

Sorry, it isn't clear to me which corner case could break here. Can the pfn_valid() be false on a DAX device page?

Quoted 6 lines
Quoted 6 lines
mm/memory-failure.c +2 -0
+
      p = pfn_to_online_page(pfn);
      if (!p) {
              res = arch_memory_failure(pfn, flags); @@ -2106,6
+2215,9 @@ int memory_failure(unsigned long pfn, int flags)
                                                               pgmap);
Quoted 9 lines

mm/memory-failure.c int memory_failure(unsigned long pfn, int flags) · line 2220

Quoted 5 lines
                              goto unlock_mutex;
                      }
mm/memory-failure.c +3 -0
+
+                     res = memory_failure_pfn(pfn, flags);
+                     goto unlock_mutex;

This path is chosen when pfn_valid returns true, which cannot happen for GPU memory's case?

Good catch. That needs to be removed.

Quoted 8 lines

Thanks,
Naoya Horiguchi

Quoted 5 lines
              }
              pr_err("%#lx: memory outside kernel control\n", pfn);
              res = -ENXIO;

--
2.17.1

On a separate note, would you rather prefer that I separate out the poison handling parts (i.e. 3/6 and 5/6) into a separate series? Or should I just keep the whole series together?

Thanks
Ankit Agrawal

#17 lore raw

Re: [PATCH v3 3/6] mm: handle poisoning of pfn without struct pages

Quoted 72 lines · Ankit Agrawal

Thanks, Naoya for reviewing the patch. Comments inline.

Quoted 70 lines

-----Original Message-----
From: HORIGUCHI NAOYA(堀口 直也) <[email protected]>
Sent: Tuesday, May 9, 2023 2:51 AM
To: Ankit Agrawal <[email protected]>
Cc: Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Aniket Agashe <[email protected]>; Neo Jia <[email protected]>; Kirti Wankhede <[email protected]>; Tarun Gupta (SW-GPU) <[email protected]>; Vikram Sethi <[email protected]>; Andy Currid <[email protected]>; Alistair Popple <[email protected]>; John Hubbard <[email protected]>; Dan Williams <[email protected]>; [email protected]; [email protected]; linux-arm- [email protected]; [email protected]
Subject: Re: [PATCH v3 3/6] mm: handle poisoning of pfn without struct pages

External email: Use caution opening links or attachments

Quoted 5 lines · On Wed, Apr 05, 2023 at 11:01:31AM -0700, [email protected] wrote:

From: Ankit Agrawal <[email protected]>

The kernel MM does not currently handle ECC errors / poison on a memory region that is not backed by struct pages. In this series, mapping request from QEMU to the device memory is executed using

remap_pfn_range().

Quoted 10 lines

Hence added a new mechanism to handle memory failure on such memory.

Make kernel MM expose a function to allow modules managing the device memory to register a failure function and the address space that is associated with the device memory. MM maintains this information as interval tree. The registered memory failure function is used by MM to notify the module of the PFN, so that the module may take any required action. The module for example may use the information to track the poisoned pages.

Quoted 2 lines

In this implementation, kernel MM follows the following sequence (mostly) similar to the memory_failure() handler for struct page backed

memory:

Quoted 11 lines

1. memory_failure() is triggered on reception of a poison error. An absence of struct page is detected and consequently memory_failure_pfn is executed.
2. memory_failure_pfn() call the newly introduced failure handler exposed by the module managing the poisoned memory to notify it of the problematic PFN.
3. memory_failure_pfn() unmaps the stage-2 mapping to the PFN.
4. memory_failure_pfn() collects the processes mapped to the PFN.
5. memory_failure_pfn() sends SIGBUS (BUS_MCEERR_AO) to all the processes mapping the faulty PFN using kill_procs().
6. An access to the faulty PFN by an operation in VM at a later point

Quoted 8 lines

of time is trapped and user_mem_abort() is called.
7. user_mem_abort() calls __gfn_to_pfn_memslot() on the PFN, and the following execution path is followed: __gfn_to_pfn_memslot() -> hva_to_pfn() -> hva_to_pfn_remapped() -> fixup_user_fault() -> handle_mm_fault() -> handle_pte_fault() -> do_fault(). do_fault() is expected to return VM_FAULT_HWPOISON on the PFN (it currently does not and is fixed as part of another patch in the series).
8. __gfn_to_pfn_memslot() then returns KVM_PFN_ERR_HWPOISON,

which

Quoted 12 lines

cause the poison with SIGBUS (BUS_MCEERR_AR) to be sent to the QEMU process through kvm_send_hwpoison_signal().

---
 include/linux/memory-failure.h |  22 +++++
 include/linux/mm.h             |   1 +
 include/ras/ras_event.h        |   1 +
 mm/memory-failure.c            | 148 +++++++++++++++++++++++++++++----
 4 files changed, 154 insertions(+), 18 deletions(-)  create mode

100644 include/linux/memory-failure.h

...

Quoted 40 lines
Quoted 25 lines

Review static int

Quoted 1 line
(quoted diff) +0 -0
@@ -2052,6 +2063,99 @@ static int

memory_failure_dev_pagemap(unsigned long pfn, int flags,

Quoted 12 lines
mm/memory-failure.c +9 -0
      return rc;
 }
 
+/**
+ * register_pfn_address_space - Register PA region for poison notification.
+ * @pfn_space: structure containing region range and callback function on
+ *             poison detection.
+ *
+ * This function is called by a kernel module to register a PA region
+and
+ * a callback function with the kernel. On detection of poison, the
+ * kernel code will go through all registered regions and call the

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags,

Quoted 10 lines
mm/memory-failure.c +10 -0
+ * appropriate callback function associated with the range. The
+kernel
+ * module is responsible for tracking the poisoned pages.
+ *
+ * Return: 0 if successfully registered,
+ *         -EBUSY if the region is already registered
+ */
+int register_pfn_address_space(struct pfn_address_space *pfn_space) {
+     if (!request_mem_region(pfn_space->node.start << PAGE_SHIFT,
+             (pfn_space->node.last - pfn_space->node.start + 1) << PAGE_SHIFT,

""))

Quoted 14 lines

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags, · line 2092

Quoted 9 lines
mm/memory-failure.c +9 -0
+             return -EBUSY;
+
+     mutex_lock(&pfn_space_lock);
+     interval_tree_insert(&pfn_space->node, &pfn_space_itree);
+     mutex_unlock(&pfn_space_lock);
+
+     return 0;
+}
+EXPORT_SYMBOL_GPL(register_pfn_address_space);

register_pfn_address_space and unregister_pfn_address_space are not compiled if CONFIG_MEMORY_FAILURE is not set, so maybe your new driver might need to depend on this config.

Quoted 4 lines

Yeah, that's right. Shall I put parts of code in the vfio-pci variant driver related to poison handling as part of #ifdef CONFIG_MEMORY_FAILURE. Or would you rather prefer I make NVGPU_VFIO_PCI config dependent on CONFIG_MEMORY_FAILURE?

If the vfio-pci variant driver plans to provide features unrelated to poison handling, "ifdef" option looks fine to me. Otherwise, the second option could be possible. Both options look to me comparably reasonable.

Quoted 103 lines
Quoted 42 lines
Quoted 18 lines
mm/memory-failure.c +17 -0
+
+/**
+ * unregister_pfn_address_space - Unregister a PA region from poison
+ * notification.
+ * @pfn_space: structure containing region range to be unregistered.
+ *
+ * This function is called by a kernel module to unregister the PA
+region
+ * from the kernel from poison tracking.
+ */
+void unregister_pfn_address_space(struct pfn_address_space
+*pfn_space) {
+     mutex_lock(&pfn_space_lock);
+     interval_tree_remove(&pfn_space->node, &pfn_space_itree);
+     mutex_unlock(&pfn_space_lock);
+     release_mem_region(pfn_space->node.start << PAGE_SHIFT,
+             (pfn_space->node.last - pfn_space->node.start + 1) <<

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags,

Quoted 10 lines
mm/memory-failure.c +10 -0
+PAGE_SHIFT); } EXPORT_SYMBOL_GPL(unregister_pfn_address_space);
+
+static int memory_failure_pfn(unsigned long pfn, int flags) {
+     struct interval_tree_node *node;
+     int rc = -EBUSY;
+     LIST_HEAD(tokill);
+
+     mutex_lock(&pfn_space_lock);
+     /*
+      * Modules registers with MM the address space mapping to the device

memory they

Review static int · line 2121

Quoted 1 line
(quoted diff) +1 -0
+      * manage. Iterate to identify exactly which address space has mapped to

this

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags,

Quoted 10 lines
mm/memory-failure.c +10 -0
+      * failing PFN.
+      */
+     for (node = interval_tree_iter_first(&pfn_space_itree, pfn, pfn); node;
+          node = interval_tree_iter_next(node, pfn, pfn)) {
+             struct pfn_address_space *pfn_space =
+                     container_of(node, struct pfn_address_space, node);
+             rc = 0;
+
+             /*
+              * Modules managing the device memory needs to be conveyed

about the

Quoted 17 lines

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags, · line 2141

Quoted 11 lines
mm/memory-failure.c +11 -0
+              * memory failure so that the poisoned PFN can be tracked.
+              */
+             pfn_space->ops->failure(pfn_space, pfn);
+
+             collect_procs_pgoff(NULL, pfn_space->mapping, pfn,
+ &tokill);
+
+             unmap_mapping_range(pfn_space->mapping, pfn << PAGE_SHIFT,
+                                 PAGE_SIZE, 0);
+     }
+     mutex_unlock(&pfn_space_lock);

If rc == 0 at this point, the given pfn seems to be outside the GPU memory, so that should be considered as "Invalid address" case whose check is removed by patch 5/6. So it might be better to sperate the case from "do handling for non struct page pfn" case.

Sorry did you mean rc !=0 here? But yeah, you are right that I should add the case for check in case a region with the desired PFN isn't found above.

Quoted 8 lines
Quoted 8 lines
mm/memory-failure.c +8 -0
+
+     /*
+      * Unlike System-RAM there is no possibility to swap in a different
+      * physical page at a given virtual address, so all userspace
+      * consumption of direct PFN memory necessitates SIGBUS (i.e.
+      * MF_MUST_KILL)
+      */
+     flags |= MF_ACTION_REQUIRED | MF_MUST_KILL;
Quoted 9 lines

mm/memory-failure.c static int memory_failure_dev_pagemap(unsigned long pfn, int flags, · line 2154

Quoted 5 lines
mm/memory-failure.c +5 -0
+     kill_procs(&tokill, true, false, pfn, flags);
+
+     pr_err("%#lx: recovery action for %s: %s\n",
+                     pfn, action_page_types[MF_MSG_PFN],
+                     action_name[rc ? MF_FAILED : MF_RECOVERED]);

Could you call action_result() to print out the summary line. It has some other common things like accounting and tracepoint.

Ack.

Quoted 11 lines
Quoted 11 lines
mm/memory-failure.c +4 -0
+
+     return rc;
+}
+
 static DEFINE_MUTEX(mf_mutex);
 
 /**
@@ -2093,6 +2197,11 @@ int memory_failure(unsigned long pfn, int flags)
      if (!(flags & MF_SW_SIMULATED))
              hw_memory_failure = true;
 
Quoted 8 lines

mm/memory-failure.c int memory_failure(unsigned long pfn, int flags) · line 2203

Quoted 4 lines
mm/memory-failure.c +4 -0
+     if (!pfn_valid(pfn) && !arch_is_platform_page(PFN_PHYS(pfn))) {
+             res = memory_failure_pfn(pfn, flags);
+             goto unlock_mutex;
+     }

This might break exisiting corner case about DAX device, so maybe this should be checked after confirming that pfn_to_online_page returns NULL.

Quoted 2 lines

Sorry, it isn't clear to me which corner case could break here. Can the pfn_valid() be false on a DAX device page?

I thought that possibility, but I commented with relying on my vague/wrong memory, sorry. pfn_valid() should always true for DAX device pages, so the above check should not break.

Quoted 28 lines
Quoted 16 lines

mm/memory-failure.c int memory_failure(unsigned long pfn, int flags) · line 2220

Quoted 12 lines
mm/memory-failure.c +5 -0
+
      p = pfn_to_online_page(pfn);
      if (!p) {
              res = arch_memory_failure(pfn, flags); @@ -2106,6
+2215,9 @@ int memory_failure(unsigned long pfn, int flags)
                                                               pgmap);
                              goto unlock_mutex;
                      }
+
+                     res = memory_failure_pfn(pfn, flags);
+                     goto unlock_mutex;

This path is chosen when pfn_valid returns true, which cannot happen for GPU memory's case?

Good catch. That needs to be removed.

Quoted 8 lines

Thanks,
Naoya Horiguchi

Quoted 5 lines
              }
              pr_err("%#lx: memory outside kernel control\n", pfn);
              res = -ENXIO;

--
2.17.1

Quoted 3 lines

On a separate note, would you rather prefer that I separate out the poison handling parts (i.e. 3/6 and 5/6) into a separate series? Or should I just keep the whole series together?

The new poison handling code is used after the vfio-pci driver is available, so I think that they may as well be merged together.

Thanks,
Naoya Horiguchi

#18 lore raw

PATCH v3 4/6 mm: Add poison error check in fixup_user_fault() for mapped PFN

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

The fixup_user_fault() currently does not expect a VM_FAULT_HWPOISON and hence does not check for it while calling vm_fault_to_errno(). Since we now have a new code path which can trigger such case, change fixup_user_fault to look for VM_FAULT_HWPOISON.

Also make hva_to_pfn_remapped check for -EHWPOISON and communicate the poison fault up to the user_mem_abort().

---
 mm/gup.c            | 2 +-
 virt/kvm/kvm_main.c | 6 ++++++
 2 files changed, 7 insertions(+), 1 deletion(-)
mm/gup.c +1 -1
index eab18ba045db..507a96e91bad 100644
@@ -1290,7 +1290,7 @@ int fixup_user_fault(struct mm_struct *mm,
 	}
 
 	if (ret & VM_FAULT_ERROR) {
-		int err = vm_fault_to_errno(ret, 0);
+		int err = vm_fault_to_errno(ret, FOLL_HWPOISON);
 
 		if (err)
 			return err;
virt/kvm/kvm_main.c +6 -0
index d255964ec331..09b6973e679d 100644
@@ -2688,6 +2688,12 @@ kvm_pfn_t hva_to_pfn(unsigned long addr, bool atomic, bool interruptible,
 		r = hva_to_pfn_remapped(vma, addr, write_fault, writable, &pfn);
 		if (r == -EAGAIN)
 			goto retry;
+
+		if (r == -EHWPOISON) {
+			pfn = KVM_PFN_ERR_HWPOISON;
+			goto exit;
+		}
+
 		if (r < 0)
 			pfn = KVM_PFN_ERR_FAULT;
 	} else {
Signature · 1 line
2.17.1
#19 lore raw

PATCH v3 5/6 mm: Change ghes code to allow poison of non-struct PFN

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

The GHES code allows calling of memory_failure() on the PFNs that pass the pfn_valid() check. This contract is broken for the remapped PFNs which fails the check and ghes_do_memory_failure() returns without triggering memory_failure().

Update code to allow memory_failure() call on PFNs failing pfn_valid().

---
 drivers/acpi/apei/ghes.c | 12 +-----------
 1 file changed, 1 insertion(+), 11 deletions(-)
drivers/acpi/apei/ghes.c +1 -11
index 34ad071a64e9..2ab7fec8127d 100644
@@ -459,20 +459,10 @@ static void ghes_kick_task_work(struct callback_head *head)
 
 static bool ghes_do_memory_failure(u64 physical_addr, int flags)
 {
-	unsigned long pfn;
-
 	if (!IS_ENABLED(CONFIG_ACPI_APEI_MEMORY_FAILURE))
 		return false;
 
-	pfn = PHYS_PFN(physical_addr);
-	if (!pfn_valid(pfn) && !arch_is_platform_page(physical_addr)) {
-		pr_warn_ratelimited(FW_WARN GHES_PFX
-		"Invalid address in generic error data: %#llx\n",
-		physical_addr);
-		return false;
-	}
-
-	memory_failure_queue(pfn, flags);
+	memory_failure_queue(PHYS_PFN(physical_addr), flags);
 	return true;
 }
 
Signature · 1 line
2.17.1
#20 lore raw

PATCH v3 6/6 vfio/nvgpu: register device memory for poison handling

In reply to [email protected]

From: Ankit Agrawal <[email protected]>

The nvgpu-vfio-pci module maps QEMU VMA to device memory through remap_pfn_range(). The new mechanism to handle poison on memory not backed by struct page is leveraged here.

nvgpu-vfio-pci defines a function pfn_memory_failure() to get the ECC PFN from the MM. The function is registered with kernel MM along with the address space and PFN range through register_pfn_address_space().

Track poisoned PFN in the nvgpu-vfio-pci module as bitmap with a bit per PFN. The PFN is communicated by the kernel MM to the module through the failure function, which sets the appropriate bit in the bitmap.

Register a VMA fault ops for the module. It returns VM_FAULT_HWPOISON in case the bit for the PFN is set in the bitmap.

Clear bitmap on reset to reflect the clean state of the device memory after reset.

---
 drivers/vfio/pci/nvgpu/main.c | 116 ++++++++++++++++++++++++++++++++--
 1 file changed, 110 insertions(+), 6 deletions(-)
drivers/vfio/pci/nvgpu/main.c +110 -6
index 2dd8cc6e0145..8ccd3fe33a0f 100644
@@ -5,6 +5,8 @@
 
 #include <linux/pci.h>
 #include <linux/vfio_pci_core.h>
+#include <linux/bitmap.h>
+#include <linux/memory-failure.h>
 
 #define DUMMY_PFN \
 	(((nvdev->mem_prop.hpa + nvdev->mem_prop.mem_length) >> PAGE_SHIFT) - 1)
@@ -12,12 +14,78 @@
 struct dev_mem_properties {
 	uint64_t hpa;
 	uint64_t mem_length;
+	unsigned long *pfn_bitmap;
 	int bar1_start_offset;
 };
 
 struct nvgpu_vfio_pci_core_device {
 	struct vfio_pci_core_device core_device;
 	struct dev_mem_properties mem_prop;
+	struct pfn_address_space pfn_address_space;
+};
+
+void nvgpu_vfio_pci_pfn_memory_failure(struct pfn_address_space *pfn_space,
+				       unsigned long pfn)
+{
+	struct nvgpu_vfio_pci_core_device *nvdev = container_of(
+		pfn_space, struct nvgpu_vfio_pci_core_device, pfn_address_space);
+
+	/*
+	 * MM has called to notify a poisoned page. Track that in the bitmap.
+	 */
+	__set_bit(pfn - (pfn_space->node.start), nvdev->mem_prop.pfn_bitmap);
+}
+
+struct pfn_address_space_ops nvgpu_vfio_pci_pas_ops = {
+	.failure = nvgpu_vfio_pci_pfn_memory_failure,
+};
+
+static int
+nvgpu_vfio_pci_register_pfn_range(struct nvgpu_vfio_pci_core_device *nvdev,
+				  struct vm_area_struct *vma)
+{
+	unsigned long nr_pages;
+	int ret = 0;
+
+	nr_pages = nvdev->mem_prop.mem_length >> PAGE_SHIFT;
+
+	nvdev->pfn_address_space.node.start = vma->vm_pgoff;
+	nvdev->pfn_address_space.node.last = vma->vm_pgoff + nr_pages - 1;
+	nvdev->pfn_address_space.ops = &nvgpu_vfio_pci_pas_ops;
+	nvdev->pfn_address_space.mapping = vma->vm_file->f_mapping;
+
+	ret = register_pfn_address_space(&(nvdev->pfn_address_space));
+
+	return ret;
+}
+
+static vm_fault_t nvgpu_vfio_pci_fault(struct vm_fault *vmf)
+{
+	unsigned long mem_offset = vmf->pgoff - vmf->vma->vm_pgoff;
+	struct nvgpu_vfio_pci_core_device *nvdev = container_of(
+		vmf->vma->vm_file->private_data,
+		struct nvgpu_vfio_pci_core_device, core_device.vdev);
+	int ret;
+
+	/*
+	 * Check if the page is poisoned.
+	 */
+	if (mem_offset < (nvdev->mem_prop.mem_length >> PAGE_SHIFT) &&
+		test_bit(mem_offset, nvdev->mem_prop.pfn_bitmap))
+		return VM_FAULT_HWPOISON;
+
+	ret = remap_pfn_range(vmf->vma,
+			vmf->vma->vm_start + (mem_offset << PAGE_SHIFT),
+			DUMMY_PFN, PAGE_SIZE,
+			vmf->vma->vm_page_prot);
+	if (ret)
+		return VM_FAULT_ERROR;
+
+	return VM_FAULT_NOPAGE;
+}
+
+static const struct vm_operations_struct nvgpu_vfio_pci_mmap_ops = {
+	.fault = nvgpu_vfio_pci_fault,
 };
 
 static int vfio_get_bar1_start_offset(struct vfio_pci_core_device *vdev)
@@ -26,8 +94,9 @@ static int vfio_get_bar1_start_offset(struct vfio_pci_core_device *vdev)
 
 	pci_read_config_byte(vdev->pdev, 0x10, &val);
 	/*
-	 * The BAR1 start offset in the PCI config space depends on the BAR0size.
-	 * Check if the BAR0 is 64b and return the approproiate BAR1 offset.
+	 * The BAR1 start offset in the PCI config space depends on the BAR0
+	 * size. Check if the BAR0 is 64b and return the approproiate BAR1
+	 * offset.
 	 */
 	if (val & PCI_BASE_ADDRESS_MEM_TYPE_64)
 		return VFIO_PCI_BAR2_REGION_INDEX;
@@ -54,6 +123,16 @@ static int nvgpu_vfio_pci_open_device(struct vfio_device *core_vdev)
 	return ret;
 }
 
+void nvgpu_vfio_pci_close_device(struct vfio_device *core_vdev)
+{
+	struct nvgpu_vfio_pci_core_device *nvdev = container_of(
+		core_vdev, struct nvgpu_vfio_pci_core_device, core_device.vdev);
+
+	unregister_pfn_address_space(&(nvdev->pfn_address_space));
+
+	vfio_pci_core_close_device(core_vdev);
+}
+
 int nvgpu_vfio_pci_mmap(struct vfio_device *core_vdev,
 			struct vm_area_struct *vma)
 {
@@ -93,8 +172,11 @@ int nvgpu_vfio_pci_mmap(struct vfio_device *core_vdev,
 		return ret;
 
 	vma->vm_pgoff = start_pfn + pgoff;
+	vma->vm_ops = &nvgpu_vfio_pci_mmap_ops;
 
-	return 0;
+	ret = nvgpu_vfio_pci_register_pfn_range(nvdev, vma);
+
+	return ret;
 }
 
 long nvgpu_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
@@ -140,7 +222,14 @@ long nvgpu_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
 		}
 
 		return vfio_pci_core_ioctl(core_vdev, cmd, arg);
-
+	case VFIO_DEVICE_RESET:
+		/*
+		 * Resetting the GPU clears up the poisoned page. Reset the
+		 * poisoned page bitmap.
+		 */
+		memset(nvdev->mem_prop.pfn_bitmap, 0,
+		       nvdev->mem_prop.mem_length >> (PAGE_SHIFT + 3));
+		return vfio_pci_core_ioctl(core_vdev, cmd, arg);
 	default:
 		return vfio_pci_core_ioctl(core_vdev, cmd, arg);
 	}
@@ -151,7 +240,7 @@ static const struct vfio_device_ops nvgpu_vfio_pci_ops = {
 	.init = vfio_pci_core_init_dev,
 	.release = vfio_pci_core_release_dev,
 	.open_device = nvgpu_vfio_pci_open_device,
-	.close_device = vfio_pci_core_close_device,
+	.close_device = nvgpu_vfio_pci_close_device,
 	.ioctl = nvgpu_vfio_pci_ioctl,
 	.read = vfio_pci_core_read,
 	.write = vfio_pci_core_write,
@@ -188,7 +277,20 @@ nvgpu_vfio_pci_fetch_memory_property(struct pci_dev *pdev,
 
 	ret = device_property_read_u64(&(pdev->dev), "nvidia,gpu-mem-size",
 				       &(nvdev->mem_prop.mem_length));
-	return ret;
+	if (ret)
+		return ret;
+
+	/*
+	 * A bitmap is maintained to teack the pages that are poisoned. Each
+	 * page is represented by a bit. Allocation size in bytes is
+	 * determined by shifting the device memory size by PAGE_SHIFT to
+	 * determine the number of pages; and further shifted by 3 as each
+	 * byte could track 8 pages.
+	 */
+	nvdev->mem_prop.pfn_bitmap
+		= vzalloc(nvdev->mem_prop.mem_length >> (PAGE_SHIFT + 3));
+
+	return 0;
 }
 
 static int nvgpu_vfio_pci_probe(struct pci_dev *pdev,
@@ -224,6 +326,8 @@ static void nvgpu_vfio_pci_remove(struct pci_dev *pdev)
 	struct nvgpu_vfio_pci_core_device *nvdev = nvgpu_drvdata(pdev);
 	struct vfio_pci_core_device *vdev = &nvdev->core_device;
 
+	vfree(nvdev->mem_prop.pfn_bitmap);
+
 	vfio_pci_core_unregister_device(vdev);
 	vfio_put_device(&vdev->vdev);
 }
Signature · 1 line
2.17.1
#21 lore raw

Re: [PATCH v3 6/6] vfio/nvgpu: register device memory for poison handling

Quoted 185 lines · On Wed, 5 Apr 2023 11:01:34 -0700 <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

The nvgpu-vfio-pci module maps QEMU VMA to device memory through remap_pfn_range(). The new mechanism to handle poison on memory not backed by struct page is leveraged here.

nvgpu-vfio-pci defines a function pfn_memory_failure() to get the ECC PFN from the MM. The function is registered with kernel MM along with the address space and PFN range through register_pfn_address_space().

Track poisoned PFN in the nvgpu-vfio-pci module as bitmap with a bit per PFN. The PFN is communicated by the kernel MM to the module through the failure function, which sets the appropriate bit in the bitmap.

Register a VMA fault ops for the module. It returns VM_FAULT_HWPOISON in case the bit for the PFN is set in the bitmap.

Clear bitmap on reset to reflect the clean state of the device memory after reset.

---
 drivers/vfio/pci/nvgpu/main.c | 116 ++++++++++++++++++++++++++++++++--
 1 file changed, 110 insertions(+), 6 deletions(-)
drivers/vfio/pci/nvgpu/main.c +94 -5
index 2dd8cc6e0145..8ccd3fe33a0f 100644
@@ -5,6 +5,8 @@
 
 #include <linux/pci.h>
 #include <linux/vfio_pci_core.h>
+#include <linux/bitmap.h>
+#include <linux/memory-failure.h>
 
 #define DUMMY_PFN \
 	(((nvdev->mem_prop.hpa + nvdev->mem_prop.mem_length) >> PAGE_SHIFT) - 1)
@@ -12,12 +14,78 @@
 struct dev_mem_properties {
 	uint64_t hpa;
 	uint64_t mem_length;
+	unsigned long *pfn_bitmap;
 	int bar1_start_offset;
 };
 
 struct nvgpu_vfio_pci_core_device {
 	struct vfio_pci_core_device core_device;
 	struct dev_mem_properties mem_prop;
+	struct pfn_address_space pfn_address_space;
+};
+
+void nvgpu_vfio_pci_pfn_memory_failure(struct pfn_address_space *pfn_space,
+				       unsigned long pfn)
+{
+	struct nvgpu_vfio_pci_core_device *nvdev = container_of(
+		pfn_space, struct nvgpu_vfio_pci_core_device, pfn_address_space);
+
+	/*
+	 * MM has called to notify a poisoned page. Track that in the bitmap.
+	 */
+	__set_bit(pfn - (pfn_space->node.start), nvdev->mem_prop.pfn_bitmap);
+}
+
+struct pfn_address_space_ops nvgpu_vfio_pci_pas_ops = {
+	.failure = nvgpu_vfio_pci_pfn_memory_failure,
+};
+
+static int
+nvgpu_vfio_pci_register_pfn_range(struct nvgpu_vfio_pci_core_device *nvdev,
+				  struct vm_area_struct *vma)
+{
+	unsigned long nr_pages;
+	int ret = 0;
+
+	nr_pages = nvdev->mem_prop.mem_length >> PAGE_SHIFT;
+
+	nvdev->pfn_address_space.node.start = vma->vm_pgoff;
+	nvdev->pfn_address_space.node.last = vma->vm_pgoff + nr_pages - 1;
+	nvdev->pfn_address_space.ops = &nvgpu_vfio_pci_pas_ops;
+	nvdev->pfn_address_space.mapping = vma->vm_file->f_mapping;
+
+	ret = register_pfn_address_space(&(nvdev->pfn_address_space));
+
+	return ret;
+}
+
+static vm_fault_t nvgpu_vfio_pci_fault(struct vm_fault *vmf)
+{
+	unsigned long mem_offset = vmf->pgoff - vmf->vma->vm_pgoff;
+	struct nvgpu_vfio_pci_core_device *nvdev = container_of(
+		vmf->vma->vm_file->private_data,
+		struct nvgpu_vfio_pci_core_device, core_device.vdev);
+	int ret;
+
+	/*
+	 * Check if the page is poisoned.
+	 */
+	if (mem_offset < (nvdev->mem_prop.mem_length >> PAGE_SHIFT) &&
+		test_bit(mem_offset, nvdev->mem_prop.pfn_bitmap))
+		return VM_FAULT_HWPOISON;
+
+	ret = remap_pfn_range(vmf->vma,
+			vmf->vma->vm_start + (mem_offset << PAGE_SHIFT),
+			DUMMY_PFN, PAGE_SIZE,
+			vmf->vma->vm_page_prot);
+	if (ret)
+		return VM_FAULT_ERROR;
+
+	return VM_FAULT_NOPAGE;
+}
+
+static const struct vm_operations_struct nvgpu_vfio_pci_mmap_ops = {
+	.fault = nvgpu_vfio_pci_fault,
 };
 
 static int vfio_get_bar1_start_offset(struct vfio_pci_core_device *vdev)
@@ -26,8 +94,9 @@ static int vfio_get_bar1_start_offset(struct vfio_pci_core_device *vdev)
 
 	pci_read_config_byte(vdev->pdev, 0x10, &val);
 	/*
-	 * The BAR1 start offset in the PCI config space depends on the BAR0size.
-	 * Check if the BAR0 is 64b and return the approproiate BAR1 offset.
+	 * The BAR1 start offset in the PCI config space depends on the BAR0
+	 * size. Check if the BAR0 is 64b and return the approproiate BAR1
+	 * offset.
 	 */
 	if (val & PCI_BASE_ADDRESS_MEM_TYPE_64)
 		return VFIO_PCI_BAR2_REGION_INDEX;
@@ -54,6 +123,16 @@ static int nvgpu_vfio_pci_open_device(struct vfio_device *core_vdev)
 	return ret;
 }
 
+void nvgpu_vfio_pci_close_device(struct vfio_device *core_vdev)
+{
+	struct nvgpu_vfio_pci_core_device *nvdev = container_of(
+		core_vdev, struct nvgpu_vfio_pci_core_device, core_device.vdev);
+
+	unregister_pfn_address_space(&(nvdev->pfn_address_space));
+
+	vfio_pci_core_close_device(core_vdev);
+}
+
 int nvgpu_vfio_pci_mmap(struct vfio_device *core_vdev,
 			struct vm_area_struct *vma)
 {
@@ -93,8 +172,11 @@ int nvgpu_vfio_pci_mmap(struct vfio_device *core_vdev,
 		return ret;
 
 	vma->vm_pgoff = start_pfn + pgoff;
+	vma->vm_ops = &nvgpu_vfio_pci_mmap_ops;
 
-	return 0;
+	ret = nvgpu_vfio_pci_register_pfn_range(nvdev, vma);
+
+	return ret;
 }
 
 long nvgpu_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
@@ -140,7 +222,14 @@ long nvgpu_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
 		}
 
 		return vfio_pci_core_ioctl(core_vdev, cmd, arg);
-
+	case VFIO_DEVICE_RESET:
+		/*
+		 * Resetting the GPU clears up the poisoned page. Reset the
+		 * poisoned page bitmap.
+		 */
+		memset(nvdev->mem_prop.pfn_bitmap, 0,
+		       nvdev->mem_prop.mem_length >> (PAGE_SHIFT + 3));
+		return vfio_pci_core_ioctl(core_vdev, cmd, arg);
 	default:
 		return vfio_pci_core_ioctl(core_vdev, cmd, arg);
 	}
@@ -151,7 +240,7 @@ static const struct vfio_device_ops nvgpu_vfio_pci_ops = {
 	.init = vfio_pci_core_init_dev,
 	.release = vfio_pci_core_release_dev,
 	.open_device = nvgpu_vfio_pci_open_device,
-	.close_device = vfio_pci_core_close_device,
+	.close_device = nvgpu_vfio_pci_close_device,
 	.ioctl = nvgpu_vfio_pci_ioctl,
 	.read = vfio_pci_core_read,
 	.write = vfio_pci_core_write,

drivers/vfio/pci/nvgpu/main.c nvgpu_vfio_pci_fetch_memory_property(struct pci_dev *pdev, · line 284

Quoted 9 lines
drivers/vfio/pci/nvgpu/main.c +5 -1
@@ -188,7 +277,20 @@ nvgpu_vfio_pci_fetch_memory_property(struct pci_dev *pdev,
 
 	ret = device_property_read_u64(&(pdev->dev), "nvidia,gpu-mem-size",
 				       &(nvdev->mem_prop.mem_length));
-	return ret;
+	if (ret)
+		return ret;
+
+	/*
+	 * A bitmap is maintained to teack the pages that are poisoned. Each
                                       ^track?
Quoted 21 lines
drivers/vfio/pci/nvgpu/main.c +11 -0
+	 * page is represented by a bit. Allocation size in bytes is
+	 * determined by shifting the device memory size by PAGE_SHIFT to
+	 * determine the number of pages; and further shifted by 3 as each
+	 * byte could track 8 pages.
+	 */
+	nvdev->mem_prop.pfn_bitmap
+		= vzalloc(nvdev->mem_prop.mem_length >> (PAGE_SHIFT + 3));
+
+	return 0;
 }
 
 static int nvgpu_vfio_pci_probe(struct pci_dev *pdev,
@@ -224,6 +326,8 @@ static void nvgpu_vfio_pci_remove(struct pci_dev *pdev)
 	struct nvgpu_vfio_pci_core_device *nvdev = nvgpu_drvdata(pdev);
 	struct vfio_pci_core_device *vdev = &nvdev->core_device;
 
+	vfree(nvdev->mem_prop.pfn_bitmap);
+
 	vfio_pci_core_unregister_device(vdev);
 	vfio_put_device(&vdev->vdev);
 }
#22 lore raw

Re: [PATCH v3 6/6] vfio/nvgpu: register device memory for poison handling

In reply to [email protected]
kernel test robot noticed the following build warnings:

Hi,

kernel test robot noticed the following build warnings:

[auto build test WARNING on awilliam-vfio/for-linus]
[also build test WARNING on kvmarm/next akpm-mm/mm-everything linus/master v6.3-rc5] [cannot apply to awilliam-vfio/next next-20230405]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/ankita-nvidia-com/kvm-determine-memory-type-from-VMA/20230406-020404
base:   https://github.com/awilliam/linux-vfio.git for-linus
patch link:    https://lore.kernel.org/r/20230405180134.16932-7-ankita%40nvidia.com
patch subject: [PATCH v3 6/6] vfio/nvgpu: register device memory for poison handling
config: sparc-allyesconfig (https://download.01.org/0day-ci/archive/20230406/[email protected]/config)
compiler: sparc64-linux-gcc (GCC) 12.1.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/f2eb3c3417670adc1615fc629e6363d50c0623b4
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review ankita-nvidia-com/kvm-determine-memory-type-from-VMA/20230406-020404 git checkout f2eb3c3417670adc1615fc629e6363d50c0623b4
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=sparc olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=sparc SHELL=/bin/bash drivers/vfio/

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> drivers/vfio/pci/nvgpu/main.c:27:6: warning: no previous prototype for 'nvgpu_vfio_pci_pfn_memory_failure' [-Wmissing-prototypes]
      27 | void nvgpu_vfio_pci_pfn_memory_failure(struct pfn_address_space *pfn_space,
         |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> drivers/vfio/pci/nvgpu/main.c:126:6: warning: no previous prototype for 'nvgpu_vfio_pci_close_device' [-Wmissing-prototypes]

126 | void nvgpu_vfio_pci_close_device(struct vfio_device *core_vdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/vfio/pci/nvgpu/main.c:136:5: warning: no previous prototype for 'nvgpu_vfio_pci_mmap' [-Wmissing-prototypes]
136 | int nvgpu_vfio_pci_mmap(struct vfio_device *core_vdev,
| ^~~~~~~~~~~~~~~~~~~
drivers/vfio/pci/nvgpu/main.c:182:6: warning: no previous prototype for 'nvgpu_vfio_pci_ioctl' [-Wmissing-prototypes]
182 | long nvgpu_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
| ^~~~~~~~~~~~~~~~~~~~

vim +/nvgpu_vfio_pci_pfn_memory_failure +27 drivers/vfio/pci/nvgpu/main.c

    26	
  > 27	void nvgpu_vfio_pci_pfn_memory_failure(struct pfn_address_space *pfn_space,
    28					       unsigned long pfn)
    29	{
    30		struct nvgpu_vfio_pci_core_device *nvdev = container_of(
    31			pfn_space, struct nvgpu_vfio_pci_core_device, pfn_address_space);
    32	
    33		/*
    34		 * MM has called to notify a poisoned page. Track that in the bitmap.
    35		 */
    36		__set_bit(pfn - (pfn_space->node.start), nvdev->mem_prop.pfn_bitmap);
    37	}
    38	
    39	struct pfn_address_space_ops nvgpu_vfio_pci_pas_ops = {
    40		.failure = nvgpu_vfio_pci_pfn_memory_failure,
    41	};
    42	
    43	static int
    44	nvgpu_vfio_pci_register_pfn_range(struct nvgpu_vfio_pci_core_device *nvdev,
    45					  struct vm_area_struct *vma)
    46	{
    47		unsigned long nr_pages;
    48		int ret = 0;
    49	
    50		nr_pages = nvdev->mem_prop.mem_length >> PAGE_SHIFT;
    51	
    52		nvdev->pfn_address_space.node.start = vma->vm_pgoff;
    53		nvdev->pfn_address_space.node.last = vma->vm_pgoff + nr_pages - 1;
    54		nvdev->pfn_address_space.ops = &nvgpu_vfio_pci_pas_ops;
    55		nvdev->pfn_address_space.mapping = vma->vm_file->f_mapping;
    56	
    57		ret = register_pfn_address_space(&(nvdev->pfn_address_space));
    58	
    59		return ret;
    60	}
    61	
    62	static vm_fault_t nvgpu_vfio_pci_fault(struct vm_fault *vmf)
    63	{
    64		unsigned long mem_offset = vmf->pgoff - vmf->vma->vm_pgoff;
    65		struct nvgpu_vfio_pci_core_device *nvdev = container_of(
    66			vmf->vma->vm_file->private_data,
    67			struct nvgpu_vfio_pci_core_device, core_device.vdev);
    68		int ret;
    69	
    70		/*
    71		 * Check if the page is poisoned.
    72		 */
    73		if (mem_offset < (nvdev->mem_prop.mem_length >> PAGE_SHIFT) &&
    74			test_bit(mem_offset, nvdev->mem_prop.pfn_bitmap))
    75			return VM_FAULT_HWPOISON;
    76	
    77		ret = remap_pfn_range(vmf->vma,
    78				vmf->vma->vm_start + (mem_offset << PAGE_SHIFT),
    79				DUMMY_PFN, PAGE_SIZE,
    80				vmf->vma->vm_page_prot);
    81		if (ret)
    82			return VM_FAULT_ERROR;
    83	
    84		return VM_FAULT_NOPAGE;
    85	}
    86	
    87	static const struct vm_operations_struct nvgpu_vfio_pci_mmap_ops = {
    88		.fault = nvgpu_vfio_pci_fault,
    89	};
    90	
    91	static int vfio_get_bar1_start_offset(struct vfio_pci_core_device *vdev)
    92	{
    93		u8 val = 0;
    94	
    95		pci_read_config_byte(vdev->pdev, 0x10, &val);
    96		/*
    97		 * The BAR1 start offset in the PCI config space depends on the BAR0
    98		 * size. Check if the BAR0 is 64b and return the approproiate BAR1
    99		 * offset.
   100		 */
   101		if (val & PCI_BASE_ADDRESS_MEM_TYPE_64)
   102			return VFIO_PCI_BAR2_REGION_INDEX;
   103	
   104		return VFIO_PCI_BAR1_REGION_INDEX;
   105	}
   106	
   107	static int nvgpu_vfio_pci_open_device(struct vfio_device *core_vdev)
   108	{
   109		struct nvgpu_vfio_pci_core_device *nvdev = container_of(
   110			core_vdev, struct nvgpu_vfio_pci_core_device, core_device.vdev);
   111		struct vfio_pci_core_device *vdev =
   112			container_of(core_vdev, struct vfio_pci_core_device, vdev);
   113		int ret;
   114	
   115		ret = vfio_pci_core_enable(vdev);
   116		if (ret)
   117			return ret;
   118	
   119		vfio_pci_core_finish_enable(vdev);
   120	
   121		nvdev->mem_prop.bar1_start_offset = vfio_get_bar1_start_offset(vdev);
   122	
   123		return ret;
   124	}
   125	
 > 126	void nvgpu_vfio_pci_close_device(struct vfio_device *core_vdev)

127 {
128 struct nvgpu_vfio_pci_core_device *nvdev = container_of(
129 core_vdev, struct nvgpu_vfio_pci_core_device, core_device.vdev);
130
131 unregister_pfn_address_space(&(nvdev->pfn_address_space));
132
133 vfio_pci_core_close_device(core_vdev);
134 }
135

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
#23 lore raw

Re: [PATCH v3 6/6] vfio/nvgpu: register device memory for poison handling

In reply to [email protected]
Quoted 7 lines · On Wed, Apr 05, 2023 at 11:01:34AM -0700, [email protected] wrote:
drivers/vfio/pci/nvgpu/main.c +2 -1
@@ -188,7 +277,20 @@ nvgpu_vfio_pci_fetch_memory_property(struct pci_dev *pdev,
 
 	ret = device_property_read_u64(&(pdev->dev), "nvidia,gpu-mem-size",
 				       &(nvdev->mem_prop.mem_length));
-	return ret;
+	if (ret)
+		return ret;

drivers/vfio/pci/nvgpu/main.c nvgpu_vfio_pci_fetch_memory_property(struct pci_dev *pdev, · line 291

Quoted 10 lines
drivers/vfio/pci/nvgpu/main.c +10 -0
+
+	/*
+	 * A bitmap is maintained to teack the pages that are poisoned. Each
+	 * page is represented by a bit. Allocation size in bytes is
+	 * determined by shifting the device memory size by PAGE_SHIFT to
+	 * determine the number of pages; and further shifted by 3 as each
+	 * byte could track 8 pages.
+	 */
+	nvdev->mem_prop.pfn_bitmap
+		= vzalloc(nvdev->mem_prop.mem_length >> (PAGE_SHIFT + 3));

This allocation needs a NULL check.

regards,
dan carpenter

Quoted 3 lines
drivers/vfio/pci/nvgpu/main.c +2 -0
+
+	return 0;
 }
#24 lore raw
In reply to [email protected]

[...]

Quoted 8 lines

This goes along with a qemu series to provides the necessary implementation of the Grace Hopper Superchip firmware specification so that the guest operating system can see the correct ACPI modeling for the coherent GPU device.
https://github.com/qemu/qemu/compare/master...ankita-nv:qemu:dev-ankit/cohmem-0330

Applied and tested over v6.3-rc4.

I briefly skimmed over the series, the patch subject prefixes are a bit misleading IMHO and could be improved:

Quoted 2 lines

Ankit Agrawal (6):
kvm: determine memory type from VMA

this is arch64 specific kvm (kvm/aarch64: ?)

Quoted 2 lines

vfio/nvgpu: expose GPU device memory as BAR1
mm: handle poisoning of pfn without struct pages

mm/memory-failure:

Quoted 1 line

mm: Add poison error check in fixup_user_fault() for mapped PFN

That's both MM and core-KVM, maybe worth splitting up.

Quoted 1 line

mm: Change ghes code to allow poison of non-struct PFN

That's drivers/acpi/apei code, not core-mm code.

Quoted 1 line

vfio/nvgpu: register device memory for poison handling

Signature · 3 lines
Thanks,

David / dhildenb
#25 lore raw

Thanks David, response inline.

[...]

Quoted 2 lines

I briefly skimmed over the series, the patch subject prefixes are a bit misleading IMHO and could be improved:

Understood. Will fix that in the next iteration.

Quoted 4 lines
Quoted 2 lines

Ankit Agrawal (6):
kvm: determine memory type from VMA

this is arch64 specific kvm (kvm/aarch64: ?)

Right. I'll change the prefix to kvm/aarch64

Quoted 3 lines
Quoted 2 lines
Quoted 1 line

vfio/nvgpu: expose GPU device memory as BAR1

mm: handle poisoning of pfn without struct pages

mm/memory-failure:

Will change the prefix.

Quoted 2 lines
Quoted 1 line

mm: Add poison error check in fixup_user_fault() for mapped PFN

That's both MM and core-KVM, maybe worth splitting up.

Ack, will do.

Quoted 2 lines
Quoted 1 line

mm: Change ghes code to allow poison of non-struct PFN

That's drivers/acpi/apei code, not core-mm code.

Ack.

Quoted 1 line
Quoted 1 line

vfio/nvgpu: register device memory for poison handling

Thanks,
Ankit Agrawal
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#26 lore raw
Quoted 8 lines · Ankit Agrawal

Thanks David, response inline.

[...]

Quoted 2 lines

I briefly skimmed over the series, the patch subject prefixes are a bit misleading IMHO and could be improved:

Quoted 8 lines

Understood. Will fix that in the next iteration.

Quoted 4 lines
Quoted 2 lines

Ankit Agrawal (6):
kvm: determine memory type from VMA

this is arch64 specific kvm (kvm/aarch64: ?)

Right. I'll change the prefix to kvm/aarch64

Please look at the git log: the idiomatic prefix is "KVM: arm64: Something starting with a capital letter"

AArch64 is almost never used anywhere in the arm64 tree.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
maintainer
#27 lore raw
In reply to [email protected]
Quoted 19 lines · On Wed, 05 Apr 2023 19:01:28 +0100, <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device for the on-chip GPU that is the logical OS representation of the internal propritary cache coherent interconnect.

This representation has a number of limitations compared to a real PCI device, in particular, it does not model the coherent GPU memory aperture as a PCI config space BAR, and PCI doesn't know anything about cacheable memory types.

Provide a VFIO PCI variant driver that adapts the unique PCI representation into a more standard PCI representation facing userspace. The GPU memory aperture is obtained from ACPI, according to the FW specification, and exported to userspace as the VFIO_REGION that covers the first PCI BAR. qemu will naturally generate a PCI device in the VM where the cacheable aperture is reported in BAR1.

Quoted 8 lines

Since this memory region is actually cache coherent with the CPU, the VFIO variant driver will mmap it into VMA using a cacheable mapping.

As this is the first time an ARM environment has placed cacheable non-struct page backed memory (eg from remap_pfn_range) into a KVM page table, fix a bug in ARM KVM where it does not copy the cacheable memory attributes from non-struct page backed PTEs to ensure the guest also gets a cacheable mapping.

This is not a bug, but a conscious design decision. As you pointed out above, nothing needed this until now, and a device mapping is the only safe thing to do as we know exactly *nothing* about the memory that gets mapped.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#28 lore raw
Quoted 30 lines · Marc Zyngier
Quoted 19 lines · On Wed, 05 Apr 2023 19:01:28 +0100, <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device for the on-chip GPU that is the logical OS representation of the internal propritary cache coherent interconnect.

This representation has a number of limitations compared to a real PCI device, in particular, it does not model the coherent GPU memory aperture as a PCI config space BAR, and PCI doesn't know anything about cacheable memory types.

Provide a VFIO PCI variant driver that adapts the unique PCI representation into a more standard PCI representation facing userspace. The GPU memory aperture is obtained from ACPI, according to the FW specification, and exported to userspace as the VFIO_REGION that covers the first PCI BAR. qemu will naturally generate a PCI device in the VM where the cacheable aperture is reported in BAR1.

Quoted 8 lines

Since this memory region is actually cache coherent with the CPU, the VFIO variant driver will mmap it into VMA using a cacheable mapping.

As this is the first time an ARM environment has placed cacheable non-struct page backed memory (eg from remap_pfn_range) into a KVM page table, fix a bug in ARM KVM where it does not copy the cacheable memory attributes from non-struct page backed PTEs to ensure the guest also gets a cacheable mapping.

Quoted 4 lines

This is not a bug, but a conscious design decision. As you pointed out above, nothing needed this until now, and a device mapping is the only safe thing to do as we know exactly *nothing* about the memory that gets mapped.

IMHO, from the mm perspective, the bug is using pfn_is_map_memory() to determine the cachability or device memory status of a PFN in a VMA. That is not what that API is for.

The cachability should be determined by the pgprot bits in the VMA.

VM_IO is the flag that says the VMA maps memory with side-effects.

I understand in ARM KVM it is not allowed for the VM and host to have different cachability, so mis-detecting host cachable memory and making it forced non-cachable in the VM is not a safe thing to do?

Jason

maintainer
#29 lore raw
Quoted 32 lines · Jason Gunthorpe
Quoted 30 lines · Marc Zyngier
Quoted 27 lines · On Wed, 05 Apr 2023 19:01:28 +0100, <[email protected]> wrote:

From: Ankit Agrawal <[email protected]>

NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device for the on-chip GPU that is the logical OS representation of the internal propritary cache coherent interconnect.

This representation has a number of limitations compared to a real PCI device, in particular, it does not model the coherent GPU memory aperture as a PCI config space BAR, and PCI doesn't know anything about cacheable memory types.

Provide a VFIO PCI variant driver that adapts the unique PCI representation into a more standard PCI representation facing userspace. The GPU memory aperture is obtained from ACPI, according to the FW specification, and exported to userspace as the VFIO_REGION that covers the first PCI BAR. qemu will naturally generate a PCI device in the VM where the cacheable aperture is reported in BAR1.

Since this memory region is actually cache coherent with the CPU, the VFIO variant driver will mmap it into VMA using a cacheable mapping.

As this is the first time an ARM environment has placed cacheable non-struct page backed memory (eg from remap_pfn_range) into a KVM page table, fix a bug in ARM KVM where it does not copy the cacheable memory attributes from non-struct page backed PTEs to ensure the guest also gets a cacheable mapping.

Quoted 8 lines
Quoted 4 lines

This is not a bug, but a conscious design decision. As you pointed out above, nothing needed this until now, and a device mapping is the only safe thing to do as we know exactly *nothing* about the memory that gets mapped.

IMHO, from the mm perspective, the bug is using pfn_is_map_memory() to determine the cachability or device memory status of a PFN in a VMA. That is not what that API is for.

It is the right API for what KVM/arm64 has been designed for. RAM gets a normal memory mapping, and everything else gets device. That may not suit your *new* use case, but that doesn't make it broken.

Quoted 8 lines

The cachability should be determined by the pgprot bits in the VMA.

VM_IO is the flag that says the VMA maps memory with side-effects.

I understand in ARM KVM it is not allowed for the VM and host to have different cachability, so mis-detecting host cachable memory and making it forced non-cachable in the VM is not a safe thing to do?

Only if you insist on not losing coherency between the two aliases used at the same time (something that would seem pretty improbable). And said coherency can be restored by using CMOs, as documented in B2.8.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#30 lore raw
Quoted 6 lines · Marc Zyngier
Quoted 3 lines

IMHO, from the mm perspective, the bug is using pfn_is_map_memory() to determine the cachability or device memory status of a PFN in a VMA. That is not what that API is for.

It is the right API for what KVM/arm64 has been designed for. RAM gets a normal memory mapping, and everything else gets device.

The MM has a pretty flexible definition of "RAM" these days. For instance, I don't think pfn_is_map_memory() works correctly for all the cases we can do now with devm_memremap_pages().

Quoted 2 lines

That may not suit your *new* use case, but that doesn't make it broken.

I've now spent alot of time working on improving VFIO and the related ecosystem. I would to get to a point where we have a consistent VFIO experience on all the platforms.

Currently, real NIC and GPU HW with wide VFIO deployments on x86 do not work fully correctly on KVM/arm64. write-combining in the VM is the big problem for existing HW, and this new CXL-like stuff has problems with cachability.

I don't really care what we call it, as long as we can agree that VFIO devices not working fully in VMs is a problem that should be fixed.

Quoted 2 lines

Only if you insist on not losing coherency between the two aliases used at the same time (something that would seem pretty improbable).

This is VFIO so there is DMA involved. My understanding has been that the SMMU is allowed to pull data out of the cache. So if the hypervisor cachable side has pulled a line into cache and the VM uncached side dirtied the physical memory, it is allowed that SMMU will read stale cache data? Thus the VM will experience data corruption on its DMAs.

With VFIO live migration I expect the hypervisor qemu side to be actively reading from the cachable memory while the VM is running to migrate it, so it does not seem improbable.

Jason