Skip to contentSkip to conversation

KVM/ARM: Some vgic fixes and init sequence KVM selftests

PATCH Superseded Eric Auger 2 participants 16 messages Apr 1, 2021 – Apr 2, 2021

Subsystems (5)
Documentation KVM Kselftest arm64 +1 more subsystems

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

v4 cover KVM/ARM: Some vgic fixes and init sequence KVM selftests

While writting vgic v3 init sequence KVM selftests I noticed some relatively minor issues. This was also the opportunity to try to fix the issue laterly reported by Zenghui, related to the RDIST_TYPER last bit emulation. The final patch is a first batch of VGIC init sequence selftests. Of course they can be augmented with a lot more register access tests, but let's try to move forward incrementally ...

Best Regards

Eric

This series can be found at:
https://github.com/eauger/linux/tree/vgic_kvmselftests_v4

History:
v3 -> v4:
- take into account Drew's comment on the kvm selftests. No change to the KVM related patches compared to v3
v2 ->v3:
- reworked last bit read accessor to handle contiguous redist regions and rdist not registered in ascending order
- removed [PATCH 5/9] KVM: arm: move has_run_once after the map_resources
v1 -> v2:
- Took into account all comments from Marc and Alexandru's except the has_run_once still after the map_resources (this would oblige me to revisit in depth the selftests)

Eric Auger (8):
KVM: arm64: vgic-v3: Fix some error codes when setting RDIST base
KVM: arm64: Fix KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION read
KVM: arm64: vgic-v3: Fix error handling in vgic_v3_set_redist_base()
KVM: arm/arm64: vgic: Reset base address on kvm_vgic_dist_destroy()
docs: kvm: devices/arm-vgic-v3: enhance KVM_DEV_ARM_VGIC_CTRL_INIT doc
KVM: arm64: Simplify argument passing to vgic_uaccess_[read|write]
KVM: arm64: vgic-v3: Expose GICR_TYPER.Last for userspace
KVM: selftests: aarch64/vgic-v3 init sequence tests

 .../virt/kvm/devices/arm-vgic-v3.rst          |   2 +-
 arch/arm64/kvm/vgic/vgic-init.c               |  13 +-
 arch/arm64/kvm/vgic/vgic-kvm-device.c         |   3 +
 arch/arm64/kvm/vgic/vgic-mmio-v3.c            | 116 +++-
 arch/arm64/kvm/vgic/vgic-mmio.c               |  10 +-
 arch/arm64/kvm/vgic/vgic.h                    |   1 +
 include/kvm/arm_vgic.h                        |   3 +
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../testing/selftests/kvm/aarch64/vgic_init.c | 652 ++++++++++++++++++
 .../testing/selftests/kvm/include/kvm_util.h  |   9 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |  77 +++
 12 files changed, 838 insertions(+), 50 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/aarch64/vgic_init.c
Signature · 6 lines
2.26.3

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#1 lore raw

PATCH v4 1/8 KVM: arm64: vgic-v3: Fix some error codes when setting RDIST base

KVM_DEV_ARM_VGIC_GRP_ADDR group doc says we should return -EEXIST in case the base address of the redist is already set. We currently return -EINVAL.

However we need to return -EINVAL in case a legacy REDIST address is attempted to be set while REDIST_REGIONS were set. This case is discriminated by looking at the count field.

---

v1 -> v2:
- simplify the check sequence

---
 arch/arm64/kvm/vgic/vgic-mmio-v3.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
arch/arm64/kvm/vgic/vgic-mmio-v3.c +7 -8
index 15a6c98ee92f0..013b737b658f8 100644
@@ -791,10 +791,6 @@ static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index,
 	size_t size = count * KVM_VGIC_V3_REDIST_SIZE;
 	int ret;
 
-	/* single rdist region already set ?*/
-	if (!count && !list_empty(rd_regions))
-		return -EINVAL;
-
 	/* cross the end of memory ? */
 	if (base + size < base)
 		return -EINVAL;
@@ -805,11 +801,14 @@ static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index,
 	} else {
 		rdreg = list_last_entry(rd_regions,
 					struct vgic_redist_region, list);
-		if (index != rdreg->index + 1)
-			return -EINVAL;
 
-		/* Cannot add an explicitly sized regions after legacy region */
-		if (!rdreg->count)
+		if ((!count) != (!rdreg->count))
+			return -EINVAL; /* Mix REDIST and REDIST_REGION */
+
+		if (!count)
+			return -EEXIST;
+
+		if (index != rdreg->index + 1)
 			return -EINVAL;
 	}
 
Signature · 6 lines
2.26.3

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
maintainer
#2 lore raw

Re: [PATCH v4 1/8] KVM: arm64: vgic-v3: Fix some error codes when setting RDIST base

Hi Eric,

Quoted 35 lines · Eric Auger

KVM_DEV_ARM_VGIC_GRP_ADDR group doc says we should return -EEXIST in case the base address of the redist is already set. We currently return -EINVAL.

However we need to return -EINVAL in case a legacy REDIST address is attempted to be set while REDIST_REGIONS were set. This case is discriminated by looking at the count field.

---

v1 -> v2:
- simplify the check sequence

---
 arch/arm64/kvm/vgic/vgic-mmio-v3.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
arch/arm64/kvm/vgic/vgic-mmio-v3.c +0 -4
index 15a6c98ee92f0..013b737b658f8 100644
@@ -791,10 +791,6 @@ static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index,
 	size_t size = count * KVM_VGIC_V3_REDIST_SIZE;
 	int ret;
 
-	/* single rdist region already set ?*/
-	if (!count && !list_empty(rd_regions))
-		return -EINVAL;
-
 	/* cross the end of memory ? */
 	if (base + size < base)
 		return -EINVAL;

arch/arm64/kvm/vgic/vgic-mmio-v3.c static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index, · line 806

Quoted 10 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +2 -4
@@ -805,11 +801,14 @@ static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index,
 	} else {
 		rdreg = list_last_entry(rd_regions,
 					struct vgic_redist_region, list);
-		if (index != rdreg->index + 1)
-			return -EINVAL;
 
-		/* Cannot add an explicitly sized regions after legacy region */
-		if (!rdreg->count)
+		if ((!count) != (!rdreg->count))
+			return -EINVAL; /* Mix REDIST and REDIST_REGION */

Urgh... The triple negation killed me. Can we come up with a more intuitive expression? Something like:

		/* Don't mix single region and discrete redist regions */
		if (!count && rdreg->count)
			return -EINVAL;

Does it capture what you want to express?

Thanks,

	M.
Signature · 5 lines
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#3 lore raw

Re: [PATCH v4 1/8] KVM: arm64: vgic-v3: Fix some error codes when setting RDIST base

Hi Marc,

Quoted 45 lines · Marc Zyngier

Hi Eric,

Quoted 41 lines · Eric Auger

KVM_DEV_ARM_VGIC_GRP_ADDR group doc says we should return -EEXIST in case the base address of the redist is already set. We currently return -EINVAL.

However we need to return -EINVAL in case a legacy REDIST address is attempted to be set while REDIST_REGIONS were set. This case is discriminated by looking at the count field.

---

v1 -> v2:
- simplify the check sequence

---
 arch/arm64/kvm/vgic/vgic-mmio-v3.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
arch/arm64/kvm/vgic/vgic-mmio-v3.c +0 -6
index 15a6c98ee92f0..013b737b658f8 100644
@@ -791,10 +791,6 @@ static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index,
 	size_t size = count * KVM_VGIC_V3_REDIST_SIZE;
 	int ret;
 
-	/* single rdist region already set ?*/
-	if (!count && !list_empty(rd_regions))
-		return -EINVAL;
-
 	/* cross the end of memory ? */
 	if (base + size < base)
 		return -EINVAL;
@@ -805,11 +801,14 @@ static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index,
 	} else {
 		rdreg = list_last_entry(rd_regions,
 					struct vgic_redist_region, list);
-		if (index != rdreg->index + 1)
-			return -EINVAL;
 
Quoted 7 lines

arch/arm64/kvm/vgic/vgic-mmio-v3.c static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index, · line 806

Quoted 4 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +2 -2
-		/* Cannot add an explicitly sized regions after legacy region */
-		if (!rdreg->count)
+		if ((!count) != (!rdreg->count))
+			return -EINVAL; /* Mix REDIST and REDIST_REGION */

Urgh... The triple negation killed me. Can we come up with a more intuitive expression? Something like:

Yes sometimes I can be "different" ;-)

Quoted 5 lines
		/* Don't mix single region and discrete redist regions */
		if (!count && rdreg->count)
			return -EINVAL;>

Does it capture what you want to express?

yes it does!

Thanks

Eric

Quoted 5 lines

Thanks,

	M.

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author
#4 lore raw

PATCH v4 2/8 KVM: arm64: Fix KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION read

In reply to Eric Auger

The doc says:
"The characteristics of a specific redistributor region can
be read by presetting the index field in the attr data. Only valid for KVM_DEV_TYPE_ARM_VGIC_V3"

Unfortunately the existing code fails to read the input attr data.

Fixes 04c110932225 ("KVM: arm/arm64: Implement KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION")

---

v1 -> v2:
- in the commit message, remove the statement that the index always is 0
- add Alexandru's R-b

---
 arch/arm64/kvm/vgic/vgic-kvm-device.c | 3 +++
 1 file changed, 3 insertions(+)
arch/arm64/kvm/vgic/vgic-kvm-device.c +3 -0
index 44419679f91ad..2f66cf2472825 100644
@@ -226,6 +226,9 @@ static int vgic_get_common_attr(struct kvm_device *dev,
 		u64 addr;
 		unsigned long type = (unsigned long)attr->attr;
 
+		if (copy_from_user(&addr, uaddr, sizeof(addr)))
+			return -EFAULT;
+
 		r = kvm_vgic_addr(dev->kvm, type, &addr, false);
 		if (r)
 			return (r == -ENODEV) ? -ENXIO : r;
Signature · 6 lines
2.26.3

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#5 lore raw

PATCH v4 3/8 KVM: arm64: vgic-v3: Fix error handling in vgic_v3_set_redist_base()

In reply to Eric Auger

vgic_v3_insert_redist_region() may succeed while vgic_register_all_redist_iodevs fails. For example this happens while adding a redistributor region overlapping a dist region. The failure only is detected on vgic_register_all_redist_iodevs when vgic_v3_check_base() gets called in vgic_register_redist_iodev().

In such a case, remove the newly added redistributor region and free it.

---

v1 -> v2:
- fix the commit message and split declaration/assignment of rdreg

---
 arch/arm64/kvm/vgic/vgic-mmio-v3.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic-mmio-v3.c +7 -1
index 013b737b658f8..987e366c80008 100644
@@ -860,8 +860,14 @@ int vgic_v3_set_redist_base(struct kvm *kvm, u32 index, u64 addr, u32 count)
 	 * afterwards will register the iodevs when needed.
 	 */
 	ret = vgic_register_all_redist_iodevs(kvm);
-	if (ret)
+	if (ret) {
+		struct vgic_redist_region *rdreg;
+
+		rdreg = vgic_v3_rdist_region_from_index(kvm, index);
+		list_del(&rdreg->list);
+		kfree(rdreg);
 		return ret;
+	}
 
 	return 0;
 }
Signature · 6 lines
2.26.3

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#6 lore raw

PATCH v4 4/8 KVM: arm/arm64: vgic: Reset base address on kvm_vgic_dist_destroy()

In reply to Eric Auger

On vgic_dist_destroy(), the addresses are not reset. However for kvm selftest purpose this would allow to continue the test execution even after a failure when running KVM_RUN. So let's reset the base addresses.

---

v1 -> v2:
- use dist-> in the else and add braces

---
 arch/arm64/kvm/vgic/vgic-init.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic-init.c +5 -1
index 052917deb1495..cf6faa0aeddb2 100644
@@ -335,13 +335,16 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm)
 	kfree(dist->spis);
 	dist->spis = NULL;
 	dist->nr_spis = 0;
+	dist->vgic_dist_base = VGIC_ADDR_UNDEF;
 
-	if (kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
+	if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
 		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list) {
 			list_del(&rdreg->list);
 			kfree(rdreg);
 		}
 		INIT_LIST_HEAD(&dist->rd_regions);
+	} else {
+		dist->vgic_cpu_base = VGIC_ADDR_UNDEF;
 	}
 
 	if (vgic_has_its(kvm))
@@ -362,6 +365,7 @@ void kvm_vgic_vcpu_destroy(struct kvm_vcpu *vcpu)
 	vgic_flush_pending_lpis(vcpu);
 
 	INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
+	vgic_cpu->rd_iodev.base_addr = VGIC_ADDR_UNDEF;
 }
 
 /* To be called with kvm->lock held */
Signature · 6 lines
2.26.3

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#7 lore raw

PATCH v4 5/8 docs: kvm: devices/arm-vgic-v3: enhance KVM_DEV_ARM_VGIC_CTRL_INIT doc

In reply to Eric Auger

kvm_arch_vcpu_precreate() returns -EBUSY if the vgic is already initialized. So let's document that KVM_DEV_ARM_VGIC_CTRL_INIT must be called after all vcpu creations.

---

v1 -> v2:
- Must be called after all vcpu creations -> Must be called after all VCPUs have been created as per Alexandru's suggestion

---
 Documentation/virt/kvm/devices/arm-vgic-v3.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/virt/kvm/devices/arm-vgic-v3.rst +1 -1
index 5dd3bff519783..51e5e57625716 100644
@@ -228,7 +228,7 @@ Groups:
 
     KVM_DEV_ARM_VGIC_CTRL_INIT
       request the initialization of the VGIC, no additional parameter in
-      kvm_device_attr.addr.
+      kvm_device_attr.addr. Must be called after all VCPUs have been created.
     KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES
       save all LPI pending bits into guest RAM pending tables.
 
Signature · 6 lines
2.26.3

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#8 lore raw

PATCH v4 6/8 KVM: arm64: Simplify argument passing to vgic_uaccess_[read|write]

In reply to Eric Auger

vgic_uaccess() takes a struct vgic_io_device argument, converts it to a struct kvm_io_device and passes it to the read/write accessor functions, which convert it back to a struct vgic_io_device. Avoid the indirection by passing the struct vgic_io_device argument directly to vgic_uaccess_{read,write}.

---

v1 -> v2:
- reworded the commit message as suggested by Alexandru

---
 arch/arm64/kvm/vgic/vgic-mmio.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
arch/arm64/kvm/vgic/vgic-mmio.c +4 -6
index b2d73fc0d1ef4..48c6067fc5ecb 100644
@@ -938,10 +938,9 @@ vgic_get_mmio_region(struct kvm_vcpu *vcpu, struct vgic_io_device *iodev,
 	return region;
 }
 
-static int vgic_uaccess_read(struct kvm_vcpu *vcpu, struct kvm_io_device *dev,
+static int vgic_uaccess_read(struct kvm_vcpu *vcpu, struct vgic_io_device *iodev,
 			     gpa_t addr, u32 *val)
 {
-	struct vgic_io_device *iodev = kvm_to_vgic_iodev(dev);
 	const struct vgic_register_region *region;
 	struct kvm_vcpu *r_vcpu;
 
@@ -960,10 +959,9 @@ static int vgic_uaccess_read(struct kvm_vcpu *vcpu, struct kvm_io_device *dev,
 	return 0;
 }
 
-static int vgic_uaccess_write(struct kvm_vcpu *vcpu, struct kvm_io_device *dev,
+static int vgic_uaccess_write(struct kvm_vcpu *vcpu, struct vgic_io_device *iodev,
 			      gpa_t addr, const u32 *val)
 {
-	struct vgic_io_device *iodev = kvm_to_vgic_iodev(dev);
 	const struct vgic_register_region *region;
 	struct kvm_vcpu *r_vcpu;
 
@@ -986,9 +984,9 @@ int vgic_uaccess(struct kvm_vcpu *vcpu, struct vgic_io_device *dev,
 		 bool is_write, int offset, u32 *val)
 {
 	if (is_write)
-		return vgic_uaccess_write(vcpu, &dev->dev, offset, val);
+		return vgic_uaccess_write(vcpu, dev, offset, val);
 	else
-		return vgic_uaccess_read(vcpu, &dev->dev, offset, val);
+		return vgic_uaccess_read(vcpu, dev, offset, val);
 }
 
 static int dispatch_mmio_read(struct kvm_vcpu *vcpu, struct kvm_io_device *dev,
Signature · 6 lines
2.26.3

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#9 lore raw

PATCH v4 7/8 KVM: arm64: vgic-v3: Expose GICR_TYPER.Last for userspace

In reply to Eric Auger

Commit 23bde34771f1 ("KVM: arm64: vgic-v3: Drop the reporting of GICR_TYPER.Last for userspace") temporarily fixed a bug identified when attempting to access the GICR_TYPER register before the redistributor region setting, but dropped the support of the LAST bit.

Emulating the GICR_TYPER.Last bit still makes sense for architecture compliance though. This patch restores its support (if the redistributor region was set) while keeping the code safe.

We introduce a new helper, vgic_mmio_vcpu_rdist_is_last() which computes whether a redistributor is the highest one of a series of redistributor contributor pages.

The spec says "Indicates whether this Redistributor is the highest-numbered Redistributor in a series of contiguous Redistributor pages."

The code is a bit convulated since there is no guarantee redistributors are added in a given reditributor region in ascending order. In that case the current implementation was wrong. Also redistributor regions can be contiguous and registered in non increasing base address order.

So the index of redistributors are stored in an array within the redistributor region structure.

With this new implementation we do not need to have a uaccess read accessor anymore.

---
 arch/arm64/kvm/vgic/vgic-init.c    |  7 +--
 arch/arm64/kvm/vgic/vgic-mmio-v3.c | 97 ++++++++++++++++++++----------
 arch/arm64/kvm/vgic/vgic.h         |  1 +
 include/kvm/arm_vgic.h             |  3 +
 4 files changed, 73 insertions(+), 35 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +3 -4
index cf6faa0aeddb2..61150c34c268c 100644
@@ -190,6 +190,7 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
 	int i;
 
 	vgic_cpu->rd_iodev.base_addr = VGIC_ADDR_UNDEF;
+	vgic_cpu->index = vcpu->vcpu_id;
 
 	INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
 	raw_spin_lock_init(&vgic_cpu->ap_list_lock);
@@ -338,10 +339,8 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm)
 	dist->vgic_dist_base = VGIC_ADDR_UNDEF;
 
 	if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
-		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list) {
-			list_del(&rdreg->list);
-			kfree(rdreg);
-		}
+		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list)
+			vgic_v3_free_redist_region(rdreg);
 		INIT_LIST_HEAD(&dist->rd_regions);
 	} else {
 		dist->vgic_cpu_base = VGIC_ADDR_UNDEF;
arch/arm64/kvm/vgic/vgic-mmio-v3.c +66 -31

164 lines not on this page · raw message

arch/arm64/kvm/vgic/vgic.h +1 -0
index 64fcd75111108..bc418c2c12141 100644
@@ -293,6 +293,7 @@ vgic_v3_rd_region_size(struct kvm *kvm, struct vgic_redist_region *rdreg)
 
 struct vgic_redist_region *vgic_v3_rdist_region_from_index(struct kvm *kvm,
 							   u32 index);
+void vgic_v3_free_redist_region(struct vgic_redist_region *rdreg);
 
 bool vgic_v3_rdist_overlap(struct kvm *kvm, gpa_t base, size_t size);
 
include/kvm/arm_vgic.h +3 -0
index 3d74f1060bd18..9a3f060ac3547 100644
@@ -197,6 +197,7 @@ struct vgic_redist_region {
 	gpa_t base;
 	u32 count; /* number of redistributors or 0 if single region */
 	u32 free_index; /* index of the next free redistributor */
+	int *rdist_indices; /* indices of the redistributors */
 	struct list_head list;
 };
 
@@ -322,6 +323,8 @@ struct vgic_cpu {
 	 */
 	struct vgic_io_device	rd_iodev;
 	struct vgic_redist_region *rdreg;
+	u32 rdreg_index;
+	int index; /* vcpu index */
 
 	/* Contains the attributes and gpa of the LPI pending tables. */
 	u64 pendbaser;
Signature · 6 lines
2.26.3

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
maintainer
#10 lore raw

Re: [PATCH v4 7/8] KVM: arm64: vgic-v3: Expose GICR_TYPER.Last for userspace

Hi Eric,

Quoted 15 lines · Eric Auger

Commit 23bde34771f1 ("KVM: arm64: vgic-v3: Drop the reporting of GICR_TYPER.Last for userspace") temporarily fixed a bug identified when attempting to access the GICR_TYPER register before the redistributor region setting, but dropped the support of the LAST bit.

Emulating the GICR_TYPER.Last bit still makes sense for architecture compliance though. This patch restores its support (if the redistributor region was set) while keeping the code safe.

We introduce a new helper, vgic_mmio_vcpu_rdist_is_last() which computes whether a redistributor is the highest one of a series of redistributor contributor pages.

Quoted 5 lines

The spec says "Indicates whether this Redistributor is the highest-numbered Redistributor in a series of contiguous Redistributor pages."

The code is a bit convulated since there is no guarantee

nit: convoluted

Quoted 5 lines

redistributors are added in a given reditributor region in ascending order. In that case the current implementation was wrong. Also redistributor regions can be contiguous and registered in non increasing base address order.

Quoted 7 lines

So the index of redistributors are stored in an array within the redistributor region structure.

With this new implementation we do not need to have a uaccess read accessor anymore.

This patch also hurt my head, a lot more than the first one. See below.

Quoted 12 lines
---
 arch/arm64/kvm/vgic/vgic-init.c    |  7 +--
 arch/arm64/kvm/vgic/vgic-mmio-v3.c | 97 ++++++++++++++++++++----------
 arch/arm64/kvm/vgic/vgic.h         |  1 +
 include/kvm/arm_vgic.h             |  3 +
 4 files changed, 73 insertions(+), 35 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -0
index cf6faa0aeddb2..61150c34c268c 100644

arch/arm64/kvm/vgic/vgic-init.c int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu) · line 193

Quoted 4 lines
arch/arm64/kvm/vgic/vgic-init.c +1 -0
@@ -190,6 +190,7 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
 	int i;
 
 	vgic_cpu->rd_iodev.base_addr = VGIC_ADDR_UNDEF;
+	vgic_cpu->index = vcpu->vcpu_id;

Is it so that vgic_cpu->index is always equal to vcpu_id? If so, why do we need another field? We can always get to the vcpu using a container_of().

arch/arm64/kvm/vgic/vgic-init.c static void kvm_vgic_dist_destroy(struct kvm *kvm) · line 343

Quoted 13 lines
arch/arm64/kvm/vgic/vgic-init.c +2 -4
 	INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
 	raw_spin_lock_init(&vgic_cpu->ap_list_lock);
@@ -338,10 +339,8 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm)
 	dist->vgic_dist_base = VGIC_ADDR_UNDEF;
 
 	if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
-		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list) {
-			list_del(&rdreg->list);
-			kfree(rdreg);
-		}
+		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list)
+			vgic_v3_free_redist_region(rdreg);

Consider moving the introduction of vgic_v3_free_redist_region() into a separate patch. On its own, that's a good readability improvement.

Quoted 21 lines
arch/arm64/kvm/vgic/vgic-init.c +0 -0
 		INIT_LIST_HEAD(&dist->rd_regions);
 	} else {
 		dist->vgic_cpu_base = VGIC_ADDR_UNDEF;
arch/arm64/kvm/vgic/vgic-mmio-v3.c +10 -0
index 987e366c80008..f6a7eed1d6adb 100644
@@ -251,45 +251,57 @@ static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu,
 		vgic_enable_lpis(vcpu);
 }
 
+static bool vgic_mmio_vcpu_rdist_is_last(struct kvm_vcpu *vcpu)
+{
+	struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
+	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+	struct vgic_redist_region *rdreg = vgic_cpu->rdreg;
+
+	if (!rdreg)
+		return false;
+
+	if (rdreg->count && vgic_cpu->rdreg_index == (rdreg->count - 1)) {

arch/arm64/kvm/vgic/vgic-mmio-v3.c static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu, · line 273

Quoted 10 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +10 -0
+		/* check whether there is no other contiguous rdist region */
+		struct list_head *rd_regions = &vgic->rd_regions;
+		struct vgic_redist_region *iter;
+
+		list_for_each_entry(iter, rd_regions, list) {
+			if (iter->base == rdreg->base + rdreg->count * KVM_VGIC_V3_REDIST_SIZE &&
+				iter->free_index > 0) {
+			/* check the first rdist index of this region, if any */
+				if (vgic_cpu->index < iter->rdist_indices[0])
+					return false;

rdist_indices[] contains the vcpu_id of the vcpu associated with a given RD in the region. At this stage, you have established that there is another region that is contiguous with the one associated with our vcpu. You also know that this adjacent region has a vcpu mapped in (free_index isn't 0). Isn't that enough to declare that our vcpu isn't last? I definitely don't understand what the index comparison does here.

It also seem to me that some of the complexity could be eliminated if the regions were kept ordered at list insertion time.

arch/arm64/kvm/vgic/vgic-mmio-v3.c static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu, · line 281

Quoted 8 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +8 -0
+			}
+		}
+	} else if (vgic_cpu->rdreg_index < rdreg->free_index - 1) {
+		/* look at the index of next rdist */
+		int next_rdist_index = rdreg->rdist_indices[vgic_cpu->rdreg_index + 1];
+
+		if (vgic_cpu->index < next_rdist_index)
+			return false;

Same thing here. We are in the middle of the allocated part of a region, which means we cannot be last. I still don't get the index check.

Quoted 63 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +9 -25
+	}
+	return true;
+}
+
 static unsigned long vgic_mmio_read_v3r_typer(struct kvm_vcpu *vcpu,
 					      gpa_t addr, unsigned int len)
 {
 	unsigned long mpidr = kvm_vcpu_get_mpidr_aff(vcpu);
-	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
-	struct vgic_redist_region *rdreg = vgic_cpu->rdreg;
 	int target_vcpu_id = vcpu->vcpu_id;
-	gpa_t last_rdist_typer = rdreg->base + GICR_TYPER +
-			(rdreg->free_index - 1) * KVM_VGIC_V3_REDIST_SIZE;
 	u64 value;
 
 	value = (u64)(mpidr & GENMASK(23, 0)) << 32;
 	value |= ((target_vcpu_id & 0xffff) << 8);
 
-	if (addr == last_rdist_typer)
+	if (vgic_has_its(vcpu->kvm))
+		value |= GICR_TYPER_PLPIS;
+
+	if (vgic_mmio_vcpu_rdist_is_last(vcpu))
 		value |= GICR_TYPER_LAST;
-	if (vgic_has_its(vcpu->kvm))
-		value |= GICR_TYPER_PLPIS;
 
 	return extract_bytes(value, addr & 7, len);
 }
 
-static unsigned long vgic_uaccess_read_v3r_typer(struct kvm_vcpu *vcpu,
-						 gpa_t addr, unsigned int len)
-{
-	unsigned long mpidr = kvm_vcpu_get_mpidr_aff(vcpu);
-	int target_vcpu_id = vcpu->vcpu_id;
-	u64 value;
-
-	value = (u64)(mpidr & GENMASK(23, 0)) << 32;
-	value |= ((target_vcpu_id & 0xffff) << 8);
-
-	if (vgic_has_its(vcpu->kvm))
-		value |= GICR_TYPER_PLPIS;
-
-	/* reporting of the Last bit is not supported for userspace */
-	return extract_bytes(value, addr & 7, len);
-}
-
 static unsigned long vgic_mmio_read_v3r_iidr(struct kvm_vcpu *vcpu,
 					     gpa_t addr, unsigned int len)
 {
@@ -612,7 +624,7 @@ static const struct vgic_register_region vgic_v3_rd_registers[] = {
 		VGIC_ACCESS_32bit),
 	REGISTER_DESC_WITH_LENGTH_UACCESS(GICR_TYPER,
 		vgic_mmio_read_v3r_typer, vgic_mmio_write_wi,
-		vgic_uaccess_read_v3r_typer, vgic_mmio_uaccess_write_wi, 8,
+		NULL, vgic_mmio_uaccess_write_wi, 8,
 		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
 	REGISTER_DESC_WITH_LENGTH(GICR_WAKER,
 		vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
@@ -714,6 +726,16 @@ int vgic_register_redist_iodev(struct kvm_vcpu *vcpu)
 		return -EINVAL;
 
 	vgic_cpu->rdreg = rdreg;

arch/arm64/kvm/vgic/vgic-mmio-v3.c int vgic_register_redist_iodev(struct kvm_vcpu *vcpu) · line 738

Quoted 10 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +10 -0
+	vgic_cpu->rdreg_index = rdreg->free_index;
+	if (!rdreg->count) {
+		void *p = krealloc(rdreg->rdist_indices,
+				   (vgic_cpu->rdreg_index + 1) * sizeof(u32),
+				   GFP_KERNEL);
+		if (!p)
+			return -ENOMEM;
+		rdreg->rdist_indices = p;
+	}
+	rdreg->rdist_indices[vgic_cpu->rdreg_index] = vgic_cpu->index;

I think I really have a problem with this array, which comes from me not understanding the two checks I previously commented on.

If we stick to the definition of 'Last', all that matters is the position of the RD in a region (rdreg_index) and potentially the presence of another contiguous region with allocated RDs in it.

IIUC, the checks should read like this:

if (vcpu->rdreg_index < (vcpu->rdreg->free_index - 1))

	last = false;
else if (vcpu->rdreg_index == (vcpu->rdreg->free_index - 1) &&
	 adjacent_region(vcpu->rdreg)->free_index > 0)
	last = false;
else
	last = true;

So why do we need to track the vcpu_id associated to a region?

Quoted 84 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +19 -6
 	rd_base = rdreg->base + rdreg->free_index * KVM_VGIC_V3_REDIST_SIZE;
 
@@ -768,7 +790,7 @@ static int vgic_register_all_redist_iodevs(struct kvm *kvm)
 }
 
 /**
- * vgic_v3_insert_redist_region - Insert a new redistributor region
+ * vgic_v3_alloc_redist_region - Allocate a new redistributor region
  *
  * Performs various checks before inserting the rdist region in the list.
  * Those tests depend on whether the size of the rdist region is known
@@ -782,8 +804,8 @@ static int vgic_register_all_redist_iodevs(struct kvm *kvm)
  *
  * Return 0 on success, < 0 otherwise
  */
-static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index,
-					gpa_t base, uint32_t count)
+static int vgic_v3_alloc_redist_region(struct kvm *kvm, uint32_t index,
+				       gpa_t base, uint32_t count)
 {
 	struct vgic_dist *d = &kvm->arch.vgic;
 	struct vgic_redist_region *rdreg;
@@ -839,6 +861,13 @@ static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index,
 	rdreg->count = count;
 	rdreg->free_index = 0;
 	rdreg->index = index;
+	if (count) {
+		rdreg->rdist_indices = kcalloc(count, sizeof(u32), GFP_KERNEL);
+		if (!rdreg->rdist_indices) {
+			ret = -ENOMEM;
+			goto free;
+		}
+	}
 
 	list_add_tail(&rdreg->list, rd_regions);
 	return 0;
@@ -847,11 +876,18 @@ static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index,
 	return ret;
 }
 
+void vgic_v3_free_redist_region(struct vgic_redist_region *rdreg)
+{
+	list_del(&rdreg->list);
+	kfree(rdreg->rdist_indices);
+	kfree(rdreg);
+}
+
 int vgic_v3_set_redist_base(struct kvm *kvm, u32 index, u64 addr, u32 count)
 {
 	int ret;
 
-	ret = vgic_v3_insert_redist_region(kvm, index, addr, count);
+	ret = vgic_v3_alloc_redist_region(kvm, index, addr, count);
 	if (ret)
 		return ret;
 
@@ -864,8 +900,7 @@ int vgic_v3_set_redist_base(struct kvm *kvm, u32 index, u64 addr, u32 count)
 		struct vgic_redist_region *rdreg;
 
 		rdreg = vgic_v3_rdist_region_from_index(kvm, index);
-		list_del(&rdreg->list);
-		kfree(rdreg);
+		vgic_v3_free_redist_region(rdreg);
 		return ret;
 	}
 
arch/arm64/kvm/vgic/vgic.h +1 -0
index 64fcd75111108..bc418c2c12141 100644
@@ -293,6 +293,7 @@ vgic_v3_rd_region_size(struct kvm *kvm, struct vgic_redist_region *rdreg)
 
 struct vgic_redist_region *vgic_v3_rdist_region_from_index(struct kvm *kvm,
 							   u32 index);
+void vgic_v3_free_redist_region(struct vgic_redist_region *rdreg);
 
 bool vgic_v3_rdist_overlap(struct kvm *kvm, gpa_t base, size_t size);
 
include/kvm/arm_vgic.h +0 -0
index 3d74f1060bd18..9a3f060ac3547 100644

include/kvm/arm_vgic.h struct vgic_redist_region { · line 200

Quoted 4 lines
include/kvm/arm_vgic.h +1 -0
@@ -197,6 +197,7 @@ struct vgic_redist_region {
 	gpa_t base;
 	u32 count; /* number of redistributors or 0 if single region */
 	u32 free_index; /* index of the next free redistributor */
+	int *rdist_indices; /* indices of the redistributors */

You are treating it as an array of u32 when allocating it. Please choose one type or the other.

include/kvm/arm_vgic.h struct vgic_cpu { · line 330

Quoted 12 lines
include/kvm/arm_vgic.h +2 -0
 	struct list_head list;
 };
 
@@ -322,6 +323,8 @@ struct vgic_cpu {
 	 */
 	struct vgic_io_device	rd_iodev;
 	struct vgic_redist_region *rdreg;
+	u32 rdreg_index;
+	int index; /* vcpu index */
 
 	/* Contains the attributes and gpa of the LPI pending tables. */
 	u64 pendbaser;

Thanks,

	M.
Signature · 5 lines
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#11 lore raw

Re: [PATCH v4 7/8] KVM: arm64: vgic-v3: Expose GICR_TYPER.Last for userspace

Hi Marc,

Quoted 113 lines · Marc Zyngier

Hi Eric,

Quoted 15 lines · Eric Auger

Commit 23bde34771f1 ("KVM: arm64: vgic-v3: Drop the reporting of GICR_TYPER.Last for userspace") temporarily fixed a bug identified when attempting to access the GICR_TYPER register before the redistributor region setting, but dropped the support of the LAST bit.

Emulating the GICR_TYPER.Last bit still makes sense for architecture compliance though. This patch restores its support (if the redistributor region was set) while keeping the code safe.

We introduce a new helper, vgic_mmio_vcpu_rdist_is_last() which computes whether a redistributor is the highest one of a series of redistributor contributor pages.

Quoted 5 lines

The spec says "Indicates whether this Redistributor is the highest-numbered Redistributor in a series of contiguous Redistributor pages."

The code is a bit convulated since there is no guarantee

nit: convoluted

Quoted 5 lines

redistributors are added in a given reditributor region in ascending order. In that case the current implementation was wrong. Also redistributor regions can be contiguous and registered in non increasing base address order.

Quoted 7 lines

So the index of redistributors are stored in an array within the redistributor region structure.

With this new implementation we do not need to have a uaccess read accessor anymore.

This patch also hurt my head, a lot more than the first one. See below.

Quoted 12 lines
---
 arch/arm64/kvm/vgic/vgic-init.c    |  7 +--
 arch/arm64/kvm/vgic/vgic-mmio-v3.c | 97 ++++++++++++++++++++----------
 arch/arm64/kvm/vgic/vgic.h         |  1 +
 include/kvm/arm_vgic.h             |  3 +
 4 files changed, 73 insertions(+), 35 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -0
index cf6faa0aeddb2..61150c34c268c 100644

arch/arm64/kvm/vgic/vgic-init.c int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu) · line 193

Quoted 4 lines
arch/arm64/kvm/vgic/vgic-init.c +1 -0
@@ -190,6 +190,7 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
 	int i;
 
 	vgic_cpu->rd_iodev.base_addr = VGIC_ADDR_UNDEF;
+	vgic_cpu->index = vcpu->vcpu_id;

Is it so that vgic_cpu->index is always equal to vcpu_id? If so, why do we need another field? We can always get to the vcpu using a container_of().

arch/arm64/kvm/vgic/vgic-init.c static void kvm_vgic_dist_destroy(struct kvm *kvm) · line 343

Quoted 13 lines
arch/arm64/kvm/vgic/vgic-init.c +2 -4
 	INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
 	raw_spin_lock_init(&vgic_cpu->ap_list_lock);
@@ -338,10 +339,8 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm)
 	dist->vgic_dist_base = VGIC_ADDR_UNDEF;
 
 	if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
-		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list) {
-			list_del(&rdreg->list);
-			kfree(rdreg);
-		}
+		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list)
+			vgic_v3_free_redist_region(rdreg);

Consider moving the introduction of vgic_v3_free_redist_region() into a separate patch. On its own, that's a good readability improvement.

Quoted 21 lines
arch/arm64/kvm/vgic/vgic-init.c +0 -0
 		INIT_LIST_HEAD(&dist->rd_regions);
 	} else {
 		dist->vgic_cpu_base = VGIC_ADDR_UNDEF;
arch/arm64/kvm/vgic/vgic-mmio-v3.c +10 -0
index 987e366c80008..f6a7eed1d6adb 100644
@@ -251,45 +251,57 @@ static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu,
 		vgic_enable_lpis(vcpu);
 }
 
+static bool vgic_mmio_vcpu_rdist_is_last(struct kvm_vcpu *vcpu)
+{
+	struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
+	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+	struct vgic_redist_region *rdreg = vgic_cpu->rdreg;
+
+	if (!rdreg)
+		return false;
+
+	if (rdreg->count && vgic_cpu->rdreg_index == (rdreg->count - 1)) {

arch/arm64/kvm/vgic/vgic-mmio-v3.c static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu, · line 273

Quoted 10 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +10 -0
+		/* check whether there is no other contiguous rdist region */
+		struct list_head *rd_regions = &vgic->rd_regions;
+		struct vgic_redist_region *iter;
+
+		list_for_each_entry(iter, rd_regions, list) {
+			if (iter->base == rdreg->base + rdreg->count * KVM_VGIC_V3_REDIST_SIZE &&
+				iter->free_index > 0) {
+			/* check the first rdist index of this region, if any */
+				if (vgic_cpu->index < iter->rdist_indices[0])
+					return false;
Quoted 7 lines

rdist_indices[] contains the vcpu_id of the vcpu associated with a given RD in the region. At this stage, you have established that there is another region that is contiguous with the one associated with our vcpu. You also know that this adjacent region has a vcpu mapped in (free_index isn't 0). Isn't that enough to declare that our vcpu isn't last? I definitely don't understand what the index comparison does here.

Assume the following case:
2 RDIST region
region #0 contains rdist 1, 2, 4
region #1, adjacent to #0 contains rdist 3

Spec days:
"Indicates whether this Redistributor is the highest-numbered Redistributor in a series of contiguous Redistributor pages."

To me 4 is last and 3 is last too.

Quoted 3 lines

It also seem to me that some of the complexity could be eliminated if the regions were kept ordered at list insertion time.

yes

Quoted 13 lines

arch/arm64/kvm/vgic/vgic-mmio-v3.c static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu, · line 281

Quoted 9 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +8 -0
+			}
+		}
+	} else if (vgic_cpu->rdreg_index < rdreg->free_index - 1) {
+		/* look at the index of next rdist */
+		int next_rdist_index = rdreg->rdist_indices[vgic_cpu->rdreg_index + 1];
+
+		if (vgic_cpu->index < next_rdist_index)
+			return false;

Same thing here. We are in the middle of the allocated part of a region, which means we cannot be last. I still don't get the index check.

Because within a region, nothing hinders rdist from being allocated in non ascending order. I exercise those cases in the kvmselftests

one single RDIST region with the following rdists allocated there:
1, 3, 2

3 and 2 are "last", right? Or did I miss something. Yes that's totally not natural to do that kind of allocation but the API allows to do that.

Quoted 63 lines
Quoted 63 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +9 -25
+	}
+	return true;
+}
+
 static unsigned long vgic_mmio_read_v3r_typer(struct kvm_vcpu *vcpu,
 					      gpa_t addr, unsigned int len)
 {
 	unsigned long mpidr = kvm_vcpu_get_mpidr_aff(vcpu);
-	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
-	struct vgic_redist_region *rdreg = vgic_cpu->rdreg;
 	int target_vcpu_id = vcpu->vcpu_id;
-	gpa_t last_rdist_typer = rdreg->base + GICR_TYPER +
-			(rdreg->free_index - 1) * KVM_VGIC_V3_REDIST_SIZE;
 	u64 value;
 
 	value = (u64)(mpidr & GENMASK(23, 0)) << 32;
 	value |= ((target_vcpu_id & 0xffff) << 8);
 
-	if (addr == last_rdist_typer)
+	if (vgic_has_its(vcpu->kvm))
+		value |= GICR_TYPER_PLPIS;
+
+	if (vgic_mmio_vcpu_rdist_is_last(vcpu))
 		value |= GICR_TYPER_LAST;
-	if (vgic_has_its(vcpu->kvm))
-		value |= GICR_TYPER_PLPIS;
 
 	return extract_bytes(value, addr & 7, len);
 }
 
-static unsigned long vgic_uaccess_read_v3r_typer(struct kvm_vcpu *vcpu,
-						 gpa_t addr, unsigned int len)
-{
-	unsigned long mpidr = kvm_vcpu_get_mpidr_aff(vcpu);
-	int target_vcpu_id = vcpu->vcpu_id;
-	u64 value;
-
-	value = (u64)(mpidr & GENMASK(23, 0)) << 32;
-	value |= ((target_vcpu_id & 0xffff) << 8);
-
-	if (vgic_has_its(vcpu->kvm))
-		value |= GICR_TYPER_PLPIS;
-
-	/* reporting of the Last bit is not supported for userspace */
-	return extract_bytes(value, addr & 7, len);
-}
-
 static unsigned long vgic_mmio_read_v3r_iidr(struct kvm_vcpu *vcpu,
 					     gpa_t addr, unsigned int len)
 {
@@ -612,7 +624,7 @@ static const struct vgic_register_region vgic_v3_rd_registers[] = {
 		VGIC_ACCESS_32bit),
 	REGISTER_DESC_WITH_LENGTH_UACCESS(GICR_TYPER,
 		vgic_mmio_read_v3r_typer, vgic_mmio_write_wi,
-		vgic_uaccess_read_v3r_typer, vgic_mmio_uaccess_write_wi, 8,
+		NULL, vgic_mmio_uaccess_write_wi, 8,
 		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
 	REGISTER_DESC_WITH_LENGTH(GICR_WAKER,
 		vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
@@ -714,6 +726,16 @@ int vgic_register_redist_iodev(struct kvm_vcpu *vcpu)
 		return -EINVAL;
 
Quoted 14 lines

arch/arm64/kvm/vgic/vgic-mmio-v3.c int vgic_register_redist_iodev(struct kvm_vcpu *vcpu) · line 738

Quoted 11 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +10 -0
 	vgic_cpu->rdreg = rdreg;
+	vgic_cpu->rdreg_index = rdreg->free_index;
+	if (!rdreg->count) {
+		void *p = krealloc(rdreg->rdist_indices,
+				   (vgic_cpu->rdreg_index + 1) * sizeof(u32),
+				   GFP_KERNEL);
+		if (!p)
+			return -ENOMEM;
+		rdreg->rdist_indices = p;
+	}
+	rdreg->rdist_indices[vgic_cpu->rdreg_index] = vgic_cpu->index;

I think I really have a problem with this array, which comes from me not understanding the two checks I previously commented on.

I hope the above clarified the array need.

Quoted 9 lines

If we stick to the definition of 'Last', all that matters is the position of the RD in a region (rdreg_index) and potentially the presence of another contiguous region with allocated RDs in it.

IIUC, the checks should read like this:

if (vcpu->rdreg_index < (vcpu->rdreg->free_index - 1))

Quoted 8 lines
	last = false;
else if (vcpu->rdreg_index == (vcpu->rdreg->free_index - 1) &&
	 adjacent_region(vcpu->rdreg)->free_index > 0)
	last = false;
else
	last = true;

So why do we need to track the vcpu_id associated to a region?

because the redistributors within a region can be in random order. That's why I need to store their number.

Does that make more sense?

Thanks

Eric

Quoted 103 lines
Quoted 85 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +19 -6
 	rd_base = rdreg->base + rdreg->free_index * KVM_VGIC_V3_REDIST_SIZE;
 
@@ -768,7 +790,7 @@ static int vgic_register_all_redist_iodevs(struct kvm *kvm)
 }
 
 /**
- * vgic_v3_insert_redist_region - Insert a new redistributor region
+ * vgic_v3_alloc_redist_region - Allocate a new redistributor region
  *
  * Performs various checks before inserting the rdist region in the list.
  * Those tests depend on whether the size of the rdist region is known
@@ -782,8 +804,8 @@ static int vgic_register_all_redist_iodevs(struct kvm *kvm)
  *
  * Return 0 on success, < 0 otherwise
  */
-static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index,
-					gpa_t base, uint32_t count)
+static int vgic_v3_alloc_redist_region(struct kvm *kvm, uint32_t index,
+				       gpa_t base, uint32_t count)
 {
 	struct vgic_dist *d = &kvm->arch.vgic;
 	struct vgic_redist_region *rdreg;
@@ -839,6 +861,13 @@ static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index,
 	rdreg->count = count;
 	rdreg->free_index = 0;
 	rdreg->index = index;
+	if (count) {
+		rdreg->rdist_indices = kcalloc(count, sizeof(u32), GFP_KERNEL);
+		if (!rdreg->rdist_indices) {
+			ret = -ENOMEM;
+			goto free;
+		}
+	}
 
 	list_add_tail(&rdreg->list, rd_regions);
 	return 0;
@@ -847,11 +876,18 @@ static int vgic_v3_insert_redist_region(struct kvm *kvm, uint32_t index,
 	return ret;
 }
 
+void vgic_v3_free_redist_region(struct vgic_redist_region *rdreg)
+{
+	list_del(&rdreg->list);
+	kfree(rdreg->rdist_indices);
+	kfree(rdreg);
+}
+
 int vgic_v3_set_redist_base(struct kvm *kvm, u32 index, u64 addr, u32 count)
 {
 	int ret;
 
-	ret = vgic_v3_insert_redist_region(kvm, index, addr, count);
+	ret = vgic_v3_alloc_redist_region(kvm, index, addr, count);
 	if (ret)
 		return ret;
 
@@ -864,8 +900,7 @@ int vgic_v3_set_redist_base(struct kvm *kvm, u32 index, u64 addr, u32 count)
 		struct vgic_redist_region *rdreg;
 
 		rdreg = vgic_v3_rdist_region_from_index(kvm, index);
-		list_del(&rdreg->list);
-		kfree(rdreg);
+		vgic_v3_free_redist_region(rdreg);
 		return ret;
 	}
 
arch/arm64/kvm/vgic/vgic.h +1 -0
index 64fcd75111108..bc418c2c12141 100644
@@ -293,6 +293,7 @@ vgic_v3_rd_region_size(struct kvm *kvm, struct vgic_redist_region *rdreg)
 
 struct vgic_redist_region *vgic_v3_rdist_region_from_index(struct kvm *kvm,
 							   u32 index);
+void vgic_v3_free_redist_region(struct vgic_redist_region *rdreg);
 
 bool vgic_v3_rdist_overlap(struct kvm *kvm, gpa_t base, size_t size);
 
include/kvm/arm_vgic.h +0 -0
index 3d74f1060bd18..9a3f060ac3547 100644

include/kvm/arm_vgic.h struct vgic_redist_region { · line 200

Quoted 4 lines
include/kvm/arm_vgic.h +1 -0
@@ -197,6 +197,7 @@ struct vgic_redist_region {
 	gpa_t base;
 	u32 count; /* number of redistributors or 0 if single region */
 	u32 free_index; /* index of the next free redistributor */
+	int *rdist_indices; /* indices of the redistributors */

You are treating it as an array of u32 when allocating it. Please choose one type or the other.

Quoted 10 lines
include/kvm/arm_vgic.h +2 -0
 	struct list_head list;
 };
 
@@ -322,6 +323,8 @@ struct vgic_cpu {
 	 */
 	struct vgic_io_device	rd_iodev;
 	struct vgic_redist_region *rdreg;
+	u32 rdreg_index;
+	int index; /* vcpu index */
 
Quoted 7 lines

include/kvm/arm_vgic.h struct vgic_cpu { · line 330

Quoted 2 lines
include/kvm/arm_vgic.h +0 -0
 	/* Contains the attributes and gpa of the LPI pending tables. */
 	u64 pendbaser;

Thanks,

	M.

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

maintainer
#12 lore raw

Re: [PATCH v4 7/8] KVM: arm64: vgic-v3: Expose GICR_TYPER.Last for userspace

Quoted 129 lines · Auger Eric

Hi Marc,

Quoted 113 lines · Marc Zyngier

Hi Eric,

Quoted 15 lines · Eric Auger

Commit 23bde34771f1 ("KVM: arm64: vgic-v3: Drop the reporting of GICR_TYPER.Last for userspace") temporarily fixed a bug identified when attempting to access the GICR_TYPER register before the redistributor region setting, but dropped the support of the LAST bit.

Emulating the GICR_TYPER.Last bit still makes sense for architecture compliance though. This patch restores its support (if the redistributor region was set) while keeping the code safe.

We introduce a new helper, vgic_mmio_vcpu_rdist_is_last() which computes whether a redistributor is the highest one of a series of redistributor contributor pages.

Quoted 5 lines

The spec says "Indicates whether this Redistributor is the highest-numbered Redistributor in a series of contiguous Redistributor pages."

The code is a bit convulated since there is no guarantee

nit: convoluted

Quoted 5 lines

redistributors are added in a given reditributor region in ascending order. In that case the current implementation was wrong. Also redistributor regions can be contiguous and registered in non increasing base address order.

Quoted 7 lines

So the index of redistributors are stored in an array within the redistributor region structure.

With this new implementation we do not need to have a uaccess read accessor anymore.

This patch also hurt my head, a lot more than the first one. See below.

Quoted 12 lines
---
 arch/arm64/kvm/vgic/vgic-init.c    |  7 +--
 arch/arm64/kvm/vgic/vgic-mmio-v3.c | 97 ++++++++++++++++++++----------
 arch/arm64/kvm/vgic/vgic.h         |  1 +
 include/kvm/arm_vgic.h             |  3 +
 4 files changed, 73 insertions(+), 35 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -0
index cf6faa0aeddb2..61150c34c268c 100644

arch/arm64/kvm/vgic/vgic-init.c int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu) · line 193

Quoted 4 lines
arch/arm64/kvm/vgic/vgic-init.c +1 -0
@@ -190,6 +190,7 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
 	int i;
 
 	vgic_cpu->rd_iodev.base_addr = VGIC_ADDR_UNDEF;
+	vgic_cpu->index = vcpu->vcpu_id;

Is it so that vgic_cpu->index is always equal to vcpu_id? If so, why do we need another field? We can always get to the vcpu using a container_of().

arch/arm64/kvm/vgic/vgic-init.c static void kvm_vgic_dist_destroy(struct kvm *kvm) · line 343

Quoted 13 lines
arch/arm64/kvm/vgic/vgic-init.c +2 -4
 	INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
 	raw_spin_lock_init(&vgic_cpu->ap_list_lock);
@@ -338,10 +339,8 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm)
 	dist->vgic_dist_base = VGIC_ADDR_UNDEF;
 
 	if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
-		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list) {
-			list_del(&rdreg->list);
-			kfree(rdreg);
-		}
+		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list)
+			vgic_v3_free_redist_region(rdreg);

Consider moving the introduction of vgic_v3_free_redist_region() into a separate patch. On its own, that's a good readability improvement.

Quoted 21 lines
arch/arm64/kvm/vgic/vgic-init.c +0 -0
 		INIT_LIST_HEAD(&dist->rd_regions);
 	} else {
 		dist->vgic_cpu_base = VGIC_ADDR_UNDEF;
arch/arm64/kvm/vgic/vgic-mmio-v3.c +10 -0
index 987e366c80008..f6a7eed1d6adb 100644
@@ -251,45 +251,57 @@ static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu,
 		vgic_enable_lpis(vcpu);
 }
 
+static bool vgic_mmio_vcpu_rdist_is_last(struct kvm_vcpu *vcpu)
+{
+	struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
+	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+	struct vgic_redist_region *rdreg = vgic_cpu->rdreg;
+
+	if (!rdreg)
+		return false;
+
+	if (rdreg->count && vgic_cpu->rdreg_index == (rdreg->count - 1)) {

arch/arm64/kvm/vgic/vgic-mmio-v3.c static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu, · line 273

Quoted 10 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +10 -0
+		/* check whether there is no other contiguous rdist region */
+		struct list_head *rd_regions = &vgic->rd_regions;
+		struct vgic_redist_region *iter;
+
+		list_for_each_entry(iter, rd_regions, list) {
+			if (iter->base == rdreg->base + rdreg->count * KVM_VGIC_V3_REDIST_SIZE &&
+				iter->free_index > 0) {
+			/* check the first rdist index of this region, if any */
+				if (vgic_cpu->index < iter->rdist_indices[0])
+					return false;
Quoted 7 lines

rdist_indices[] contains the vcpu_id of the vcpu associated with a given RD in the region. At this stage, you have established that there is another region that is contiguous with the one associated with our vcpu. You also know that this adjacent region has a vcpu mapped in (free_index isn't 0). Isn't that enough to declare that our vcpu isn't last? I definitely don't understand what the index comparison does here.

Assume the following case:
2 RDIST region
region #0 contains rdist 1, 2, 4
region #1, adjacent to #0 contains rdist 3

Quoted 6 lines

Spec days:
"Indicates whether this Redistributor is the highest-numbered Redistributor in a series of contiguous Redistributor pages."

To me 4 is last and 3 is last too.

No, only 3 is last, assuming that region 0 is full. I think the phrasing in the spec is just really bad. What this describes is that at the end of a set of contiguous set of RDs, that last RD has Last set. If two regions are contiguous, that's undistinguishable from a single, larger region.

There is no such thing as a "redistributor number" anyway. The closest thing there is would be "processor number", but that has nothing to do with the RD itself.

Quoted 19 lines
Quoted 5 lines

It also seem to me that some of the complexity could be eliminated if the regions were kept ordered at list insertion time.

yes

Quoted 13 lines

arch/arm64/kvm/vgic/vgic-mmio-v3.c static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu, · line 281

Quoted 9 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +8 -0
+			}
+		}
+	} else if (vgic_cpu->rdreg_index < rdreg->free_index - 1) {
+		/* look at the index of next rdist */
+		int next_rdist_index = rdreg->rdist_indices[vgic_cpu->rdreg_index + 1];
+
+		if (vgic_cpu->index < next_rdist_index)
+			return false;

Same thing here. We are in the middle of the allocated part of a region, which means we cannot be last. I still don't get the index check.

Quoted 8 lines

Because within a region, nothing hinders rdist from being allocated in non ascending order. I exercise those cases in the kvmselftests

one single RDIST region with the following rdists allocated there:
1, 3, 2

3 and 2 are "last", right? Or did I miss something. Yes that's totally not natural to do that kind of allocation but the API allows to do that.

No, only 2 is last. I think you got tripped by the bizarre language in the spec, and the behaviour of this Last bit is much simpler than what you ended up with.

Thanks,

	M.
Signature · 5 lines
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author
#13 lore raw

Re: [PATCH v4 7/8] KVM: arm64: vgic-v3: Expose GICR_TYPER.Last for userspace

Hi Marc,

Quoted 144 lines · Marc Zyngier
Quoted 129 lines · Auger Eric

Hi Marc,

Quoted 113 lines · Marc Zyngier

Hi Eric,

Quoted 15 lines · Eric Auger

Commit 23bde34771f1 ("KVM: arm64: vgic-v3: Drop the reporting of GICR_TYPER.Last for userspace") temporarily fixed a bug identified when attempting to access the GICR_TYPER register before the redistributor region setting, but dropped the support of the LAST bit.

Emulating the GICR_TYPER.Last bit still makes sense for architecture compliance though. This patch restores its support (if the redistributor region was set) while keeping the code safe.

We introduce a new helper, vgic_mmio_vcpu_rdist_is_last() which computes whether a redistributor is the highest one of a series of redistributor contributor pages.

Quoted 5 lines

The spec says "Indicates whether this Redistributor is the highest-numbered Redistributor in a series of contiguous Redistributor pages."

The code is a bit convulated since there is no guarantee

nit: convoluted

Quoted 5 lines

redistributors are added in a given reditributor region in ascending order. In that case the current implementation was wrong. Also redistributor regions can be contiguous and registered in non increasing base address order.

Quoted 7 lines

So the index of redistributors are stored in an array within the redistributor region structure.

With this new implementation we do not need to have a uaccess read accessor anymore.

This patch also hurt my head, a lot more than the first one. See below.

Quoted 12 lines
---
 arch/arm64/kvm/vgic/vgic-init.c    |  7 +--
 arch/arm64/kvm/vgic/vgic-mmio-v3.c | 97 ++++++++++++++++++++----------
 arch/arm64/kvm/vgic/vgic.h         |  1 +
 include/kvm/arm_vgic.h             |  3 +
 4 files changed, 73 insertions(+), 35 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -0
index cf6faa0aeddb2..61150c34c268c 100644

arch/arm64/kvm/vgic/vgic-init.c int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu) · line 193

Quoted 4 lines
arch/arm64/kvm/vgic/vgic-init.c +1 -0
@@ -190,6 +190,7 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
 	int i;
 
 	vgic_cpu->rd_iodev.base_addr = VGIC_ADDR_UNDEF;
+	vgic_cpu->index = vcpu->vcpu_id;

Is it so that vgic_cpu->index is always equal to vcpu_id? If so, why do we need another field? We can always get to the vcpu using a container_of().

arch/arm64/kvm/vgic/vgic-init.c static void kvm_vgic_dist_destroy(struct kvm *kvm) · line 343

Quoted 13 lines
arch/arm64/kvm/vgic/vgic-init.c +2 -4
 	INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
 	raw_spin_lock_init(&vgic_cpu->ap_list_lock);
@@ -338,10 +339,8 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm)
 	dist->vgic_dist_base = VGIC_ADDR_UNDEF;
 
 	if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
-		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list) {
-			list_del(&rdreg->list);
-			kfree(rdreg);
-		}
+		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list)
+			vgic_v3_free_redist_region(rdreg);

Consider moving the introduction of vgic_v3_free_redist_region() into a separate patch. On its own, that's a good readability improvement.

Quoted 21 lines
arch/arm64/kvm/vgic/vgic-init.c +0 -0
 		INIT_LIST_HEAD(&dist->rd_regions);
 	} else {
 		dist->vgic_cpu_base = VGIC_ADDR_UNDEF;
arch/arm64/kvm/vgic/vgic-mmio-v3.c +10 -0
index 987e366c80008..f6a7eed1d6adb 100644
@@ -251,45 +251,57 @@ static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu,
 		vgic_enable_lpis(vcpu);
 }
 
+static bool vgic_mmio_vcpu_rdist_is_last(struct kvm_vcpu *vcpu)
+{
+	struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
+	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+	struct vgic_redist_region *rdreg = vgic_cpu->rdreg;
+
+	if (!rdreg)
+		return false;
+
+	if (rdreg->count && vgic_cpu->rdreg_index == (rdreg->count - 1)) {

arch/arm64/kvm/vgic/vgic-mmio-v3.c static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu, · line 273

Quoted 10 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +10 -0
+		/* check whether there is no other contiguous rdist region */
+		struct list_head *rd_regions = &vgic->rd_regions;
+		struct vgic_redist_region *iter;
+
+		list_for_each_entry(iter, rd_regions, list) {
+			if (iter->base == rdreg->base + rdreg->count * KVM_VGIC_V3_REDIST_SIZE &&
+				iter->free_index > 0) {
+			/* check the first rdist index of this region, if any */
+				if (vgic_cpu->index < iter->rdist_indices[0])
+					return false;
Quoted 7 lines

rdist_indices[] contains the vcpu_id of the vcpu associated with a given RD in the region. At this stage, you have established that there is another region that is contiguous with the one associated with our vcpu. You also know that this adjacent region has a vcpu mapped in (free_index isn't 0). Isn't that enough to declare that our vcpu isn't last? I definitely don't understand what the index comparison does here.

Assume the following case:
2 RDIST region
region #0 contains rdist 1, 2, 4
region #1, adjacent to #0 contains rdist 3

Quoted 6 lines

Spec days:
"Indicates whether this Redistributor is the highest-numbered Redistributor in a series of contiguous Redistributor pages."

To me 4 is last and 3 is last too.

No, only 3 is last, assuming that region 0 is full. I think the phrasing in the spec is just really bad. What this describes is that at the end of a set of contiguous set of RDs, that last RD has Last set. If two regions are contiguous, that's undistinguishable from a single, larger region.

Quoted 3 lines

There is no such thing as a "redistributor number" anyway. The closest thing there is would be "processor number", but that has nothing to do with the RD itself.

Hum OK. That's a different understanding of the spec wording indeed. For me redistributor number was the index of the vcpu.

But well, you're understanding is definitively simpler to implement and also matches what was implemented for single RDIST region.

Quoted 26 lines
Quoted 26 lines
Quoted 6 lines

It also seem to me that some of the complexity could be eliminated if the regions were kept ordered at list insertion time.

yes

Quoted 13 lines

arch/arm64/kvm/vgic/vgic-mmio-v3.c static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu, · line 281

Quoted 9 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +8 -0
+			}
+		}
+	} else if (vgic_cpu->rdreg_index < rdreg->free_index - 1) {
+		/* look at the index of next rdist */
+		int next_rdist_index = rdreg->rdist_indices[vgic_cpu->rdreg_index + 1];
+
+		if (vgic_cpu->index < next_rdist_index)
+			return false;

Same thing here. We are in the middle of the allocated part of a region, which means we cannot be last. I still don't get the index check.

Because within a region, nothing hinders rdist from being allocated in non ascending order. I exercise those cases in the kvmselftests

one single RDIST region with the following rdists allocated there:
1, 3, 2

Quoted 6 lines
Quoted 2 lines

3 and 2 are "last", right? Or did I miss something. Yes that's totally not natural to do that kind of allocation but the API allows to do that.

No, only 2 is last. I think you got tripped by the bizarre language in the spec, and the behaviour of this Last bit is much simpler than what you ended up with.

OK, I will respin according to your suggestion then.

Thanks

Eric

Quoted 5 lines

Thanks,

	M.

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

maintainer
#14 lore raw

Re: [PATCH v4 7/8] KVM: arm64: vgic-v3: Expose GICR_TYPER.Last for userspace

Hi Eric,

Quoted 148 lines · Auger Eric

Hi Marc,

Quoted 144 lines · Marc Zyngier
Quoted 129 lines · Auger Eric

Hi Marc,

Quoted 113 lines · Marc Zyngier

Hi Eric,

Quoted 15 lines · Eric Auger

Commit 23bde34771f1 ("KVM: arm64: vgic-v3: Drop the reporting of GICR_TYPER.Last for userspace") temporarily fixed a bug identified when attempting to access the GICR_TYPER register before the redistributor region setting, but dropped the support of the LAST bit.

Emulating the GICR_TYPER.Last bit still makes sense for architecture compliance though. This patch restores its support (if the redistributor region was set) while keeping the code safe.

We introduce a new helper, vgic_mmio_vcpu_rdist_is_last() which computes whether a redistributor is the highest one of a series of redistributor contributor pages.

Quoted 5 lines

The spec says "Indicates whether this Redistributor is the highest-numbered Redistributor in a series of contiguous Redistributor pages."

The code is a bit convulated since there is no guarantee

nit: convoluted

Quoted 5 lines

redistributors are added in a given reditributor region in ascending order. In that case the current implementation was wrong. Also redistributor regions can be contiguous and registered in non increasing base address order.

Quoted 7 lines

So the index of redistributors are stored in an array within the redistributor region structure.

With this new implementation we do not need to have a uaccess read accessor anymore.

This patch also hurt my head, a lot more than the first one. See below.

Quoted 12 lines
---
 arch/arm64/kvm/vgic/vgic-init.c    |  7 +--
 arch/arm64/kvm/vgic/vgic-mmio-v3.c | 97 ++++++++++++++++++++----------
 arch/arm64/kvm/vgic/vgic.h         |  1 +
 include/kvm/arm_vgic.h             |  3 +
 4 files changed, 73 insertions(+), 35 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -0
index cf6faa0aeddb2..61150c34c268c 100644

arch/arm64/kvm/vgic/vgic-init.c int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu) · line 193

Quoted 4 lines
arch/arm64/kvm/vgic/vgic-init.c +1 -0
@@ -190,6 +190,7 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
 	int i;
 
 	vgic_cpu->rd_iodev.base_addr = VGIC_ADDR_UNDEF;
+	vgic_cpu->index = vcpu->vcpu_id;

Is it so that vgic_cpu->index is always equal to vcpu_id? If so, why do we need another field? We can always get to the vcpu using a container_of().

arch/arm64/kvm/vgic/vgic-init.c static void kvm_vgic_dist_destroy(struct kvm *kvm) · line 343

Quoted 13 lines
arch/arm64/kvm/vgic/vgic-init.c +2 -4
 	INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
 	raw_spin_lock_init(&vgic_cpu->ap_list_lock);
@@ -338,10 +339,8 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm)
 	dist->vgic_dist_base = VGIC_ADDR_UNDEF;
 
 	if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
-		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list) {
-			list_del(&rdreg->list);
-			kfree(rdreg);
-		}
+		list_for_each_entry_safe(rdreg, next, &dist->rd_regions, list)
+			vgic_v3_free_redist_region(rdreg);

Consider moving the introduction of vgic_v3_free_redist_region() into a separate patch. On its own, that's a good readability improvement.

Quoted 21 lines
arch/arm64/kvm/vgic/vgic-init.c +0 -0
 		INIT_LIST_HEAD(&dist->rd_regions);
 	} else {
 		dist->vgic_cpu_base = VGIC_ADDR_UNDEF;
arch/arm64/kvm/vgic/vgic-mmio-v3.c +10 -0
index 987e366c80008..f6a7eed1d6adb 100644
@@ -251,45 +251,57 @@ static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu,
 		vgic_enable_lpis(vcpu);
 }
 
+static bool vgic_mmio_vcpu_rdist_is_last(struct kvm_vcpu *vcpu)
+{
+	struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
+	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+	struct vgic_redist_region *rdreg = vgic_cpu->rdreg;
+
+	if (!rdreg)
+		return false;
+
+	if (rdreg->count && vgic_cpu->rdreg_index == (rdreg->count - 1)) {

arch/arm64/kvm/vgic/vgic-mmio-v3.c static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu, · line 273

Quoted 10 lines
arch/arm64/kvm/vgic/vgic-mmio-v3.c +10 -0
+		/* check whether there is no other contiguous rdist region */
+		struct list_head *rd_regions = &vgic->rd_regions;
+		struct vgic_redist_region *iter;
+
+		list_for_each_entry(iter, rd_regions, list) {
+			if (iter->base == rdreg->base + rdreg->count * KVM_VGIC_V3_REDIST_SIZE &&
+				iter->free_index > 0) {
+			/* check the first rdist index of this region, if any */
+				if (vgic_cpu->index < iter->rdist_indices[0])
+					return false;
Quoted 7 lines

rdist_indices[] contains the vcpu_id of the vcpu associated with a given RD in the region. At this stage, you have established that there is another region that is contiguous with the one associated with our vcpu. You also know that this adjacent region has a vcpu mapped in (free_index isn't 0). Isn't that enough to declare that our vcpu isn't last? I definitely don't understand what the index comparison does here.

Assume the following case:
2 RDIST region
region #0 contains rdist 1, 2, 4
region #1, adjacent to #0 contains rdist 3

Quoted 6 lines

Spec days:
"Indicates whether this Redistributor is the highest-numbered Redistributor in a series of contiguous Redistributor pages."

To me 4 is last and 3 is last too.

No, only 3 is last, assuming that region 0 is full. I think the phrasing in the spec is just really bad. What this describes is that at the end of a set of contiguous set of RDs, that last RD has Last set. If two regions are contiguous, that's undistinguishable from a single, larger region.

Quoted 6 lines
Quoted 3 lines

There is no such thing as a "redistributor number" anyway. The closest thing there is would be "processor number", but that has nothing to do with the RD itself.

Hum OK. That's a different understanding of the spec wording indeed. For me redistributor number was the index of the vcpu.

I think that's the source of the confusion. There really is nothing like a redistributor number. There is a processor number when GICR_TYPER.PTA=0 (that the guest uses as the target CPU when moving a LPI), but that's it. The layout is totally dumb, and the last frame in a contiguous sequence of frames is, well, last. The content of the frames doesn't matter in the least.

Quoted 2 lines

But well, you're understanding is definitively simpler to implement and also matches what was implemented for single RDIST region.

That's a key insight. There is no reason why the RD layout would defer between a single region and multiple regions.

Think of it from a HW perspective. You design a SoC that has "clusters" of CPUs, and you lay down a bunch of RDs, one set per cluster. Each set has a "Last" RD frame, and that's all there is to it.

I'll try and see if ARM people are willing to clarify the spec (for which an update is long overdue).

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
author
#15 lore raw

PATCH v4 8/8 KVM: selftests: aarch64/vgic-v3 init sequence tests

In reply to Eric Auger

The tests exercise the VGIC_V3 device creation including the associated KVM_DEV_ARM_VGIC_GRP_ADDR group attributes:

- KVM_VGIC_V3_ADDR_TYPE_DIST/REDIST
- KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION

Some other tests dedicate to KVM_DEV_ARM_VGIC_GRP_REDIST_REGS group and especially the GICR_TYPER read. The goal was to test the case recently fixed by commit 23bde34771f1
("KVM: arm64: vgic-v3: Drop the reporting of GICR_TYPER.Last for userspace").

The API under test can be found at
Documentation/virt/kvm/devices/arm-vgic-v3.rst

---

v3 -> v4:
- update .gitignore
- More vgic-mmio-v3.c change into the previous patch
- rename fuzz_dist_rdist into test_dist_rdist
- cleanup in run_vcpu and guest_code
- max_ipa_bits is global
- s/fuzz/subtest
- added test_kvm_device,
- moved ucall_init() just before the cpu run
- use vm_create_default_with_vcpus
- use vm_gic struct, vm_gic_create, vm_gic_destroy
- revwrite util.c helpers to comply with the usual style

---
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../testing/selftests/kvm/aarch64/vgic_init.c | 652 ++++++++++++++++++
 .../testing/selftests/kvm/include/kvm_util.h  |   9 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |  77 +++
 5 files changed, 740 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/aarch64/vgic_init.c
tools/testing/selftests/kvm/.gitignore +1 -0
index 7bd7e776c266a..bb862f91f6409 100644
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 /aarch64/get-reg-list
 /aarch64/get-reg-list-sve
+/aarch64/vgic_init
 /s390x/memop
 /s390x/resets
 /s390x/sync_regs_test
tools/testing/selftests/kvm/Makefile +1 -0
index 67eebb53235fd..2fd4801de9ca7 100644
@@ -78,6 +78,7 @@ TEST_GEN_PROGS_x86_64 += steal_time
 
 TEST_GEN_PROGS_aarch64 += aarch64/get-reg-list
 TEST_GEN_PROGS_aarch64 += aarch64/get-reg-list-sve
+TEST_GEN_PROGS_aarch64 += aarch64/vgic_init
 TEST_GEN_PROGS_aarch64 += demand_paging_test
 TEST_GEN_PROGS_aarch64 += dirty_log_test
 TEST_GEN_PROGS_aarch64 += dirty_log_perf_test
tools/testing/selftests/kvm/aarch64/vgic_init.c new file +652 -0

652 lines not on this page · raw message

tools/testing/selftests/kvm/include/kvm_util.h +9 -0
index 0f4258eaa629e..2b4b325cde010 100644
@@ -225,6 +225,15 @@ int vcpu_nested_state_set(struct kvm_vm *vm, uint32_t vcpuid,
 #endif
 void *vcpu_map_dirty_ring(struct kvm_vm *vm, uint32_t vcpuid);
 
+int _kvm_device_check_attr(int dev_fd, uint32_t group, uint64_t attr);
+int kvm_device_check_attr(int dev_fd, uint32_t group, uint64_t attr);
+int _kvm_create_device(struct kvm_vm *vm, uint64_t type, bool test);
+int kvm_create_device(struct kvm_vm *vm, uint64_t type, bool test);
+int _kvm_device_access(int dev_fd, uint32_t group, uint64_t attr,
+		       void *val, bool write);
+int kvm_device_access(int dev_fd, uint32_t group, uint64_t attr,
+		      void *val, bool write);
+
 const char *exit_reason_str(unsigned int exit_reason);
 
 void virt_pgd_alloc(struct kvm_vm *vm, uint32_t pgd_memslot);
tools/testing/selftests/kvm/lib/kvm_util.c +77 -0

83 lines not on this page · raw message

Signature · 6 lines
2.26.3

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm