Skip to contentSkip to conversation

KVM: PPC: Book3S HV: XIVE: assorted fixes on vCPU and RAM limits

PATCH In review Cédric Le Goater 4 participants 12 messages May 20, 2019 – May 31, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover KVM: PPC: Book3S HV: XIVE: assorted fixes on vCPU and RAM limits

Hello,

Here are a couple of fixes for issues in the XIVE KVM device when testing the limits : RAM size and number of vCPUS.

Based on 5.2-rc1.

Available on GitHub:

https://github.com/legoater/linux/commits/xive-5.2

Thanks,

C.

Cédric Le Goater (3):
KVM: PPC: Book3S HV: XIVE: clear file mapping when device is released
KVM: PPC: Book3S HV: XIVE: do not test the EQ flag validity when
reseting
KVM: PPC: Book3S HV: XIVE: fix the enforced limit on the vCPU
identifier

 arch/powerpc/kvm/book3s_xive_native.c | 46 ++++++++++++++++-----------
 1 file changed, 27 insertions(+), 19 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/3 KVM: PPC: Book3S HV: XIVE: clear file mapping when device is released

Improve the release of the XIVE KVM device by clearing the file address_space which is used to unmap the interrupt ESB pages when a device is passed-through.

---
 arch/powerpc/kvm/book3s_xive_native.c | 8 ++++++++
 1 file changed, 8 insertions(+)
arch/powerpc/kvm/book3s_xive_native.c +8 -0
index 6a8e698c4b6e..796d86549cfe 100644
@@ -979,6 +979,14 @@ static void kvmppc_xive_native_release(struct kvm_device *dev)
 
 	pr_devel("Releasing xive native device\n");
 
+	/*
+	 * Clear the KVM device file address_space which is used to
+	 * unmap the ESB pages when a device is passed-through.
+	 */
+	mutex_lock(&xive->mapping_lock);
+	xive->mapping = NULL;
+	mutex_unlock(&xive->mapping_lock);
+
 	/*
 	 * Clearing mmu_ready temporarily while holding kvm->lock
 	 * is a way of ensuring that no vcpus can enter the guest
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/3 KVM: PPC: Book3S HV: XIVE: do not test the EQ flag validity when resetting

In reply to Cédric Le Goater

When a CPU is hot-unplugged, the EQ is deconfigured using a zero size and a zero address. In this case, there is no need to check the flag and queue size validity. Move the checks after the queue reset code section to fix CPU hot-unplug.

---
 arch/powerpc/kvm/book3s_xive_native.c | 36 +++++++++++++--------------
 1 file changed, 18 insertions(+), 18 deletions(-)
arch/powerpc/kvm/book3s_xive_native.c +18 -18
index 796d86549cfe..3fdea6bf4e97 100644
@@ -565,24 +565,6 @@ static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
 		 __func__, server, priority, kvm_eq.flags,
 		 kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
 
-	/*
-	 * sPAPR specifies a "Unconditional Notify (n) flag" for the
-	 * H_INT_SET_QUEUE_CONFIG hcall which forces notification
-	 * without using the coalescing mechanisms provided by the
-	 * XIVE END ESBs. This is required on KVM as notification
-	 * using the END ESBs is not supported.
-	 */
-	if (kvm_eq.flags != KVM_XIVE_EQ_ALWAYS_NOTIFY) {
-		pr_err("invalid flags %d\n", kvm_eq.flags);
-		return -EINVAL;
-	}
-
-	rc = xive_native_validate_queue_size(kvm_eq.qshift);
-	if (rc) {
-		pr_err("invalid queue size %d\n", kvm_eq.qshift);
-		return rc;
-	}
-
 	/* reset queue and disable queueing */
 	if (!kvm_eq.qshift) {
 		q->guest_qaddr  = 0;
@@ -604,6 +586,24 @@ static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
 		return 0;
 	}
 
+	/*
+	 * sPAPR specifies a "Unconditional Notify (n) flag" for the
+	 * H_INT_SET_QUEUE_CONFIG hcall which forces notification
+	 * without using the coalescing mechanisms provided by the
+	 * XIVE END ESBs. This is required on KVM as notification
+	 * using the END ESBs is not supported.
+	 */
+	if (kvm_eq.flags != KVM_XIVE_EQ_ALWAYS_NOTIFY) {
+		pr_err("invalid flags %d\n", kvm_eq.flags);
+		return -EINVAL;
+	}
+
+	rc = xive_native_validate_queue_size(kvm_eq.qshift);
+	if (rc) {
+		pr_err("invalid queue size %d\n", kvm_eq.qshift);
+		return rc;
+	}
+
 	if (kvm_eq.qaddr & ((1ull << kvm_eq.qshift) - 1)) {
 		pr_err("queue page is not aligned %llx/%llx\n", kvm_eq.qaddr,
 		       1ull << kvm_eq.qshift);
Signature · 1 line
2.20.1
Reviewed-by on v1 2/3
#3 lore raw

Re: [PATCH 2/3] KVM: PPC: Book3S HV: XIVE: do not test the EQ flag validity when resetting

Quoted 9 lines · On Mon, 20 May 2019 09:15:13 +0200 Cédric Le Goater <[email protected]> wrote:

When a CPU is hot-unplugged, the EQ is deconfigured using a zero size and a zero address. In this case, there is no need to check the flag and queue size validity. Move the checks after the queue reset code section to fix CPU hot-unplug.

---

Quoted 57 lines
 arch/powerpc/kvm/book3s_xive_native.c | 36 +++++++++++++--------------
 1 file changed, 18 insertions(+), 18 deletions(-)
arch/powerpc/kvm/book3s_xive_native.c +18 -18
index 796d86549cfe..3fdea6bf4e97 100644
@@ -565,24 +565,6 @@ static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
 		 __func__, server, priority, kvm_eq.flags,
 		 kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
 
-	/*
-	 * sPAPR specifies a "Unconditional Notify (n) flag" for the
-	 * H_INT_SET_QUEUE_CONFIG hcall which forces notification
-	 * without using the coalescing mechanisms provided by the
-	 * XIVE END ESBs. This is required on KVM as notification
-	 * using the END ESBs is not supported.
-	 */
-	if (kvm_eq.flags != KVM_XIVE_EQ_ALWAYS_NOTIFY) {
-		pr_err("invalid flags %d\n", kvm_eq.flags);
-		return -EINVAL;
-	}
-
-	rc = xive_native_validate_queue_size(kvm_eq.qshift);
-	if (rc) {
-		pr_err("invalid queue size %d\n", kvm_eq.qshift);
-		return rc;
-	}
-
 	/* reset queue and disable queueing */
 	if (!kvm_eq.qshift) {
 		q->guest_qaddr  = 0;
@@ -604,6 +586,24 @@ static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
 		return 0;
 	}
 
+	/*
+	 * sPAPR specifies a "Unconditional Notify (n) flag" for the
+	 * H_INT_SET_QUEUE_CONFIG hcall which forces notification
+	 * without using the coalescing mechanisms provided by the
+	 * XIVE END ESBs. This is required on KVM as notification
+	 * using the END ESBs is not supported.
+	 */
+	if (kvm_eq.flags != KVM_XIVE_EQ_ALWAYS_NOTIFY) {
+		pr_err("invalid flags %d\n", kvm_eq.flags);
+		return -EINVAL;
+	}
+
+	rc = xive_native_validate_queue_size(kvm_eq.qshift);
+	if (rc) {
+		pr_err("invalid queue size %d\n", kvm_eq.qshift);
+		return rc;
+	}
+
 	if (kvm_eq.qaddr & ((1ull << kvm_eq.qshift) - 1)) {
 		pr_err("queue page is not aligned %llx/%llx\n", kvm_eq.qaddr,
 		       1ull << kvm_eq.qshift);
#4 lore raw

PATCH 3/3 KVM: PPC: Book3S HV: XIVE: fix the enforced limit on the vCPU identifier

In reply to Cédric Le Goater

When a vCPU is connected to the KVM device, it is done using its vCPU identifier in the guest. Fix the enforced limit on the vCPU identifier by taking into account the SMT mode.

---
 arch/powerpc/kvm/book3s_xive_native.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/kvm/book3s_xive_native.c +1 -1
index 3fdea6bf4e97..25b6b0e2d02a 100644
@@ -109,7 +109,7 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
 		return -EPERM;
 	if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
 		return -EBUSY;
-	if (server_num >= KVM_MAX_VCPUS) {
+	if (server_num >= (KVM_MAX_VCPUS * vcpu->kvm->arch.emul_smt_mode)) {
 		pr_devel("Out of bounds !\n");
 		return -EINVAL;
 	}
Signature · 1 line
2.20.1
Reviewed-by on v1 3/3
#5 lore raw

Re: [PATCH 3/3] KVM: PPC: Book3S HV: XIVE: fix the enforced limit on the vCPU identifier

Quoted 4 lines · On Mon, 20 May 2019 09:15:14 +0200 Cédric Le Goater <[email protected]> wrote:

When a vCPU is connected to the KVM device, it is done using its vCPU identifier in the guest. Fix the enforced limit on the vCPU identifier by taking into account the SMT mode.

Quoted 20 lines
---
 arch/powerpc/kvm/book3s_xive_native.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/kvm/book3s_xive_native.c +1 -1
index 3fdea6bf4e97..25b6b0e2d02a 100644
@@ -109,7 +109,7 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
 		return -EPERM;
 	if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
 		return -EBUSY;
-	if (server_num >= KVM_MAX_VCPUS) {
+	if (server_num >= (KVM_MAX_VCPUS * vcpu->kvm->arch.emul_smt_mode)) {
 		pr_devel("Out of bounds !\n");
 		return -EINVAL;
 	}
#6 lore raw
In reply to Cédric Le Goater
Quoted 4 lines · Cédric Le Goater

Hello,

Here are a couple of fixes for issues in the XIVE KVM device when testing the limits : RAM size and number of vCPUS.

How serious are the problems these patches fix? I'm wondering if I need to make a backport for RHEL8.1.

Quoted 21 lines

Based on 5.2-rc1.

Available on GitHub:

https://github.com/legoater/linux/commits/xive-5.2

Thanks,

C.

Cédric Le Goater (3):
KVM: PPC: Book3S HV: XIVE: clear file mapping when device is released
KVM: PPC: Book3S HV: XIVE: do not test the EQ flag validity when
reseting
KVM: PPC: Book3S HV: XIVE: fix the enforced limit on the vCPU
identifier

 arch/powerpc/kvm/book3s_xive_native.c | 46 ++++++++++++++++-----------
 1 file changed, 27 insertions(+), 19 deletions(-)
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#7 lore raw
Quoted 9 lines · On Thu, 23 May 2019 09:30:43 +1000 David Gibson <[email protected]>…
Quoted 4 lines · Cédric Le Goater

Hello,

Here are a couple of fixes for issues in the XIVE KVM device when testing the limits : RAM size and number of vCPUS.

How serious are the problems these patches fix? I'm wondering if I need to make a backport for RHEL8.1.

Patch 2/3 fixes a QEMU error when hot-unplugging a vCPU:

qemu-system-ppc64: KVM_SET_DEVICE_ATTR failed: Group 4 attr 0x0000000000000046: Invalid argument

Patch 3/3 fixes an issue where the guest freezes at some point when doing vCPU hot-plug/unplug in a loop.

Both issues have a BZ at IBM. They can be mirrored to RH if needed.

Quoted 22 lines
Quoted 21 lines

Based on 5.2-rc1.

Available on GitHub:

https://github.com/legoater/linux/commits/xive-5.2

Thanks,

C.

Cédric Le Goater (3):
KVM: PPC: Book3S HV: XIVE: clear file mapping when device is released
KVM: PPC: Book3S HV: XIVE: do not test the EQ flag validity when
reseting
KVM: PPC: Book3S HV: XIVE: fix the enforced limit on the vCPU
identifier

 arch/powerpc/kvm/book3s_xive_native.c | 46 ++++++++++++++++-----------
 1 file changed, 27 insertions(+), 19 deletions(-)
#8 lore raw
Quoted 13 lines · Greg Kurz
Quoted 9 lines · On Thu, 23 May 2019 09:30:43 +1000 David Gibson <[email protected]>…
Quoted 4 lines · Cédric Le Goater

Hello,

Here are a couple of fixes for issues in the XIVE KVM device when testing the limits : RAM size and number of vCPUS.

How serious are the problems these patches fix? I'm wondering if I need to make a backport for RHEL8.1.

Quoted 7 lines

Patch 2/3 fixes a QEMU error when hot-unplugging a vCPU:

qemu-system-ppc64: KVM_SET_DEVICE_ATTR failed: Group 4 attr 0x0000000000000046: Invalid argument

Patch 3/3 fixes an issue where the guest freezes at some point when doing vCPU hot-plug/unplug in a loop.

Oh.. weird. It's not clear to me how it would do that.

Quoted 1 line

Both issues have a BZ at IBM. They can be mirrored to RH if needed.

That would be helpful, thanks.

Quoted 24 lines
Quoted 23 lines
Quoted 22 lines

Based on 5.2-rc1.

Available on GitHub:

https://github.com/legoater/linux/commits/xive-5.2

Thanks,

C.

Cédric Le Goater (3):
KVM: PPC: Book3S HV: XIVE: clear file mapping when device is released
KVM: PPC: Book3S HV: XIVE: do not test the EQ flag validity when
reseting
KVM: PPC: Book3S HV: XIVE: fix the enforced limit on the vCPU
identifier

 arch/powerpc/kvm/book3s_xive_native.c | 46 ++++++++++++++++-----------
 1 file changed, 27 insertions(+), 19 deletions(-)
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#9 lore raw
Quoted 16 lines · On Thu, 23 May 2019 16:27:15 +1000 David Gibson <[email protected]>…
Quoted 15 lines · Greg Kurz
Quoted 9 lines · On Thu, 23 May 2019 09:30:43 +1000 David Gibson <[email protected]>…
Quoted 4 lines · Cédric Le Goater

Hello,

Here are a couple of fixes for issues in the XIVE KVM device when testing the limits : RAM size and number of vCPUS.

How serious are the problems these patches fix? I'm wondering if I need to make a backport for RHEL8.1.

Patch 2/3 fixes a QEMU error when hot-unplugging a vCPU:

Quoted 8 lines
Quoted 5 lines

qemu-system-ppc64: KVM_SET_DEVICE_ATTR failed: Group 4 attr 0x0000000000000046: Invalid argument

Patch 3/3 fixes an issue where the guest freezes at some point when doing vCPU hot-plug/unplug in a loop.

Oh.. weird. It's not clear to me how it would do that.

Cedric provided a better description in some other mail: guest with 1024 vCPUs.

Quoted 4 lines
Quoted 1 line

Both issues have a BZ at IBM. They can be mirrored to RH if needed.

That would be helpful, thanks.

Ok, I'll take care of that.

Quoted 30 lines
Quoted 27 lines
Quoted 26 lines
Quoted 25 lines

Based on 5.2-rc1.

Available on GitHub:

https://github.com/legoater/linux/commits/xive-5.2

Thanks,

C.

Cédric Le Goater (3):
KVM: PPC: Book3S HV: XIVE: clear file mapping when device is released
KVM: PPC: Book3S HV: XIVE: do not test the EQ flag validity when
reseting
KVM: PPC: Book3S HV: XIVE: fix the enforced limit on the vCPU
identifier

 arch/powerpc/kvm/book3s_xive_native.c | 46 ++++++++++++++++-----------
 1 file changed, 27 insertions(+), 19 deletions(-)
#10 lore raw
In reply to David Gibson
Quoted 5 lines · David Gibson
Quoted 4 lines · Cédric Le Goater

Hello,

Here are a couple of fixes for issues in the XIVE KVM device when testing the limits : RAM size and number of vCPUS.

This summary is wrong. RAM size was fixed in QEMU.

Quoted 2 lines

How serious are the problems these patches fix? I'm wondering if I need to make a backport for RHEL8.1.

Patch 1 is a cleanup patch. It does not fix any critical issues.

Patch 2 fixes CPU hotplug. The test on the EQ flag is at the wrong place :/ This is important I think.

Patch 3 fixes the maximum number of vCPUS supported. This one is less important maybe, unless we want to run a guest with 1024 vCPUs. Which is quite slow to run on most P9 systems.

QEMU emits a warning :

warning: Number of SMP cpus requested (1024) exceeds the recommended cpus supported by KVM (120)

May be we should refuse to run QEMU when that number is above a certain threshold ?

C.

Quoted 22 lines
Quoted 21 lines

Based on 5.2-rc1.

Available on GitHub:

https://github.com/legoater/linux/commits/xive-5.2

Thanks,

C.

Cédric Le Goater (3):
KVM: PPC: Book3S HV: XIVE: clear file mapping when device is released
KVM: PPC: Book3S HV: XIVE: do not test the EQ flag validity when
reseting
KVM: PPC: Book3S HV: XIVE: fix the enforced limit on the vCPU
identifier

 arch/powerpc/kvm/book3s_xive_native.c | 46 ++++++++++++++++-----------
 1 file changed, 27 insertions(+), 19 deletions(-)
#11 lore raw
In reply to Cédric Le Goater
Quoted 13 lines · Cédric Le Goater

Hello,

Here are a couple of fixes for issues in the XIVE KVM device when testing the limits : RAM size and number of vCPUS.

Based on 5.2-rc1.

Available on GitHub:

https://github.com/legoater/linux/commits/xive-5.2

Thanks,

Quoted 8 lines

C.

Cédric Le Goater (3):
KVM: PPC: Book3S HV: XIVE: clear file mapping when device is released
KVM: PPC: Book3S HV: XIVE: do not test the EQ flag validity when
reseting
KVM: PPC: Book3S HV: XIVE: fix the enforced limit on the vCPU
identifier

Thanks, series applied to my kvm-ppc-fixes branch.

Paul.