Skip to contentSkip to conversation

KVM: irqchip: synchronize srcu only if needed

PATCH Superseded Yi Wang 3 participants 11 messages Jan 21, 2024 – Jan 24, 2024

Subsystems (5)
x86 KVM s390 KVM x86 +1 more subsystems

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Yi Wang

author
#0 lore raw

v2 cover KVM: irqchip: synchronize srcu only if needed

From: Yi Wang <[email protected]>

We found that it may cost more than 20 milliseconds very accidentally to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms already.

The reason is that when vmm(qemu/CloudHypervisor) invokes KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and might_sleep and kworker of srcu may cost some delay during this period. One way makes sence is setup empty irq routing when creating vm and so that x86/s390 don't need to setup empty/dummy irq routing.

Note: I have no s390 machine so the s390 patch has not been tested.

Changelog:
----------
v1->v2:
- setup empty irq routing in kvm_create_vm
- don't setup irq routing in x86 KVM_CAP_SPLIT_IRQCHIP
- don't setup irq routing in s390 KVM_CREATE_IRQCHIP

v1: https://lore.kernel.org/kvm/[email protected]/

Yi Wang (4):
KVM: irqchip: add setup empty irq routing function
KVM: setup empty irq routing when create vm
KVM: x86: don't setup empty irq routing when KVM_CAP_SPLIT_IRQCHIP
KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP

 arch/s390/kvm/kvm-s390.c | 13 +++++--------
 arch/x86/kvm/irq.h       |  1 -
 arch/x86/kvm/irq_comm.c  |  5 -----
 arch/x86/kvm/x86.c       |  3 ---
 include/linux/kvm_host.h |  1 +
 virt/kvm/irqchip.c       | 19 +++++++++++++++++++
 virt/kvm/kvm_main.c      |  4 ++++
 7 files changed, 29 insertions(+), 17 deletions(-)
Signature · 1 line
2.39.3
author
#1 lore raw

PATCH v2 1/4 KVM: irqchip: add setup empty irq routing function

Add a new function to setup empty irq routing in kvm path, which can be invoded in non-architecture-specific functions. The difference compared to the kvm_setup_empty_irq_routing() is this function just alloc the empty irq routing and does not need synchronize srcu, as we will call it in kvm_create_vm().

This patch is a preparatory step for an upcoming patch to avoid delay in KVM_CAP_SPLIT_IRQCHIP ioctl.

---
 include/linux/kvm_host.h |  1 +
 virt/kvm/irqchip.c       | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+)
include/linux/kvm_host.h +1 -0
index 4944136efaa2..e91525c0a4ea 100644
@@ -2000,6 +2000,7 @@ int kvm_set_irq_routing(struct kvm *kvm,
 			const struct kvm_irq_routing_entry *entries,
 			unsigned nr,
 			unsigned flags);
+int kvm_setup_empty_irq_routing_lockless(struct kvm *kvm);
 int kvm_set_routing_entry(struct kvm *kvm,
 			  struct kvm_kernel_irq_routing_entry *e,
 			  const struct kvm_irq_routing_entry *ue);
virt/kvm/irqchip.c +19 -0
index 1e567d1f6d3d..90fc43bd0fe4 100644
@@ -237,3 +237,22 @@ int kvm_set_irq_routing(struct kvm *kvm,
 
 	return r;
 }
+
+int kvm_setup_empty_irq_routing_lockless(struct kvm *kvm)
+{
+	struct kvm_irq_routing_table *new;
+	u32 i, j;
+
+	new = kzalloc(struct_size(new, map, 1), GFP_KERNEL_ACCOUNT);
+	if (!new)
+		return -ENOMEM;
+
+	new->nr_rt_entries = 1;
+	for (i = 0; i < KVM_NR_IRQCHIPS; i++)
+		for (j = 0; j < KVM_IRQCHIP_NUM_PINS; j++)
+			new->chip[i][j] = -1;
+
+	RCU_INIT_POINTER(kvm->irq_routing, new);
+
+	return 0;
+}
Signature · 1 line
2.39.3
#2 lore raw

Re: [v2 1/4] KVM: irqchip: add setup empty irq routing function

Quoted 8 lines · Yi Wang

Add a new function to setup empty irq routing in kvm path, which can be invoded in non-architecture-specific functions. The difference compared to the kvm_setup_empty_irq_routing() is this function just alloc the empty irq routing and does not need synchronize srcu, as we will call it in kvm_create_vm().

This patch is a preparatory step for an upcoming patch to avoid delay in KVM_CAP_SPLIT_IRQCHIP ioctl.

Adding a function in a separate patch from its callsites is never useful. Please squash this into the second patch.

Signature · 2 lines
Thanks,
Oliver
author
#3 lore raw

Re: [v2 1/4] KVM: irqchip: add setup empty irq routing function

Quoted 8 lines · Oliver Upton
Quoted 6 lines · Yi Wang

Add a new function to setup empty irq routing in kvm path, which can be invoded in non-architecture-specific functions. The difference compared to the kvm_setup_empty_irq_routing() is this function just alloc the empty irq routing and does not need synchronize srcu, as we will call it in kvm_create_vm().

Quoted 5 lines
Quoted 2 lines

This patch is a preparatory step for an upcoming patch to avoid delay in KVM_CAP_SPLIT_IRQCHIP ioctl.

Adding a function in a separate patch from its callsites is never useful. Please squash this into the second patch.

Thanks for your review and suggestion. I will update this patch ASAP.

Quoted 4 lines

--
Thanks,
Oliver

Signature · 3 lines
---
Best wishes
Yi Wang
author
#4 lore raw

PATCH v2 2/4 KVM: setup empty irq routing when create vm

In reply to Yi Wang

Setup empty irq routing when kvm_create_vm(), so that x86 and s390 no longer need to set empty/dummy irq routing when creating an IRQCHIP 'cause it avoid an synchronize_srcu.

---
 virt/kvm/kvm_main.c | 4 ++++
 1 file changed, 4 insertions(+)
virt/kvm/kvm_main.c +4 -0
index 7db96875ac46..db1b13fc0502 100644
@@ -1242,6 +1242,10 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
 	if (r)
 		goto out_err;
 
+	r = kvm_setup_empty_irq_routing_lockless(kvm);
+	if (r)
+		goto out_err;
+
 	mutex_lock(&kvm_lock);
 	list_add(&kvm->vm_list, &vm_list);
 	mutex_unlock(&kvm_lock);
Signature · 1 line
2.39.3
author
#5 lore raw

PATCH v2 3/4 KVM: x86: don't setup empty irq routing when KVM_CAP_SPLIT_IRQCHIP

In reply to Yi Wang

We found that it may cost more than 20 milliseconds very accidentally to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms already.

The reason is that when vmm(qemu/CloudHypervisor) invokes KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and might_sleep and kworker of srcu may cost some delay during this period.

As we have set up empty irq routing when creating vm, so this is no need now.

---
 arch/x86/kvm/irq.h      | 1 -
 arch/x86/kvm/irq_comm.c | 5 -----
 arch/x86/kvm/x86.c      | 3 ---
 3 files changed, 9 deletions(-)
arch/x86/kvm/irq.h +0 -1
index c2d7cfe82d00..76d46b2f41dd 100644
@@ -106,7 +106,6 @@ void __kvm_migrate_timers(struct kvm_vcpu *vcpu);
 int apic_has_pending_timer(struct kvm_vcpu *vcpu);
 
 int kvm_setup_default_irq_routing(struct kvm *kvm);
-int kvm_setup_empty_irq_routing(struct kvm *kvm);
 int kvm_irq_delivery_to_apic(struct kvm *kvm, struct kvm_lapic *src,
 			     struct kvm_lapic_irq *irq,
 			     struct dest_map *dest_map);
arch/x86/kvm/irq_comm.c +0 -5
index 16d076a1b91a..99bf53b94175 100644
@@ -392,11 +392,6 @@ int kvm_setup_default_irq_routing(struct kvm *kvm)
 
 static const struct kvm_irq_routing_entry empty_routing[] = {};
 
-int kvm_setup_empty_irq_routing(struct kvm *kvm)
-{
-	return kvm_set_irq_routing(kvm, empty_routing, 0, 0);
-}
-
 void kvm_arch_post_irq_routing_update(struct kvm *kvm)
 {
 	if (!irqchip_split(kvm))
arch/x86/kvm/x86.c +0 -3
index cec0fc2a4b1c..6a2e786aca22 100644
@@ -6414,9 +6414,6 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 			goto split_irqchip_unlock;
 		if (kvm->created_vcpus)
 			goto split_irqchip_unlock;
-		r = kvm_setup_empty_irq_routing(kvm);
-		if (r)
-			goto split_irqchip_unlock;
 		/* Pairs with irqchip_in_kernel. */
 		smp_wmb();
 		kvm->arch.irqchip_mode = KVM_IRQCHIP_SPLIT;
Signature · 1 line
2.39.3
author
#6 lore raw

PATCH v2 4/4 KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP

In reply to Yi Wang

As we have setup empty irq routing in kvm_create_vm(), there's no need to setup dummy routing when KVM_CREATE_IRQCHIP.

---
 arch/s390/kvm/kvm-s390.c | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)
arch/s390/kvm/kvm-s390.c +5 -8
index acc81ca6492e..7c836c973b75 100644
@@ -2999,14 +2999,11 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
 		break;
 	}
 	case KVM_CREATE_IRQCHIP: {
-		struct kvm_irq_routing_entry routing;
-
-		r = -EINVAL;
-		if (kvm->arch.use_irqchip) {
-			/* Set up dummy routing. */
-			memset(&routing, 0, sizeof(routing));
-			r = kvm_set_irq_routing(kvm, &routing, 0, 0);
-		}
+		/*
+		 * As we have set up empty routing, there is no need to
+		 * setup dummy routing here.
+		 */
+		r = 0;
 		break;
 	}
 	case KVM_SET_DEVICE_ATTR: {
Signature · 1 line
2.39.3
#7 lore raw

Re: [v2 4/4] KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP

Quoted 18 lines · Yi Wang

As we have setup empty irq routing in kvm_create_vm(), there's no need to setup dummy routing when KVM_CREATE_IRQCHIP.

---
 arch/s390/kvm/kvm-s390.c | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)
arch/s390/kvm/kvm-s390.c +0 -2
index acc81ca6492e..7c836c973b75 100644
@@ -2999,14 +2999,11 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
 		break;
 	}
 	case KVM_CREATE_IRQCHIP: {
-		struct kvm_irq_routing_entry routing;
-

arch/s390/kvm/kvm-s390.c int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) · line 3005

Quoted 10 lines
arch/s390/kvm/kvm-s390.c +4 -6
-		r = -EINVAL;
-		if (kvm->arch.use_irqchip) {
-			/* Set up dummy routing. */
-			memset(&routing, 0, sizeof(routing));
-			r = kvm_set_irq_routing(kvm, &routing, 0, 0);
-		}
+		/*
+		 * As we have set up empty routing, there is no need to
+		 * setup dummy routing here.
+		 */

Where exactly?

In the context of this patch series it is rather obvious, but this comment does not stand on its own. You can either throw the reader a bone by mentioning where the dummy routing is created or just drop the comment altogether.

Quoted 7 lines
arch/s390/kvm/kvm-s390.c +1 -0
+		r = 0;
 		break;
 	}
 	case KVM_SET_DEVICE_ATTR: {
Signature · 1 line
2.39.3
Signature · 2 lines
Thanks,
Oliver
author
#8 lore raw

Re: [v2 4/4] KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP

Quoted 33 lines · Oliver Upton
Quoted 18 lines · Yi Wang

As we have setup empty irq routing in kvm_create_vm(), there's no need to setup dummy routing when KVM_CREATE_IRQCHIP.

---
 arch/s390/kvm/kvm-s390.c | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)
arch/s390/kvm/kvm-s390.c +0 -2
index acc81ca6492e..7c836c973b75 100644
@@ -2999,14 +2999,11 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
              break;
      }
      case KVM_CREATE_IRQCHIP: {
-             struct kvm_irq_routing_entry routing;
-

arch/s390/kvm/kvm-s390.c int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) · line 3005

Quoted 10 lines
arch/s390/kvm/kvm-s390.c +4 -6
-             r = -EINVAL;
-             if (kvm->arch.use_irqchip) {
-                     /* Set up dummy routing. */
-                     memset(&routing, 0, sizeof(routing));
-                     r = kvm_set_irq_routing(kvm, &routing, 0, 0);
-             }
+             /*
+              * As we have set up empty routing, there is no need to
+              * setup dummy routing here.
+              */

Where exactly?

Quoted 4 lines

In the context of this patch series it is rather obvious, but this comment does not stand on its own. You can either throw the reader a bone by mentioning where the dummy routing is created or just drop the comment altogether.

Yeap, you are right. I will drop this in the upcoming patch.

Quoted 12 lines
Quoted 8 lines
arch/s390/kvm/kvm-s390.c +1 -1
+             r = 0;
              break;
      }
      case KVM_SET_DEVICE_ATTR: {
--

2.39.3

--
Thanks,
Oliver

Signature · 3 lines
---
Best wishes
Yi Wang
#9 lore raw
In reply to Yi Wang

Am 21.01.24 um 12:17 schrieb Yi Wang:

Quoted 6 lines

From: Yi Wang <[email protected]>

We found that it may cost more than 20 milliseconds very accidentally to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms already.

Quoted 7 lines

The reason is that when vmm(qemu/CloudHypervisor) invokes KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and might_sleep and kworker of srcu may cost some delay during this period. One way makes sence is setup empty irq routing when creating vm and so that x86/s390 don't need to setup empty/dummy irq routing.

Note: I have no s390 machine so the s390 patch has not been tested.

I just did a quick sniff and it still seems to work. No performance check etc.

author
#10 lore raw
Quoted 14 lines · Christian Borntraeger

Am 21.01.24 um 12:17 schrieb Yi Wang:

Quoted 12 lines

From: Yi Wang <[email protected]>

We found that it may cost more than 20 milliseconds very accidentally to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms already.

The reason is that when vmm(qemu/CloudHypervisor) invokes KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and might_sleep and kworker of srcu may cost some delay during this period. One way makes sence is setup empty irq routing when creating vm and so that x86/s390 don't need to setup empty/dummy irq routing.

Quoted 3 lines
Quoted 1 line

Note: I have no s390 machine so the s390 patch has not been tested.

I just did a quick sniff and it still seems to work. No performance check etc.

Thanks very much, Christian!

---
Best wishes
Yi Wang