KVM: irqchip: synchronize srcu only if needed
v2 cover KVM: irqchip: synchronize srcu only if needed
From: Yi Wang <[email protected]>
We found that it may cost more than 20 milliseconds very accidentally to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms already.
The reason is that when vmm(qemu/CloudHypervisor) invokes KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and might_sleep and kworker of srcu may cost some delay during this period. One way makes sence is setup empty irq routing when creating vm and so that x86/s390 don't need to setup empty/dummy irq routing.
Note: I have no s390 machine so the s390 patch has not been tested.
Changelog:
----------v1->v2:
- setup empty irq routing in kvm_create_vm
- don't setup irq routing in x86 KVM_CAP_SPLIT_IRQCHIP
- don't setup irq routing in s390 KVM_CREATE_IRQCHIP
v1: https://lore.kernel.org/kvm/[email protected]/
Yi Wang (4):
KVM: irqchip: add setup empty irq routing function
KVM: setup empty irq routing when create vm
KVM: x86: don't setup empty irq routing when KVM_CAP_SPLIT_IRQCHIP
KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP
arch/s390/kvm/kvm-s390.c | 13 +++++-------- arch/x86/kvm/irq.h | 1 - arch/x86/kvm/irq_comm.c | 5 ----- arch/x86/kvm/x86.c | 3 --- include/linux/kvm_host.h | 1 + virt/kvm/irqchip.c | 19 +++++++++++++++++++ virt/kvm/kvm_main.c | 4 ++++ 7 files changed, 29 insertions(+), 17 deletions(-)
Signature · 1 line
2.39.3
PATCH v2 1/4 KVM: irqchip: add setup empty irq routing function
Add a new function to setup empty irq routing in kvm path, which can be invoded in non-architecture-specific functions. The difference compared to the kvm_setup_empty_irq_routing() is this function just alloc the empty irq routing and does not need synchronize srcu, as we will call it in kvm_create_vm().
This patch is a preparatory step for an upcoming patch to avoid delay in KVM_CAP_SPLIT_IRQCHIP ioctl.
--- include/linux/kvm_host.h | 1 + virt/kvm/irqchip.c | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+)
include/linux/kvm_host.h +1 -0
@@ -2000,6 +2000,7 @@ int kvm_set_irq_routing(struct kvm *kvm,const struct kvm_irq_routing_entry *entries, unsigned nr, unsigned flags); +int kvm_setup_empty_irq_routing_lockless (struct kvm *kvm);int kvm_set_routing_entry (struct kvm *kvm,struct kvm_kernel_irq_routing_entry *e,const struct kvm_irq_routing_entry *ue);
virt/kvm/irqchip.c +19 -0
@@ -237,3 +237,22 @@ int kvm_set_irq_routing(struct kvm *kvm,return r; } + +int kvm_setup_empty_irq_routing_lockless (struct kvm *kvm) +{ +struct kvm_irq_routing_table *new; +u32 i, j; + + new =kzalloc (struct_size (new, map,1 ),GFP_KERNEL_ACCOUNT ); +if (!new) +return -ENOMEM ; + + new->nr_rt_entries =1 ; +for (i =0 ; i <KVM_NR_IRQCHIPS ; i++) +for (j =0 ; j <KVM_IRQCHIP_NUM_PINS ; j++) + new->chip[i][j] =-1 ; + +RCU_INIT_POINTER (kvm->irq_routing, new); + +return 0 ; +}
Signature · 1 line
2.39.3
Re: [v2 1/4] KVM: irqchip: add setup empty irq routing function
Quoted 8 lines · Yi Wang
Add a new function to setup empty irq routing in kvm path, which can be invoded in non-architecture-specific functions. The difference compared to the
kvm_setup_empty_irq_routing()is this function just alloc the empty irq routing and does not need synchronize srcu, as we will call it inkvm_create_vm().This patch is a preparatory step for an upcoming patch to avoid delay in
KVM_CAP_SPLIT_IRQCHIPioctl.
Adding a function in a separate patch from its callsites is never useful. Please squash this into the second patch.
Signature · 2 lines
Thanks, Oliver
Re: [v2 1/4] KVM: irqchip: add setup empty irq routing function
Quoted 8 lines · Oliver Upton
Quoted 6 lines · Yi Wang
Add a new function to setup empty irq routing in kvm path, which can be invoded in non-architecture-specific functions. The difference compared to the
kvm_setup_empty_irq_routing()is this function just alloc the empty irq routing and does not need synchronize srcu, as we will call it inkvm_create_vm().
Quoted 5 lines
Quoted 2 lines
This patch is a preparatory step for an upcoming patch to avoid delay in
KVM_CAP_SPLIT_IRQCHIPioctl.Adding a function in a separate patch from its callsites is never useful. Please squash this into the second patch.
Thanks for your review and suggestion. I will update this patch ASAP.
Quoted 4 lines
--
Thanks,
Oliver
Signature · 3 lines
--- Best wishes Yi Wang
PATCH v2 2/4 KVM: setup empty irq routing when create vm
In reply to Yi WangSetup empty irq routing when kvm_create_vm(), so that x86 and s390 no longer need to set empty/dummy irq routing when creating an IRQCHIP 'cause it avoid an synchronize_srcu.
--- virt/kvm/kvm_main.c | 4 ++++ 1 file changed, 4 insertions(+)
virt/kvm/kvm_main.c +4 -0
@@ -1242,6 +1242,10 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)if (r) goto out_err; + r =kvm_setup_empty_irq_routing_lockless (kvm); +if (r) + goto out_err; +mutex_lock (&kvm_lock);list_add (&kvm->vm_list, &vm_list);mutex_unlock (&kvm_lock);
Signature · 1 line
2.39.3
PATCH v2 3/4 KVM: x86: don't setup empty irq routing when KVM_CAP_SPLIT_IRQCHIP
In reply to Yi WangWe found that it may cost more than 20 milliseconds very accidentally to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms already.
The reason is that when vmm(qemu/CloudHypervisor) invokes KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and might_sleep and kworker of srcu may cost some delay during this period.
As we have set up empty irq routing when creating vm, so this is no need now.
--- arch/x86/kvm/irq.h | 1 - arch/x86/kvm/irq_comm.c | 5 ----- arch/x86/kvm/x86.c | 3 --- 3 files changed, 9 deletions(-)
arch/x86/kvm/irq.h +0 -1
@@ -106,7 +106,6 @@ void __kvm_migrate_timers(struct kvm_vcpu *vcpu);int apic_has_pending_timer (struct kvm_vcpu *vcpu);int kvm_setup_default_irq_routing (struct kvm *kvm); -int kvm_setup_empty_irq_routing (struct kvm *kvm);int kvm_irq_delivery_to_apic (struct kvm *kvm,struct kvm_lapic *src,struct kvm_lapic_irq *irq,struct dest_map *dest_map);
arch/x86/kvm/irq_comm.c +0 -5
@@ -392,11 +392,6 @@ int kvm_setup_default_irq_routing(struct kvm *kvm)static const struct kvm_irq_routing_entry empty_routing[] = {}; -int kvm_setup_empty_irq_routing (struct kvm *kvm) -{ -return kvm_set_irq_routing (kvm, empty_routing,0 ,0 ); -} -void kvm_arch_post_irq_routing_update (struct kvm *kvm) {if (!irqchip_split (kvm))
arch/x86/kvm/x86.c +0 -3
@@ -6414,9 +6414,6 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, goto split_irqchip_unlock;if (kvm->created_vcpus) goto split_irqchip_unlock; - r =kvm_setup_empty_irq_routing (kvm); -if (r) - goto split_irqchip_unlock;/* Pairs with irqchip_in_kernel. */ smp_wmb (); kvm->arch.irqchip_mode =KVM_IRQCHIP_SPLIT ;
Signature · 1 line
2.39.3
PATCH v2 4/4 KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP
In reply to Yi WangAs we have setup empty irq routing in kvm_create_vm(), there's no need to setup dummy routing when KVM_CREATE_IRQCHIP.
--- arch/s390/kvm/kvm-s390.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-)
arch/s390/kvm/kvm-s390.c +5 -8
@@ -2999,14 +2999,11 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)break ; }case KVM_CREATE_IRQCHIP : { -struct kvm_irq_routing_entry routing; - - r = -EINVAL ; -if (kvm->arch.use_irqchip) { -/* Set up dummy routing. */ -memset (&routing,0 ,sizeof (routing)); - r =kvm_set_irq_routing (kvm, &routing,0 ,0 ); - } +/* +* As we have set up empty routing, there is no need to +* setup dummy routing here. +*/ + r =0 ;break ; }case KVM_SET_DEVICE_ATTR : {
Signature · 1 line
2.39.3
Re: [v2 4/4] KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP
Quoted 18 lines · Yi Wang
As we have setup empty irq routing in
kvm_create_vm(), there's no need to setup dummy routing whenKVM_CREATE_IRQCHIP.Signed-off-by Yi Wang <[email protected]>--- arch/s390/kvm/kvm-s390.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-)arch/s390/kvm/kvm-s390.c +0
-2@@ -2999,14 +2999,11 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)break ; }case KVM_CREATE_IRQCHIP : { -struct kvm_irq_routing_entry routing; -
arch/s390/kvm/kvm-s390.c int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) · line 3005
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +4
-6- r = -EINVAL ; -if (kvm->arch.use_irqchip) { -/* Set up dummy routing. */ -memset (&routing,0 ,sizeof (routing)); - r =kvm_set_irq_routing (kvm, &routing,0 ,0 ); - } +/* +* As we have set up empty routing, there is no need to +* setup dummy routing here. +*/
Where exactly?
In the context of this patch series it is rather obvious, but this comment does not stand on its own. You can either throw the reader a bone by mentioning where the dummy routing is created or just drop the comment altogether.
Quoted 7 lines
arch/s390/kvm/kvm-s390.c +1
-0+ r =0 ;break ; }case KVM_SET_DEVICE_ATTR : {Signature · 1 line
2.39.3
Signature · 2 lines
Thanks, Oliver
Re: [v2 4/4] KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP
Quoted 33 lines · Oliver Upton
Quoted 18 lines · Yi Wang
As we have setup empty irq routing in
kvm_create_vm(), there's no need to setup dummy routing whenKVM_CREATE_IRQCHIP.Signed-off-by Yi Wang <[email protected]>--- arch/s390/kvm/kvm-s390.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-)arch/s390/kvm/kvm-s390.c +0
-2@@ -2999,14 +2999,11 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)break ; }case KVM_CREATE_IRQCHIP : { -struct kvm_irq_routing_entry routing; -arch/s390/kvm/kvm-s390.c int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) · line 3005
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +4
-6- r = -EINVAL ; -if (kvm->arch.use_irqchip) { -/* Set up dummy routing. */ -memset (&routing,0 ,sizeof (routing)); - r =kvm_set_irq_routing (kvm, &routing,0 ,0 ); - } +/* +* As we have set up empty routing, there is no need to +* setup dummy routing here. +*/ Where exactly?
Quoted 4 lines
In the context of this patch series it is rather obvious, but this comment does not stand on its own. You can either throw the reader a bone by mentioning where the dummy routing is created or just drop the comment altogether.
Yeap, you are right. I will drop this in the upcoming patch.
Quoted 12 lines
Quoted 8 lines
arch/s390/kvm/kvm-s390.c +1
-1+ r =0 ;break ; }case KVM_SET_DEVICE_ATTR : { --2.39.3
--
Thanks,
Oliver
Signature · 3 lines
--- Best wishes Yi Wang
Christian Borntraeger
maintainerAm 21.01.24 um 12:17 schrieb Yi Wang:
Quoted 6 lines
From: Yi Wang <[email protected]>
We found that it may cost more than 20 milliseconds very accidentally to enable cap of
KVM_CAP_SPLIT_IRQCHIPon a host which has many vms already.
Quoted 7 lines
The reason is that when vmm(qemu/CloudHypervisor) invokes
KVM_CAP_SPLIT_IRQCHIPkvm will callsynchronize_srcu_expedited()andmight_sleepand kworker of srcu may cost some delay during this period. One way makes sence is setup empty irq routing when creating vm and so that x86/s390 don't need to setup empty/dummy irq routing.Note: I have no s390 machine so the s390 patch has not been tested.
I just did a quick sniff and it still seems to work. No performance check etc.
Quoted 14 lines · Christian Borntraeger
Am 21.01.24 um 12:17 schrieb Yi Wang:
Quoted 12 lines
From: Yi Wang <[email protected]>
We found that it may cost more than 20 milliseconds very accidentally to enable cap of
KVM_CAP_SPLIT_IRQCHIPon a host which has many vms already.The reason is that when vmm(qemu/CloudHypervisor) invokes
KVM_CAP_SPLIT_IRQCHIPkvm will callsynchronize_srcu_expedited()andmight_sleepand kworker of srcu may cost some delay during this period. One way makes sence is setup empty irq routing when creating vm and so that x86/s390 don't need to setup empty/dummy irq routing.
Quoted 3 lines
Quoted 1 line
Note: I have no s390 machine so the s390 patch has not been tested.
I just did a quick sniff and it still seems to work. No performance check etc.
Thanks very much, Christian!
---
Best wishes
Yi Wang