KVM: irqchip: synchronize srcu only if needed
v3 cover KVM: irqchip: synchronize srcu only if needed
From: Yi Wang <[email protected]>
We found that it may cost more than 20 milliseconds very accidentally to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms already.
The reason is that when vmm(qemu/CloudHypervisor) invokes KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and might_sleep and kworker of srcu may cost some delay during this period. One way makes sence is setup empty irq routing when creating vm and so that x86/s390 don't need to setup empty/dummy irq routing.
Note: I have no s390 machine so this patch has not been tested thoroughly on s390 platform. Thanks to Christian for a quick test on s390 and it still seems to work[1].
Changelog:
----------
v3:
- squash setup empty routing function and use of that into one commit
- drop the comment in s390 part
v2:
- setup empty irq routing in kvm_create_vm
- don't setup irq routing in x86 KVM_CAP_SPLIT_IRQCHIP
- don't setup irq routing in s390 KVM_CREATE_IRQCHIP
v1: https://lore.kernel.org/kvm/[email protected]/
1. https://lore.kernel.org/lkml/[email protected]/
Yi Wang (3):
KVM: setup empty irq routing when create vm
KVM: x86: don't setup empty irq routing when KVM_CAP_SPLIT_IRQCHIP
KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP
arch/s390/kvm/kvm-s390.c | 9 +-------- arch/x86/kvm/irq.h | 1 - arch/x86/kvm/irq_comm.c | 5 ----- arch/x86/kvm/x86.c | 3 --- include/linux/kvm_host.h | 1 + virt/kvm/irqchip.c | 19 +++++++++++++++++++ virt/kvm/kvm_main.c | 4 ++++ 7 files changed, 25 insertions(+), 17 deletions(-)
Signature · 1 line
2.39.3
PATCH RESEND v3 1/3 KVM: setup empty irq routing when create vm
From: Yi Wang <[email protected]>
Add a new function to setup empty irq routing in kvm path, which can be invoded in non-architecture-specific functions. The difference compared to the kvm_setup_empty_irq_routing() is this function just alloc the empty irq routing and does not need synchronize srcu, as we will call it in kvm_create_vm().
Using the new adding function, we can setup empty irq routing when kvm_create_vm(), so that x86 and s390 no longer need to set empty/dummy irq routing when creating an IRQCHIP 'cause it avoid an synchronize_srcu.
--- include/linux/kvm_host.h | 1 + virt/kvm/irqchip.c | 19 +++++++++++++++++++ virt/kvm/kvm_main.c | 4 ++++ 3 files changed, 24 insertions(+)
include/linux/kvm_host.h +1 -0
@@ -2000,6 +2000,7 @@ int kvm_set_irq_routing(struct kvm *kvm,const struct kvm_irq_routing_entry *entries, unsigned nr, unsigned flags); +int kvm_setup_empty_irq_routing_lockless (struct kvm *kvm);int kvm_set_routing_entry (struct kvm *kvm,struct kvm_kernel_irq_routing_entry *e,const struct kvm_irq_routing_entry *ue);
virt/kvm/irqchip.c +19 -0
@@ -237,3 +237,22 @@ int kvm_set_irq_routing(struct kvm *kvm,return r; } + +int kvm_setup_empty_irq_routing_lockless (struct kvm *kvm) +{ +struct kvm_irq_routing_table *new; +u32 i, j; + + new =kzalloc (struct_size (new, map,1 ),GFP_KERNEL_ACCOUNT ); +if (!new) +return -ENOMEM ; + + new->nr_rt_entries =1 ; +for (i =0 ; i <KVM_NR_IRQCHIPS ; i++) +for (j =0 ; j <KVM_IRQCHIP_NUM_PINS ; j++) + new->chip[i][j] =-1 ; + +RCU_INIT_POINTER (kvm->irq_routing, new); + +return 0 ; +}
virt/kvm/kvm_main.c +4 -0
@@ -1242,6 +1242,10 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)if (r) goto out_err; + r =kvm_setup_empty_irq_routing_lockless (kvm); +if (r) + goto out_err; +mutex_lock (&kvm_lock);list_add (&kvm->vm_list, &vm_list);mutex_unlock (&kvm_lock);
Signature · 1 line
2.39.3
Re: [RESEND v3 1/3] KVM: setup empty irq routing when create vm
Quoted 4 lines · Yi Wang
From: Yi Wang <[email protected]>
Add a new function to setup empty irq routing in kvm path, which can be invoded in non-architecture-specific functions. The difference
:s/invoded/invoked/
Dongli Zhang
Re: [RESEND v3 1/3] KVM: setup empty irq routing when create vm
Quoted 10 lines · Dongli Zhang
Quoted 4 lines · Yi Wang
From: Yi Wang <[email protected]>
Add a new function to setup empty irq routing in kvm path, which can be invoded in non-architecture-specific functions. The difference
:s/invoded/invoked/
Thanks for pointing this out.
Sean, Oliver, Christian, Do you have any other thoughts? Any suggestions are appreciated.
Quoted 2 lines
Dongli Zhang
Signature · 3 lines
--- Best wishes Yi Wang
Re: [RESEND v3 1/3] KVM: setup empty irq routing when create vm
In reply to Yi WangQuoted 44 lines · Yi Wang
From: Yi Wang <[email protected]>
Add a new function to setup empty irq routing in kvm path, which can be invoded in non-architecture-specific functions. The difference compared to the
kvm_setup_empty_irq_routing()is this function just alloc the empty irq routing and does not need synchronize srcu, as we will call it inkvm_create_vm().Using the new adding function, we can setup empty irq routing when
kvm_create_vm(), so that x86 and s390 no longer need to set empty/dummy irq routing when creating an IRQCHIP 'cause it avoid ansynchronize_srcu.Signed-off-by Yi Wang <[email protected]>--- include/linux/kvm_host.h | 1 + virt/kvm/irqchip.c | 19 +++++++++++++++++++ virt/kvm/kvm_main.c | 4 ++++ 3 files changed, 24 insertions(+)include/linux/kvm_host.h +1
-0@@ -2000,6 +2000,7 @@ int kvm_set_irq_routing(struct kvm *kvm,const struct kvm_irq_routing_entry *entries, unsigned nr, unsigned flags); +int kvm_setup_empty_irq_routing_lockless (struct kvm *kvm);int kvm_set_routing_entry (struct kvm *kvm,struct kvm_kernel_irq_routing_entry *e,const struct kvm_irq_routing_entry *ue);virt/kvm/irqchip.c +4
-0@@ -237,3 +237,22 @@ int kvm_set_irq_routing(struct kvm *kvm,return r; } + +int kvm_setup_empty_irq_routing_lockless (struct kvm *kvm) +{ +struct kvm_irq_routing_table *new;
virt/kvm/irqchip.c int kvm_set_irq_routing(struct kvm *kvm, · line 253
Quoted 10 lines
virt/kvm/irqchip.c +10
-0+u32 i, j; + + new =kzalloc (struct_size (new, map,1 ),GFP_KERNEL_ACCOUNT ); +if (!new) +return -ENOMEM ; + + new->nr_rt_entries =1 ; +for (i =0 ; i <KVM_NR_IRQCHIPS ; i++) +for (j =0 ; j <KVM_IRQCHIP_NUM_PINS ; j++) + new->chip[i][j] =-1 ;
Maybe it looks nicer by:
size = sizeof(int) * KVM_NR_IRQCHIPS *KVM_IRQCHIP_NUM_PINS; memset(new->chip, -1, size);
Quoted 20 lines
virt/kvm/irqchip.c +5
-0+ +RCU_INIT_POINTER (kvm->irq_routing, new); + +return 0 ; +}virt/kvm/kvm_main.c +4
-0@@ -1242,6 +1242,10 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)if (r) goto out_err; + r =kvm_setup_empty_irq_routing_lockless (kvm); +if (r) + goto out_err; +mutex_lock (&kvm_lock);list_add (&kvm->vm_list, &vm_list);mutex_unlock (&kvm_lock);
Re: [RESEND v3 1/3] KVM: setup empty irq routing when create vm
Quoted 14 lines · Weijiang
Quoted 12 lines · Yi Wang
From: Yi Wang <[email protected]>
virt/kvm/irqchip.c +10
-0+ +int kvm_setup_empty_irq_routing_lockless (struct kvm *kvm) +{ +struct kvm_irq_routing_table *new; +u32 i, j; + + new =kzalloc (struct_size (new, map,1 ),GFP_KERNEL_ACCOUNT ); +if (!new) +return -ENOMEM ; +
Quoted 9 lines
virt/kvm/irqchip.c int kvm_set_irq_routing(struct kvm *kvm, · line 253
Quoted 4 lines
virt/kvm/irqchip.c +4
-0+ new->nr_rt_entries =1 ; +for (i =0 ; i <KVM_NR_IRQCHIPS ; i++) +for (j =0 ; j <KVM_IRQCHIP_NUM_PINS ; j++) + new->chip[i][j] =-1 ;Maybe it looks nicer by:
size = sizeof(int) *KVM_NR_IRQCHIPS*KVM_IRQCHIP_NUM_PINS;memset(new->chip, -1, size);
It seems better, I'll update this patch. Thx a lot!
Quoted 6 lines
virt/kvm/irqchip.c int kvm_set_irq_routing(struct kvm *kvm, · line 257
Quoted 4 lines
virt/kvm/irqchip.c +4
-0+ +RCU_INIT_POINTER (kvm->irq_routing, new); + +return 0 ;
---
Best wishes
Yi Wang
PATCH RESEND v3 2/3 KVM: x86: don't setup empty irq routing when KVM_CAP_SPLIT_IRQCHIP
In reply to Yi WangFrom: Yi Wang <[email protected]>
We found that it may cost more than 20 milliseconds very accidentally to enable cap of KVM_CAP_SPLIT_IRQCHIP on a host which has many vms already.
The reason is that when vmm(qemu/CloudHypervisor) invokes KVM_CAP_SPLIT_IRQCHIP kvm will call synchronize_srcu_expedited() and might_sleep and kworker of srcu may cost some delay during this period.
As we have set up empty irq routing when creating vm, so this is no need now.
--- arch/x86/kvm/irq.h | 1 - arch/x86/kvm/irq_comm.c | 5 ----- arch/x86/kvm/x86.c | 3 --- 3 files changed, 9 deletions(-)
arch/x86/kvm/irq.h +0 -1
@@ -106,7 +106,6 @@ void __kvm_migrate_timers(struct kvm_vcpu *vcpu);int apic_has_pending_timer (struct kvm_vcpu *vcpu);int kvm_setup_default_irq_routing (struct kvm *kvm); -int kvm_setup_empty_irq_routing (struct kvm *kvm);int kvm_irq_delivery_to_apic (struct kvm *kvm,struct kvm_lapic *src,struct kvm_lapic_irq *irq,struct dest_map *dest_map);
arch/x86/kvm/irq_comm.c +0 -5
@@ -392,11 +392,6 @@ int kvm_setup_default_irq_routing(struct kvm *kvm)static const struct kvm_irq_routing_entry empty_routing[] = {}; -int kvm_setup_empty_irq_routing (struct kvm *kvm) -{ -return kvm_set_irq_routing (kvm, empty_routing,0 ,0 ); -} -void kvm_arch_post_irq_routing_update (struct kvm *kvm) {if (!irqchip_split (kvm))
arch/x86/kvm/x86.c +0 -3
@@ -6414,9 +6414,6 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, goto split_irqchip_unlock;if (kvm->created_vcpus) goto split_irqchip_unlock; - r =kvm_setup_empty_irq_routing (kvm); -if (r) - goto split_irqchip_unlock;/* Pairs with irqchip_in_kernel. */ smp_wmb (); kvm->arch.irqchip_mode =KVM_IRQCHIP_SPLIT ;
Signature · 1 line
2.39.3
PATCH RESEND v3 3/3 KVM: s390: don't setup dummy routing when KVM_CREATE_IRQCHIP
In reply to Yi WangFrom: Yi Wang <[email protected]>
As we have setup empty irq routing in kvm_create_vm(), there's no need to setup dummy routing when KVM_CREATE_IRQCHIP.
--- arch/s390/kvm/kvm-s390.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-)
arch/s390/kvm/kvm-s390.c +1 -8
@@ -2999,14 +2999,7 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)break ; }case KVM_CREATE_IRQCHIP : { -struct kvm_irq_routing_entry routing; - - r = -EINVAL ; -if (kvm->arch.use_irqchip) { -/* Set up dummy routing. */ -memset (&routing,0 ,sizeof (routing)); - r =kvm_set_irq_routing (kvm, &routing,0 ,0 ); - } + r =0 ;break ; }case KVM_SET_DEVICE_ATTR : {
Signature · 1 line
2.39.3