Skip to content

RFC more fine grained locking for IRQ injection

Gleb Natapov kvm 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest10 patches 15 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/10Move irq routing data structure to rcu lockinginclude/linux/kvm_host.h virt/kvm/irq_comm.c virt/kvm/kvm_main.c+26 -320 replies
2/10Unregister ack notifier callback on PIT freeing.arch/x86/kvm/i8254.c+2 -00 replies
3/10Move irq ack notifier list to arch independent code.arch/ia64/include/asm/kvm_host.h arch/x86/include/asm/kvm_host.h include/linux/kvm_host.h +2 more+4 -40 replies
4/10Convert irq notifiers lists to RCU locking.virt/kvm/irq_comm.c+11 -90 replies
5/10Protect irq_sources_bitmap by kvm->lock instead of kvm->irq_lockarch/x86/kvm/i8254.c virt/kvm/irq_comm.c+6 -40 replies
6/10Move irq routing to its own locking.include/linux/kvm_host.h virt/kvm/irq_comm.c virt/kvm/kvm_main.c+4 -30 replies
7/10Move irq notifiers lists to its own locking.include/linux/kvm_host.h virt/kvm/irq_comm.c virt/kvm/kvm_main.c+10 -80 replies
8/10Move IO APIC to its own lock.arch/ia64/kvm/kvm-ia64.c arch/x86/kvm/lapic.c arch/x86/kvm/x86.c +2 more+73 -409 replies
9/10Drop kvm->irq_lock lock.arch/ia64/kvm/kvm-ia64.c arch/x86/kvm/i8254.c arch/x86/kvm/lapic.c +5 more+2 -200 replies
10/10Change irq routing table to use gsi indexed array.include/linux/kvm_host.h virt/kvm/irq_comm.c+47 -266 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/10] [RFC] more fine grained locking for IRQ injection

kvm->irq_lock protects too much stuff, but still fail to protect everything it was design to protect (see ack notifiers call in pic). I want to make IRQ injection logic use more fine grained locking. This patch series split kvm->irq_lock mutex to smaller spinlocks each one protects only one thing. Irq routing, irq notifier lists and ioapic gain their own spinlock. pic is already uses its own lock. This patch series also makes interrupt injection to lapic lockless (several kvm_irq_delivery_to_apic() may run in parallel), but access to lapic was never fully locked in the first place. VCPU could access lapic in parallel with interrupt injection.

This patch series is exactly like previous two combined + one more patch that changes irq_routing to use gsi indexed array to map from gsi to irq chip. But now the patch series comes with numbers. Here is oprofile data of one WindowsXP boot with two cpus generated by the command
"opreport -l | grep kvm | awk '{print $1" "$5}' | grep irq"

Master: With patches:
996 kvm_set_irq 412 __apic_accept_irq
502 __apic_accept_irq 298 kvm_set_irq
250 kvm_irq_delivery_to_apic 228 kvm_irq_delivery_to_api
162 kvm_ioapic_set_irq 178 kvm_ioapic_set_irq
162 kvm_pic_set_irq 137 enable_irq_window
95 kvm_apic_set_irq 122 kvm_pic_set_irq
82 kvm_set_pic_irq 71 kvm_apic_set_irq
77 enable_irq_window 55 pic_get_irq
60 vmx_inject_irq 48 vmx_inject_irq
52 pic_get_irq 44 kvm_set_ioapic_irq
49 kvm_set_ioapic_irq 42 kvm_set_pic_irq
48 pic_update_irq 42 pic_update_irq
40 pic_irq_request 17 kvm_notify_acked_irq
37 kvm_notify_acked_irq 17 pic_irq_request
1 kvm_inject_pit_timer_irqs

And the same with the patchset + small userspace hack (remove pic from irq routing table when guest stops use it):
418 __apic_accept_irq
242 kvm_irq_delivery_to_apic
212 kvm_set_irq
112 kvm_ioapic_set_irq
98 enable_irq_window
81 kvm_apic_set_irq
72 vmx_inject_irq
60 kvm_pic_set_irq
30 kvm_set_ioapic_irq
20 pic_get_irq
12 pic_update_irq
11 kvm_set_pic_irq
10 pic_irq_request
7 kvm_notify_acked_irq

Gleb Natapov (9):
Unregister ack notifier callback on PIT freeing.
Move irq ack notifier list to arch independent code. Convert irq notifiers lists to RCU locking.
Protect irq_sources_bitmap by kvm->lock instead of kvm->irq_lock Move irq routing to its own locking.
Move irq notifiers lists to its own locking.
Move IO APIC to its own lock.
Drop kvm->irq_lock lock.
Change irq routing table to use gsi indexed array.

 arch/ia64/include/asm/kvm_host.h |    1 -
 arch/ia64/kvm/kvm-ia64.c         |   29 +++++++---
 arch/x86/include/asm/kvm_host.h  |    1 -
 arch/x86/kvm/i8254.c             |    6 +-
 arch/x86/kvm/lapic.c             |    7 +--
 arch/x86/kvm/x86.c               |   32 +++++++----
 include/linux/kvm_host.h         |   15 ++++-
 virt/kvm/eventfd.c               |    2 -
 virt/kvm/ioapic.c                |   50 ++++++++++------
 virt/kvm/ioapic.h                |    1 +
 virt/kvm/irq_comm.c              |  117 +++++++++++++++++++++-----------------
 virt/kvm/kvm_main.c              |    8 +-
 12 files changed, 157 insertions(+), 112 deletions(-)