Fix interrupt latency of HVM PCI passthrough devices.
PATCH v4 1/1 Fix interrupt latency of HVM PCI passthrough devices.
Please see this patch which I hope meets your standards.
There are two pieces of code that might be a bit odd:
1). The usage of 'int cpu = smp_processor_id()' and then using
it just once, instead of using '*this_cpu' in schedule_dpci_for. The reason is that Andrew pointed out to me (in the tasklet review) that we want to spend as little time as possible in the critical code and hence moving the 'cpu = smp_processor_id()' out would do that.
2). The assert:
ASSERT(list_empty(&per_cpu(dpci_list, cpu)));
Other code in the hypervisor has 'ASSERT( list_empty..') but the syntax in the io.c file is to have no spaces in the ASSERT. If you would like it to be the same - ASSERT( x ), I can prepare a cleanup patch to fix that.
Otherwise it should be all good.
xen/arch/x86/domain.c | 4 +- xen/drivers/passthrough/io.c | 168 ++++++++++++++++++++++++++++++++++++++-- xen/drivers/passthrough/pci.c | 24 +++++-- xen/include/xen/hvm/irq.h | 2 +- xen/include/xen/pci.h | 2 +- xen/include/xen/sched.h | 6 ++ xen/include/xen/softirq.h | 3 + 7 files changed, 191 insertions(+), 18 deletions(-)
Konrad Rzeszutek Wilk (1):
dpci: Replace tasklet with an softirq (v4)
PATCH 1/1 dpci: Replace tasklet with an softirq (v4)
From: Konrad Rzeszutek Wilk <[email protected]>
The existing tasklet mechanism has a single global spinlock that is taken every-time the global list is touched. And we use this lock quite a lot - when we call do_tasklet_work which is called via an softirq and from the idle loop. We take the lock on any operation on the tasklet_list.
The problem we are facing is that there are quite a lot of tasklets scheduled. The most common one that is invoked is the one injecting the VIRQ_TIMER in the guest. Guests are not insane and don't set the one-shot or periodic clocks to be in sub 1ms intervals (causing said tasklet to be scheduled for such small intervalls).
The problem appears when PCI passthrough devices are used over many sockets and we have an mix of heavy-interrupt guests and idle guests. The idle guests end up seeing 1/10 of its RUNNING timeslice eaten by the hypervisor (and 40% steal time).
The mechanism by which we inject PCI interrupts is by hvm_do_IRQ_dpci which schedules the hvm_dirq_assist tasklet every time an interrupt is received. The callchain is:
_asm_vmexit_handler
-> vmx_vmexit_handler
->vmx_do_extint
-> do_IRQ
-> __do_IRQ_guest
-> hvm_do_IRQ_dpci
tasklet_schedule(&dpci->dirq_tasklet);
[takes lock to put the tasklet on][later on the schedule_tail is invoked which is 'vmx_do_resume']
vmx_do_resume
-> vmx_asm_do_vmentry
-> call vmx_intr_assist
-> vmx_process_softirqs
-> do_softirq
[executes the tasklet function, takes the
lock again]While on other CPUs they might be sitting in a idle loop and invoked to deliver an VIRQ_TIMER, which also ends up taking the lock twice: first to schedule the v->arch.hvm_vcpu.assert_evtchn_irq_tasklet (accounted to the guests' BLOCKED_state); then to execute it - which is accounted for in the guest's RUNTIME_state.
The end result is that on a 8 socket machine with PCI passthrough, where four sockets are busy with interrupts, and the other sockets have idle guests - we end up with the idle guests having around 40% steal time and 1/10 of its timeslice (3ms out of 30 ms) being tied up taking the lock. The latency of the PCI interrupts delieved to guest is also hindered.
With this patch the problem disappears completly. That is removing the lock for the PCI passthrough use-case (the 'hvm_dirq_assist' case) by not using tasklets at all.
The patch is simple - instead of scheduling an tasklet
we schedule our own softirq - HVM_DPCI_SOFTIRQ, which will take care of running 'hvm_dirq_assist'. The information we need on each CPU is which 'struct domain' structure the 'hvm_dirq_assist' needs to run on. That is simple solved by threading the 'struct domain' through a linked list. The rule of only running one 'hvm_dirq_assist' for only one 'struct domain' is also preserved by having 'schedule_dpci_for' ignore any subsequent calls for an domain which has already been scheduled.
---
v2: On top of ref-cnts also have wait loop for the outstanding
'struct domain' that need to be processed.
v3: Add -ERETRY, fix up StyleGuide issues
v4: Clean it up mode, redo per_cpu, this_cpu logic
--- xen/arch/x86/domain.c | 4 +- xen/drivers/passthrough/io.c | 168 ++++++++++++++++++++++++++++++++++++++-- xen/drivers/passthrough/pci.c | 24 +++++-- xen/include/xen/hvm/irq.h | 2 +- xen/include/xen/pci.h | 2 +- xen/include/xen/sched.h | 6 ++ xen/include/xen/softirq.h | 3 + 7 files changed, 191 insertions(+), 18 deletions(-)
xen/arch/x86/domain.c +3 -1
@@ -1921,7 +1921,9 @@ int domain_relinquish_resources(struct domain *d)switch ( d->arch.relmem ) {case RELMEM_not_started: -pci_release_devices (d); + ret =pci_release_devices (d); +if ( ret ) +return ret;/* Tear down paging-assistance stuff. */ paging_teardown (d);
xen/drivers/passthrough/io.c +159 -9
202 lines not on this page · raw message
xen/drivers/passthrough/pci.c +18 -6
@@ -770,37 +770,47 @@ static int pci_clean_dpci_irq(struct domain *d,return 0 ; } -static void pci_clean_dpci_irqs (struct domain *d) +static int pci_clean_dpci_irqs (struct domain *d) {struct hvm_irq_dpci *hvm_irq_dpci =NULL ;if ( !iommu_enabled ) -return ; +return -ESRCH ;if ( !is_hvm_domain (d) ) -return ; +return -EINVAL ;spin_lock (&d->event_lock); hvm_irq_dpci =domain_get_irq_dpci (d);if ( hvm_irq_dpci !=NULL ) { -tasklet_kill (&hvm_irq_dpci->dirq_tasklet); +int ret =dpci_kill (d); +if ( ret ) + { +spin_unlock (&d->event_lock); +return ret; + }pt_pirq_iterate (d, pci_clean_dpci_irq,NULL ); d->arch.hvm_domain.irq.dpci =NULL ;free_hvm_irq_dpci (hvm_irq_dpci); }spin_unlock (&d->event_lock); +return 0 ; } -void pci_release_devices (struct domain *d) +int pci_release_devices (struct domain *d) {struct pci_dev *pdev;u8 bus, devfn; +int ret;spin_lock (&pcidevs_lock); -pci_clean_dpci_irqs (d); + ret =pci_clean_dpci_irqs (d); +if ( ret && ret == -EAGAIN ) +return ret; +while ( (pdev =pci_get_pdev_by_domain (d,-1 ,-1 ,-1 )) ) { bus = pdev->bus;
@@ -811,6 +821,8 @@ void pci_release_devices(struct domain *d) PCI_SLOT(devfn), PCI_FUNC(devfn)); }spin_unlock (&pcidevs_lock); + +return 0 ; }#define PCI_CLASS_BRIDGE_HOST 0x0600
xen/include/xen/hvm/irq.h +1 -1
@@ -88,7 +88,6 @@ struct hvm_irq_dpci {DECLARE_BITMAP (isairq_map,NR_ISAIRQS );/* Record of mapped Links */ uint8_t link_cnt[NR_LINK ]; -struct tasklet dirq_tasklet; };/* Machine IRQ to guest device/intx mapping. */
@@ -109,6 +108,7 @@ int pt_pirq_iterate(struct domain *d, struct hvm_pirq_dpci *, void *arg), void *arg); +int dpci_kill (struct domain *d);/* Modify state of a PCI INTx wire. */ void hvm_pci_intx_assert (struct domain *d,unsigned int device,unsigned int intx);
xen/include/xen/pci.h +1 -1
@@ -99,7 +99,7 @@ struct pci_dev *pci_lock_domain_pdev(void setup_hwdom_pci_devices (struct domain *,int (*)(u8 devfn,struct pci_dev *)); -void pci_release_devices (struct domain *d); +int pci_release_devices (struct domain *d);int pci_add_segment (u16 seg);const unsigned long *pci_get_ro_map (u16 seg);int pci_add_device (u16 seg,u8 bus,u8 devfn,const struct pci_dev_info *);
xen/include/xen/sched.h +6 -0
@@ -456,6 +456,12 @@ struct domain/* vNUMA topology accesses are protected by rwlock. */ rwlock_t vnuma_rwlock;struct vnuma_info *vnuma; + +#ifdef HAS_PASSTHROUGH +/* For HVM_DPCI_SOFTIRQ. We use refcnt when scheduled to run. */ +struct list_head list; +unsigned long state; +#endif };struct domain_setup_info
xen/include/xen/softirq.h +3 -0
@@ -8,6 +8,9 @@ enum {NEW_TLBFLUSH_CLOCK_PERIOD_SOFTIRQ ,RCU_SOFTIRQ ,TASKLET_SOFTIRQ , +#ifdef HAS_PASSTHROUGH +HVM_DPCI_SOFTIRQ , +#endif NR_COMMON_SOFTIRQS };
Signature · 1 line
1.7.7.6
Re: [PATCH] dpci: Replace tasklet with an softirq (v4)
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 10.09.14 at 22:26, <[email protected]> wrote:
The patch is simple - instead of scheduling an tasklet
we schedule our own softirq -HVM_DPCI_SOFTIRQ, which will take care of running 'hvm_dirq_assist'. The information we need on each CPU is which 'struct domain' structure the 'hvm_dirq_assist' needs to run on. That is simple solved by threading the 'struct domain' through a linked list. The rule of only running one 'hvm_dirq_assist' for only one 'struct domain' is also preserved by having 'schedule_dpci_for' ignore any subsequent calls for an domain which has already been scheduled.
Quoted 6 lines
Signed-off-by Konrad Rzeszutek Wilk <[email protected]>---
v2: On top of ref-cnts also have wait loop for the outstanding
'struct domain' that need to be processed.
v3: Add-ERETRY, fix up StyleGuide issues
v4: Clean it up mode, redoper_cpu,this_cpulogic
Quoted 8 lines
--- xen/arch/x86/domain.c | 4 +- xen/drivers/passthrough/io.c | 168 ++++++++++++++++++++++++++++++++++++++-- xen/drivers/passthrough/pci.c | 24 +++++-- xen/include/xen/hvm/irq.h | 2 +- xen/include/xen/pci.h | 2 +- xen/include/xen/sched.h | 6 ++ xen/include/xen/softirq.h | 3 +
It also looks like you failed to Cc a couple of people, even if the changes outside of xen/drivers/passthrough/ are small.
Quoted 6 lines
xen/drivers/passthrough/io.c +0
-0@@ -20,14 +20,123 @@#include <xen/event.h> #include <xen/iommu.h>
xen/drivers/passthrough/io.c line 30
Quoted 10 lines
xen/drivers/passthrough/io.c +2
-2+#include <xen/cpu.h> #include <xen/irq.h> #include <asm/hvm/irq.h> #include <asm/hvm/iommu.h> #include <asm/hvm/support.h> #include <xen/hvm/irq.h> -#include <xen/tasklet.h> -static void hvm_dirq_assist (unsigned long _d); +static void hvm_dirq_assist (struct domain *d);
I think by not placing some (most?) of your new code at the top of the file, you could avoid the need for this declaration altogether.
xen/drivers/passthrough/io.c line 75
Quoted 9 lines
xen/drivers/passthrough/io.c +9
-0+static void schedule_dpci_for (struct domain *d) +{ +if ( !test_and_set_bit (STATE_SCHED , &d->state) ) + { +unsigned long flags; +struct list_head *list; +int cpu =smp_processor_id (); + +INIT_LIST_HEAD (&d->list);
Why does this need doing on each schedule operation?
xen/drivers/passthrough/io.c line 82
Quoted 7 lines
xen/drivers/passthrough/io.c +7
-0+ +get_knownalive_domain (d);/* To be put by 'dpci_softirq' */ + +local_irq_save (flags); + list = &per_cpu (dpci_list, cpu); +list_add_tail (&d->list, list); +local_irq_restore (flags);
I doubt that the avoidance of this_cpu() here really buys us much. In fact I think the "list" local variable is quite pointless (hampering readability) too.
Furthermore, with this much of restructuring I wonder whether the domain centric model here really is the right one: All hvm_dirq_assist() does is iterate over all IRQs, yet the raising side already knows which struct hvm_pirq_dpci instance is of interest. (It was quite odd for the old code to have a per-IRQ tasklet, but have the tasklet action then iterate over all IRQs for the domain.) Such a switch would seem to have the potential of reducing the complexity in dpci_softirq().
(As a side note, _hvm_dirq_assist() seems poorly coded too: The loop there does a number of loop-invariant things. I'll try to remember to see whether this can be done in a less convoluted way.)
Quoted 13 lines
xen/drivers/passthrough/io.c +13
-0+ +raise_softirq (HVM_DPCI_SOFTIRQ ); + } +} + +int dpci_kill (struct domain *d) +{ +if (test_and_set_bit (STATE_SCHED , &d->state) ) +return -EAGAIN ; + +if (test_bit (STATE_RUN , &d->state) ) +return -EAGAIN ; +
xen/drivers/passthrough/io.c line 105
Quoted 10 lines
xen/drivers/passthrough/io.c +10
-0+clear_bit (STATE_SCHED , &d->state); + +return 0 ; +} + +static void dpci_softirq (void ) +{ +struct domain *d; +int cpu =smp_processor_id (); +struct list_head *list;
Same here - pretty pointless local variable, to some degree hiding what the actual operation is. "cpu" otoh, being used more than once here, is certainly useful to retain. But please all of those variables throughout the patch should be "unsigned int".
xen/drivers/passthrough/io.c line 108
Quoted 3 lines
xen/drivers/passthrough/io.c +3
-0+struct list_head our_list; + +INIT_LIST_HEAD (&our_list);
Please use LIST_HEAD() instead.
Quoted 12 lines
xen/drivers/passthrough/io.c +12
-0+ +local_irq_disable (); + list = &per_cpu (dpci_list, cpu); +list_splice_init (list, &our_list); +local_irq_enable (); + +while ( !list_empty (&our_list) ) + { + d = list_entry(our_list.next, struct domain, list); +list_del (&d->list); + +if ( !test_and_set_bit (STATE_RUN , &d->state) )
xen/drivers/passthrough/io.c line 130
Quoted 10 lines
xen/drivers/passthrough/io.c +10
-0+ { +if ( !test_and_clear_bit (STATE_SCHED , &d->state) ) +BUG (); +hvm_dirq_assist (d); +clear_bit (STATE_RUN , &d->state); +put_domain (d); +continue ; + } + +INIT_LIST_HEAD (&d->list);
And again - why?
xen/drivers/passthrough/io.c
Quoted 12 lines
xen/drivers/passthrough/io.c +5
-2-void pci_release_devices (struct domain *d) +int pci_release_devices (struct domain *d) {struct pci_dev *pdev;u8 bus, devfn; +int ret;spin_lock (&pcidevs_lock); -pci_clean_dpci_irqs (d); + ret =pci_clean_dpci_irqs (d); +if ( ret && ret == -EAGAIN ) +return ret;
Just "if ( ret == -EAGAIN )" would have the same effect afaict.
xen/include/xen/sched.h struct domain · line 464
Quoted 12 lines
xen/include/xen/sched.h +6
-0@@ -456,6 +456,12 @@ struct domain/* vNUMA topology accesses are protected by rwlock. */ rwlock_t vnuma_rwlock;struct vnuma_info *vnuma; + +#ifdef HAS_PASSTHROUGH +/* For HVM_DPCI_SOFTIRQ. We use refcnt when scheduled to run. */ +struct list_head list; +unsigned long state; +#endif
These either need a name prefix or putting into a suitably named container struct (I'd personally prefer the latter). All of course only if the domain centric model needs to be retained for some reason.
Jan
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 10.09.14 at 22:26, <[email protected]> wrote:
2). The assert:
ASSERT(list_empty(&per_cpu(dpci_list, cpu)));Other code in the hypervisor has '
ASSERT( list_empty..')but the syntax in theio.cfile is to have no spaces in the ASSERT. If you would like it to be the same - ASSERT( x ), I can prepare a cleanup patch to fix that.
If you want to invest time into things like this, just go ahead. But based on what you write above I'm not sure which direction you want to change things: ASSERT(x) is the correct form, since ASSERT is not a language keyword and also not a close equivalent (like for_each_cpu might be considered for example).
Jan