From: David Vrabel Subject: Re: [Xen-devel] [PATCH] xen: reuse the same pirq allocated when driver load first time Date: Tue, 21 May 2013 11:07:10 +0100 Message-ID: <519B474E.4000202@citrix.com> References: <20130513161714.GC10401@phenom.dumpdata.com> <20130513182055.GC14177@phenom.dumpdata.com> <20130514142013.GA10173@konrad-lan.dumpdata.com> <5195944A.3050608@oracle.com> <20130520175706.GA27973@phenom.dumpdata.com> <20130520203855.GA30616@phenom.dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20130520203855.GA30616@phenom.dumpdata.com> Sender: linux-kernel-owner@vger.kernel.org To: Konrad Rzeszutek Wilk Cc: Stefano Stabellini , "xen-devel@lists.xensource.com" , Feng Jin , Zhenzhong Duan , Yuval Shaia , "linux-kernel@vger.kernel.org" , Chien Yen , Ingo Molnar , "H. Peter Anvin" , Thomas Gleixner List-Id: xen-devel@lists.xenproject.org On 20/05/13 21:38, Konrad Rzeszutek Wilk wrote: >> At this point I think that upstream option is to save the PIRQ value and re-use it. >> Will post a patch for it. > > Here is the patch. It works for me when passing in a NIC driver. > >>From 509499568d1cdf1f2a3fb53773c991f4b063eb56 Mon Sep 17 00:00:00 2001 > From: Konrad Rzeszutek Wilk > Date: Mon, 20 May 2013 16:08:16 -0400 > Subject: [PATCH] xen/pci: Track PVHVM PIRQs. > > The PIRQs that the hypervisor provides for the guest are a limited > resource. They are acquired via PHYSDEVOP_get_free_pirq and in > theory should be returned back to the hypervisor via PHYSDEVOP_unmap_pirq > hypercall. Unfortunatly that is not the case. > > This means that if there is a PCI device that has been passed in > the guest and does a loop of 'rmmod ;modprobe " > we end up exhausting all of the PIRQs that are available. > > For example (with kernel built as debug), we get this: > 00:05.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) > [ 152.659396] e1000e 0000:00:05.0: xen: msi bound to pirq=53 > [ 152.665856] e1000e 0000:00:05.0: xen: msi --> pirq=53 --> irq=73 > .. snip > [ 188.276835] e1000e 0000:00:05.0: xen: msi bound to pirq=51 > [ 188.283194] e1000e 0000:00:05.0: xen: msi --> pirq=51 --> irq=73 > > .. and so on, until the pirq value is zero. This is an acute problem > when many PCI devices with many MSI-X entries are passed in the guest. > > There is an alternative solution where we assume that on PCI > initialization (so when user passes in the PCI device) QEMU will init > the MSI and MSI-X entries to zero. Based on that assumptions and > that the Linux MSI API will write the PIRQ value to the MSI/MSI-X > (and used by QEMU through the life-cycle of the PCI device), we can > also depend on that. That means if MSI (or MSI-X entries) are read back > and are not 0, we can re-use that PIRQ value. However this patch > guards against future changes in QEMU in case that assumption > is incorrect. > > Reported-by: Zhenzhong Duan > CC: Stefano Stabellini > Signed-off-by: Konrad Rzeszutek Wilk > --- > drivers/xen/events.c | 124 +++++++++++++++++++++++++++++++++++++++++++++++++- > 1 files changed, 122 insertions(+), 2 deletions(-) > > diff --git a/drivers/xen/events.c b/drivers/xen/events.c > index 6a6bbe4..8aae21a 100644 > --- a/drivers/xen/events.c > +++ b/drivers/xen/events.c > @@ -112,6 +112,27 @@ struct irq_info { > #define PIRQ_NEEDS_EOI (1 << 0) > #define PIRQ_SHAREABLE (1 << 1) > > +/* > + * The PHYSDEVOP_get_free_pirq allocates a set of PIRQs for the guest and > + * the PHYSDEVOP_unmap_pirq is suppose to return them to the hypervisor. > + * Unfortunatly that is not the case and we exhaust all of the PIRQs that are > + * allocated for the domain if a driver is loaded/unloaded in a loop. > + * The pirq_info serves a cache of the allocated PIRQs so that we can reuse > + * for drivers. Note, it is only used by the MSI, MSI-X routines. > + */ Ick. Let's fix the bug in the hypervisor instead of hacking up the kernel like this. Looking at the hypervisor code I couldn't see anything obviously wrong. I do note that Xen doesn't free the pirq until it has been unbound by the guest. Xen will warn if the guest unmaps a pirq that is still bound ("domD: forcing unbind of pirq P"). Is this what is happening? If so, that would suggest a bug in the guest rather than the hypervisor. David