Skip to contentSkip to conversation

uio: add generic driver for PCI 2.3 devices

RFC PATCH In review Michael S. Tsirkin 4 participants 9 messages Jul 9, 2009 – Jul 12, 2009

Subsystems (4)
lore
Share & tools
lore

Michael S. Tsirkin

author maintainer
#0 lore raw

RFC PATCH 1/1 uio: add generic driver for PCI 2.3 devices

Resending with corrected addresses. Sorry about the churn.

-----------

I got annoyed by the fact that we don't support shared interrupts with PCI in assigned devides, so here's a draft patch to add that support in kernel through uio.

I intend to send this to lkml, but meanwhile I'd appreciate some early feedback/flames from people on the list.

Thanks!

----------->

This adds a generic uio driver that can bind to any PCI device. First user will be virtualization where a qemu userspace process needs to give guest OS access to the device.

Interrupts are handled using the Interrupt Disable bit in the PCI command register and Interrupt Status bit in the PCI status register. All devices compliant to PCI 2.3 (circa 2002) and all compliant PCI Express devices should support these bits. Driver detects this support, and won't bind to devices which do not support the Interrupt Disable Bit in the command register.

It's expected that MSI/MSI-X support will be added to this driver in the future, to interface with virtualization irqfd/eventfd infrastructure. Another area to examine, and of interest to virtualization, is iommu.

---
 drivers/uio/Kconfig           |   10 ++
 drivers/uio/Makefile          |    1 +
 drivers/uio/uio_pci_generic.c |  202 +++++++++++++++++++++++++++++++++++++++++
 include/linux/pci_regs.h      |    1 +
 4 files changed, 214 insertions(+), 0 deletions(-)
 create mode 100644 drivers/uio/uio_pci_generic.c
drivers/uio/Kconfig +10 -0
index 7f86534..0f14c8e 100644
@@ -89,4 +89,14 @@ config UIO_SERCOS3
 
 	  If you compile this as a module, it will be called uio_sercos3.
 
+config UIO_PCI_GENERIC
+	tristate "Generic driver for PCI 2.3 and PCI Express cards"
+	depends on PCI
+	default n
+	help
+	  Generic driver that you can bind, dynamically, to any
+	  PCI 2.3 compliant and PCI Express card. It is useful,
+	  primarily, for virtualization scenarios.
+	  If you compile this as a module, it will be called uio_pci_generic.
+
 endif
drivers/uio/Makefile +1 -0
index 5c2586d..73b2e75 100644
@@ -5,3 +5,4 @@ obj-$(CONFIG_UIO_PDRV_GENIRQ)	+= uio_pdrv_genirq.o
 obj-$(CONFIG_UIO_SMX)	+= uio_smx.o
 obj-$(CONFIG_UIO_AEC)	+= uio_aec.o
 obj-$(CONFIG_UIO_SERCOS3)	+= uio_sercos3.o
+obj-$(CONFIG_UIO_PCI_GENERIC)	+= uio_pci_generic.o
drivers/uio/uio_pci_generic.c new file +202 -0

202 lines not on this page · raw message

include/linux/pci_regs.h +1 -0
index 616bf8b..bfb9b31 100644
@@ -42,6 +42,7 @@
 #define  PCI_COMMAND_INTX_DISABLE 0x400 /* INTx Emulation Disable */
 
 #define PCI_STATUS		0x06	/* 16 bits */
+#define  PCI_STATUS_INTERRUPT	0x08	/* Interrupt status */
 #define  PCI_STATUS_CAP_LIST	0x10	/* Support Capability List */
 #define  PCI_STATUS_66MHZ	0x20	/* Support 66 Mhz PCI 2.1 bus */
 #define  PCI_STATUS_UDF		0x40	/* Support User Definable Features [obsolete] */
Signature · 3 lines
1.6.2.5

----- End forwarded message -----
#1 lore raw
Quoted 26 lines · Michael S. Tsirkin

Resending with corrected addresses. Sorry about the churn.

-----------

I got annoyed by the fact that we don't support shared interrupts with PCI in assigned devides, so here's a draft patch to add that support in kernel through uio.

I intend to send this to lkml, but meanwhile I'd appreciate some early feedback/flames from people on the list.

Thanks!

----------->

This adds a generic uio driver that can bind to any PCI device. First user will be virtualization where a qemu userspace process needs to give guest OS access to the device.

Interrupts are handled using the Interrupt Disable bit in the PCI command register and Interrupt Status bit in the PCI status register. All devices compliant to PCI 2.3 (circa 2002) and all compliant PCI Express devices should support these bits. Driver detects this support, and won't bind to devices which do not support the Interrupt Disable Bit in the command register.

Quoted 6 lines

It's expected that MSI/MSI-X support will be added to this driver in the future, to interface with virtualization irqfd/eventfd infrastructure. Another area to examine, and of interest to virtualization, is iommu.

I didn't know this was possible... so we could also use this driver for vm-channel.

Quoted 13 lines
drivers/uio/uio_pci_generic.c new file +12 -0
+
+	err = pci_request_regions(pdev, "uio_pci_generic");
+	if (err) {
+		dev_err(&pdev->dev, "%s: pci_request_regions failed: %d\n",
+			 __func__, err);
+		goto err_verify;
+	}
+
+	dev = kzalloc(sizeof(struct generic_dev), GFP_KERNEL);
+	if (!dev) {
+		err = -ENOMEM;
+		goto err_alloc;

drivers/uio/uio_pci_generic.c

Quoted 10 lines
drivers/uio/uio_pci_generic.c new file +10 -0
+	}
+
+	dev->info.name = "uio_pci_generic";
+	dev->info.version = "0.01";
+	dev->info.irq = pdev->irq;
+	dev->info.irq_flags = IRQF_SHARED;
+	dev->info.handler = irqhandler;
+	dev->info.irqcontrol = irqcontrol;
+	dev->pdev = pdev;
+	spin_lock_init(&dev->lock)

I know it's not strictly needed for PCI pass through, but it would be useful to register the IO regions via UIO. The userspace implementation would then use UIO strictly instead of poking the sysfs pci info directly. I think that ends up being cleaner.

Regards,

ANthony Liguori

author maintainer
#2 lore raw
Quoted 4 lines · Anthony Liguori

I know it's not strictly needed for PCI pass through, but it would be useful to register the IO regions via UIO. The userspace implementation would then use UIO strictly instead of poking the sysfs pci info directly. I think that ends up being cleaner.

Hmm, this is good for specific drivers, but for a generic one like qemu, still need sysfs to figure out the size at least, and
we need config accesses which uio does not support now.
And if you use libpci as qemu does now, this interface will likely go unused. So .. there does not seem to be much point at the moment.

My idea is, let's start with a minimal interface, longer term let's see if we can add config access, mmap and other stuff like eventfd. Makes sense?

Signature · 1 line
MST
#3 lore raw
Quoted 7 lines · Michael S. Tsirkin
Quoted 2 lines · Anthony Liguori

I know it's not strictly needed for PCI pass through, but it would be useful to register the IO regions via UIO. The userspace implementation would then use UIO strictly instead of poking the sysfs pci info directly. I think that ends up being cleaner.

Hmm, this is good for specific drivers, but for a generic one like qemu, still need sysfs to figure out the size at least,

size of what?

Quoted 5 lines

and
we need config accesses which uio does not support now. And if you use libpci as qemu does now, this interface will likely go unused. So .. there does not seem to be much point at the moment.

Right, I would expect uio to replace libpci.

Quoted 4 lines

My idea is, let's start with a minimal interface, longer term let's see if we can add config access, mmap and other stuff like eventfd. Makes sense?

It can certainly grow more features down the road.

Regards,

Anthony Liguori

author maintainer
#4 lore raw
Quoted 9 lines · Anthony Liguori
Quoted 8 lines · Michael S. Tsirkin
Quoted 5 lines · Anthony Liguori

I know it's not strictly needed for PCI pass through, but it would be
useful to register the IO regions via UIO. The userspace
implementation would then use UIO strictly instead of poking the sysfs pci info directly. I think that ends up being cleaner.

Quoted 4 lines
Quoted 2 lines

Hmm, this is good for specific drivers, but for a generic one like qemu, still need sysfs to figure out the size at least,

size of what?

memory region which you mmap.

Quoted 6 lines
Quoted 5 lines

and
we need config accesses which uio does not support now. And if you use libpci as qemu does now, this interface will likely go unused. So .. there does not seem to be much point at the moment.

Quoted 8 lines

Right, I would expect uio to replace libpci.

Quoted 4 lines

My idea is, let's start with a minimal interface, longer term let's see if we can add config access, mmap and other stuff like eventfd. Makes sense?

It can certainly grow more features down the road.

OK, I'll try to get it included as is, then grow it.

Signature · 1 line
MST
#5 lore raw
In reply to Anthony Liguori
Quoted 2 lines · * Anthony Liguori ([email protected]) wrote:

I didn't know this was possible... so we could also use this driver for vm-channel.

With MSI things are much nicer (which we could define for vm-channel). This is for shared legacy INTx.

Quoted 20 lines
Quoted 20 lines
drivers/uio/uio_pci_generic.c new file +20 -0
+
+	err = pci_request_regions(pdev, "uio_pci_generic");
+	if (err) {
+		dev_err(&pdev->dev, "%s: pci_request_regions failed: %d\n",
+			 __func__, err);
+		goto err_verify;
+	}
+
+	dev = kzalloc(sizeof(struct generic_dev), GFP_KERNEL);
+	if (!dev) {
+		err = -ENOMEM;
+		goto err_alloc;
+	}
+
+	dev->info.name = "uio_pci_generic";
+	dev->info.version = "0.01";
+	dev->info.irq = pdev->irq;
+	dev->info.irq_flags = IRQF_SHARED;
+	dev->info.handler = irqhandler;
+	dev->info.irqcontrol = irqcontrol;
Quoted 8 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+	dev->pdev = pdev;
+	spin_lock_init(&dev->lock)

I know it's not strictly needed for PCI pass through, but it would be useful to register the IO regions via UIO. The userspace implementation would then use UIO strictly instead of poking the sysfs pci info directly. I think that ends up being cleaner.

I don't see what the advantage is?

thanks,
-chris

#6 lore raw
Quoted 8 lines · Chris Wright
Quoted 5 lines

I know it's not strictly needed for PCI pass through, but it would be useful to register the IO regions via UIO. The userspace implementation would then use UIO strictly instead of poking the sysfs pci info directly. I think that ends up being cleaner.

I don't see what the advantage is?

Have a single fd represent the assigned device, so security can be concentrated at that point. Some privileged server can then hand out the fd to qemu.

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
#7 lore raw
In reply to Michael S. Tsirkin
Quoted 1 line · * Michael S. Tsirkin ([email protected]) wrote:

+struct generic_dev {

I know I commented on this one on an earlier, private version, and naming is not my strength... maybe "struct uio_generic_pci_dev" or "struct uio_generic_pci"?

drivers/uio/uio_pci_generic.c line 37

Quoted 9 lines
drivers/uio/uio_pci_generic.c new file +9 -0
+	struct uio_info info;
+	struct pci_dev *pdev;
+	spinlock_t lock; /* guards command register accesses */
+};
+
+/* Read/modify/write command register to disable interrupts.
+ * Note: we could cache the value and optimize the read if there was a way to
+ * get notified of user changes to command register through sysfs.
+ * */

For the irqcontrol case, I don't think RMW is a problem (coming from userspace it's already a slower path).

For the irqhandler case, you can grab the full dword to get Command+Status (since you needed status anyway, and config reads are dword).

Quoted 38 lines
drivers/uio/uio_pci_generic.c new file +38 -0
+static void irqtoggle(struct generic_dev *dev, int irq_on)
+{
+	struct pci_dev *pdev = dev->pdev;
+	unsigned long flags;
+	u16 orig, new;
+
+	spin_lock_irqsave(&dev->lock, flags);
+	pci_block_user_cfg_access(pdev);
+	pci_read_config_word(pdev, PCI_COMMAND, &orig);
+	new = irq_on ? orig & ~PCI_COMMAND_INTX_DISABLE :
+		orig | PCI_COMMAND_INTX_DISABLE;
+	if (new != orig)
+		pci_write_config_word(dev->pdev, PCI_COMMAND, new);
+	pci_unblock_user_cfg_access(dev);
+	spin_unlock_irqrestore(&dev->lock, flags);
+}
+
+/* irqcontrol is use by userspace to enable/disable interrupts. */
+static int irqcontrol(struct uio_info *info, s32 irq_on)
+{
+	struct generic_dev *dev = container_of(info, struct generic_dev, info);
+	irqtoggle(dev, irq_on);
+	return 0;
+}
+
+static irqreturn_t irqhandler(int irq, struct uio_info *info)
+{
+	struct generic_dev *dev = container_of(info, struct generic_dev, info);
+	irqreturn_t ret = IRQ_NONE;
+	u16 status;
+
+	/* Check interrupt status register to see whether our device
+	 * triggered the interrupt. */
+	pci_read_config_word(dev->pdev, PCI_STATUS, &status);
+	if (!(status & PCI_STATUS_INTERRUPT))
+		goto done;
+
+	/* We triggered the interrupt, disable it. */

drivers/uio/uio_pci_generic.c line 85

Quoted 10 lines
drivers/uio/uio_pci_generic.c new file +10 -0
+	irqtoggle(dev, 0);
+	/* UIO core will signal the user process. */
+	ret = IRQ_HANDLED;
+done:
+	return ret;
+}
+
+/* Verify that the device supports Interrupt Disable bit in command register,
+ * per PCI 2.3, by flipping this bit and reading it back: this bit was readonly
+ * in PCI 2.2. */

Wonder if this should also restrict from things like bridges?

Quoted 55 lines
drivers/uio/uio_pci_generic.c new file +55 -0
+static int __devinit verify_pci_2_3(struct pci_dev *pdev)
+{
+	u16 orig, new;
+	int err = 0;
+
+	pci_block_user_cfg_access(pdev);
+	pci_read_config_word(pdev, PCI_COMMAND, &orig);
+	pci_write_config_word(pdev, PCI_COMMAND,
+			      orig ^ PCI_COMMAND_INTX_DISABLE);
+	pci_read_config_word(pdev, PCI_COMMAND, &new);
+	/* There's no way to protect against
+	 * hardware bugs or detect them reliably, but as long as we know
+	 * what the value should be, let's go ahead and check it. */
+	if ((new ^ orig) & ~PCI_COMMAND_INTX_DISABLE) {
+		err = -EBUSY;
+		dev_err(&pdev->dev, "Command changed from 0x%x to 0x%x: "
+			"driver or HW bug?\n", orig, new);
+		goto err;
+	}
+	if (!((new ^ orig) & PCI_COMMAND_INTX_DISABLE)) {
+		dev_warn(&pdev->dev, "Device does not support "
+			 "disabling interrupts: unable to bind.\n");
+		err = -ENODEV;
+		goto err;
+	}
+	/* Now restore the original value. */
+	pci_write_config_word(pdev, PCI_COMMAND, orig);
+err:
+	pci_unblock_user_cfg_access(pdev);
+	return err;
+}
+
+static int __devinit probe(struct pci_dev *pdev,
+			   const struct pci_device_id *id)
+{
+	struct generic_dev *dev;
+	int err;
+
+	err = pci_enable_device(pdev);
+	if (err) {
+		dev_err(&pdev->dev, "%s: pci_enable_device failed: %d\n",
+			 __func__, err);
+		return err;
+	}
+
+	err = verify_pci_2_3(pdev);
+	if (err)
+		goto err_verify;
+
+	err = pci_request_regions(pdev, "uio_pci_generic");
+	if (err) {
+		dev_err(&pdev->dev, "%s: pci_request_regions failed: %d\n",
+			 __func__, err);
+		goto err_verify;
+	}

drivers/uio/uio_pci_generic.c line 150

Quoted 10 lines
drivers/uio/uio_pci_generic.c new file +10 -0
+
+	dev = kzalloc(sizeof(struct generic_dev), GFP_KERNEL);
+	if (!dev) {
+		err = -ENOMEM;
+		goto err_alloc;
+	}
+
+	dev->info.name = "uio_pci_generic";
+	dev->info.version = "0.01";
+	dev->info.irq = pdev->irq;

May need to verify pdev->irq in verify too?

drivers/uio/uio_pci_generic.c line 157

Quoted 7 lines
drivers/uio/uio_pci_generic.c new file +7 -0
+	dev->info.irq_flags = IRQF_SHARED;
+	dev->info.handler = irqhandler;
+	dev->info.irqcontrol = irqcontrol;
+	dev->pdev = pdev;
+	spin_lock_init(&dev->lock);
+
+	pci_reset_function(pdev);

I think this could be a bit much, since it will fall back to secondary bus reset.

thanks,
-chris

author maintainer
#8 lore raw
Quoted 6 lines · Chris Wright
Quoted 1 line · * Michael S. Tsirkin ([email protected]) wrote:

+struct generic_dev {

I know I commented on this one on an earlier, private version, and naming is not my strength... maybe "struct uio_generic_pci_dev" or "struct uio_generic_pci"?

Hmm, I'd like to keep it short. the struct is private in file after all. Longer name will not let me init variables of this type on the same line. generic_dev is enough to make grep/find in file happy?

Quoted 4 lines
Quoted 4 lines
drivers/uio/uio_pci_generic.c new file +4 -0
+	struct uio_info info;
+	struct pci_dev *pdev;
+	spinlock_t lock; /* guards command register accesses */
+};
Quoted 8 lines

drivers/uio/uio_pci_generic.c line 37

Quoted 5 lines
drivers/uio/uio_pci_generic.c new file +5 -0
+
+/* Read/modify/write command register to disable interrupts.
+ * Note: we could cache the value and optimize the read if there was a way to
+ * get notified of user changes to command register through sysfs.
+ * */

For the irqcontrol case, I don't think RMW is a problem (coming from userspace it's already a slower path).

I still expect it to be noticeable ...

Quoted 2 lines

For the irqhandler case, you can grab the full dword to get Command+Status (since you needed status anyway, and config reads are dword).

Good point.

Quoted 42 lines
Quoted 42 lines
drivers/uio/uio_pci_generic.c new file +42 -0
+static void irqtoggle(struct generic_dev *dev, int irq_on)
+{
+	struct pci_dev *pdev = dev->pdev;
+	unsigned long flags;
+	u16 orig, new;
+
+	spin_lock_irqsave(&dev->lock, flags);
+	pci_block_user_cfg_access(pdev);
+	pci_read_config_word(pdev, PCI_COMMAND, &orig);
+	new = irq_on ? orig & ~PCI_COMMAND_INTX_DISABLE :
+		orig | PCI_COMMAND_INTX_DISABLE;
+	if (new != orig)
+		pci_write_config_word(dev->pdev, PCI_COMMAND, new);
+	pci_unblock_user_cfg_access(dev);
+	spin_unlock_irqrestore(&dev->lock, flags);
+}
+
+/* irqcontrol is use by userspace to enable/disable interrupts. */
+static int irqcontrol(struct uio_info *info, s32 irq_on)
+{
+	struct generic_dev *dev = container_of(info, struct generic_dev, info);
+	irqtoggle(dev, irq_on);
+	return 0;
+}
+
+static irqreturn_t irqhandler(int irq, struct uio_info *info)
+{
+	struct generic_dev *dev = container_of(info, struct generic_dev, info);
+	irqreturn_t ret = IRQ_NONE;
+	u16 status;
+
+	/* Check interrupt status register to see whether our device
+	 * triggered the interrupt. */
+	pci_read_config_word(dev->pdev, PCI_STATUS, &status);
+	if (!(status & PCI_STATUS_INTERRUPT))
+		goto done;
+
+	/* We triggered the interrupt, disable it. */
+	irqtoggle(dev, 0);
+	/* UIO core will signal the user process. */
+	ret = IRQ_HANDLED;
+done:
Quoted 8 lines

drivers/uio/uio_pci_generic.c line 85

Quoted 6 lines
drivers/uio/uio_pci_generic.c new file +6 -0
+	return ret;
+}
+
+/* Verify that the device supports Interrupt Disable bit in command register,
+ * per PCI 2.3, by flipping this bit and reading it back: this bit was readonly
+ * in PCI 2.2. */

Wonder if this should also restrict from things like bridges?

Nope, why should it? If the device deasserts interrupt, the bridge does not care about the reason.

Quoted 59 lines
Quoted 59 lines
drivers/uio/uio_pci_generic.c new file +59 -0
+static int __devinit verify_pci_2_3(struct pci_dev *pdev)
+{
+	u16 orig, new;
+	int err = 0;
+
+	pci_block_user_cfg_access(pdev);
+	pci_read_config_word(pdev, PCI_COMMAND, &orig);
+	pci_write_config_word(pdev, PCI_COMMAND,
+			      orig ^ PCI_COMMAND_INTX_DISABLE);
+	pci_read_config_word(pdev, PCI_COMMAND, &new);
+	/* There's no way to protect against
+	 * hardware bugs or detect them reliably, but as long as we know
+	 * what the value should be, let's go ahead and check it. */
+	if ((new ^ orig) & ~PCI_COMMAND_INTX_DISABLE) {
+		err = -EBUSY;
+		dev_err(&pdev->dev, "Command changed from 0x%x to 0x%x: "
+			"driver or HW bug?\n", orig, new);
+		goto err;
+	}
+	if (!((new ^ orig) & PCI_COMMAND_INTX_DISABLE)) {
+		dev_warn(&pdev->dev, "Device does not support "
+			 "disabling interrupts: unable to bind.\n");
+		err = -ENODEV;
+		goto err;
+	}
+	/* Now restore the original value. */
+	pci_write_config_word(pdev, PCI_COMMAND, orig);
+err:
+	pci_unblock_user_cfg_access(pdev);
+	return err;
+}
+
+static int __devinit probe(struct pci_dev *pdev,
+			   const struct pci_device_id *id)
+{
+	struct generic_dev *dev;
+	int err;
+
+	err = pci_enable_device(pdev);
+	if (err) {
+		dev_err(&pdev->dev, "%s: pci_enable_device failed: %d\n",
+			 __func__, err);
+		return err;
+	}
+
+	err = verify_pci_2_3(pdev);
+	if (err)
+		goto err_verify;
+
+	err = pci_request_regions(pdev, "uio_pci_generic");
+	if (err) {
+		dev_err(&pdev->dev, "%s: pci_request_regions failed: %d\n",
+			 __func__, err);
+		goto err_verify;
+	}
+
+	dev = kzalloc(sizeof(struct generic_dev), GFP_KERNEL);
+	if (!dev) {
+		err = -ENOMEM;
Quoted 8 lines

drivers/uio/uio_pci_generic.c line 150

Quoted 6 lines
drivers/uio/uio_pci_generic.c new file +6 -0
+		goto err_alloc;
+	}
+
+	dev->info.name = "uio_pci_generic";
+	dev->info.version = "0.01";
+	dev->info.irq = pdev->irq;

May need to verify pdev->irq in verify too?

Good point. Although I am not sure why might irq be 0, but from code this seems possible.

Quoted 11 lines

drivers/uio/uio_pci_generic.c line 157

Quoted 7 lines
drivers/uio/uio_pci_generic.c new file +7 -0
+	dev->info.irq_flags = IRQF_SHARED;
+	dev->info.handler = irqhandler;
+	dev->info.irqcontrol = irqcontrol;
+	dev->pdev = pdev;
+	spin_lock_init(&dev->lock);
+
+	pci_reset_function(pdev);

I think this could be a bit much,
since it will fall back to secondary
bus reset.

The reason I put it here, is because with userspace driver the device might be left in a very strange state if you kill the driver abruptly. With uio I don't currently get notification on open (which would be ideal) so user will have to unbind/rebind to reset the device. And btw, an unfortunate side affect that I didn't realise is that this actually is restricted to PCI express devices.

So, what I think I will do is add a patch in pci-sysfs.c with a sysfs entry that resets the device. Should be pretty useful generally. Makes sense?

Quoted 2 lines

thanks,
-chris