"Controller" pcibackend and frontend extensions

On ia64, we've run into the case where the I/O hierarchies are more
complicated than the current set of driver domain backends can describe. Some platforms make use of translation offsets for I/O port and MMIO ranges. Without knowledge of these translation offsets, devices are unusable by driver domains. For instance, here's an example of a tulip card that lives under a PCI root bus making use of an I/O port translation:

# lspci -v -s 02:05.0
02:05.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
        Subsystem: Hewlett-Packard Company Unknown device 125a
        Flags: bus master, medium devsel, latency 128, IRQ 67
        I/O ports at 2001100 [size=128]
        Memory at 90102000 (32-bit, non-prefetchable) [size=1K]
        Expansion ROM at 90080000 [disabled] [size=256K]

# setpci -s 02:05.0 BASE_ADDRESS_0
00001101

# cat /proc/ioports 
...
02000000-0200ffff : PCI Bus 0000:01
  02001000-02001fff : PCI Bus #02
    02001100-0200117f : 0000:02:05.0
      02001100-0200117f : tulip
...

# cat /proc/iomem
...
80100000000-80103ffffff : PCI Bus 0000:01 I/O Ports 02000000-0200ffff
...

I/O port spaces are of course limited to 64k, but on this system
multiple I/O port spaces are available (one per PCI root bridge in this case). On ia64, I/O port spaces are typically a sparse encoding of an MMIO range. The legacy I/O port range is decoded directly by the processor, additional ranges are decoded by the I/O hardware. To access I/O port 0x1100 on this device, the driver needs to do an inb/outb to address 0x2001100. The kernel will then swizzle the bits to create an MMIO transaction within the MMIO range for that set of I/O ports.

To support this, I've created the "controller" backend as shown
below. This is unfortunately an ia64-specific backend, but I don't see any mechanism to generically support the kinds of things this backend needs to do. PCI controllers on ia64 are created to represent the PCI root bridges found in ACPI. These root bridge ACPI nodes have _CRS (Current Resource Setting) methods that describe the address ranges consumed by the bus below the root bridge. Address ranges described with a translation attribute make use of a translation offset to reach the desired address. This information must be provided to a driver domain guest to allow it to access the devices.

Given this architecture, the obvious choice is to create virtual PCI
buses based on controllers. All devices physically under the same controller are virtualized under the same domain:bus. Within a bus, device slots are virtualized much like the slot backend. The tricky part comes with how to describe the address translation for a controller to the guest driver domain. For this, I chose to store the information in xenbus. We already make use of the following keys for driver domains:

root_num /* Number of PCI roots exposed */
root-X /* domain:bus information for root X */

To this, I've added:

root-X-resources /* number of resources for root X */
root-X-resource-Y /* resource umber Y for root X */
root-resource-magic /* synchronization/versioning for resource info */

I debated for a while how to expose the root-X-resource-Y information
and came up with a simple ASCII dump of the struct acpi_resource returned from the ACPI _CRS method. This isn't quite a silly as it sounds because the structure is a fixed size regardless of word length, and it's contents are largely based on fixed tables found in the ACPI spec. This makes it relatively immune to frequent changes. The PCI backend stores the ASCII byte stream of the controller resources into xenbus, the PCI frontend then extracts the byte stream, and decodes it back into a struct acpi_resource for use.

The only changes to the existing code to support the frontend is a
trivial addition of passing the bus number to pcifront_init_sd() and a hook to setup the root windows after the bus is scanned. No changes are required for the controller backend.

I would like to see this backend become the default backend for ia64,
but it probably needs testing on other systems before we can make the switch. I would appreciate testing and/or review feedback. To make use of extended I/O port spaces, you'll need the patch I sent to xen-ia64-devel last week to register the I/O port spaces with Xen. Thanks,

Alex

---

 b/linux-2.6-xen-sparse/drivers/xen/pciback/controller.c |  404 ++++++++++++++++
 linux-2.6-xen-sparse/arch/ia64/pci/pci.c                |   28 +
 linux-2.6-xen-sparse/drivers/xen/Kconfig                |   18 
 linux-2.6-xen-sparse/drivers/xen/pciback/Makefile       |    1 
 linux-2.6-xen-sparse/drivers/xen/pcifront/pci_op.c      |    4 
 linux-2.6-xen-sparse/drivers/xen/pcifront/pcifront.h    |    3 
 linux-2.6-xen-sparse/include/xen/pcifront.h             |  135 +++++
 7 files changed, 582 insertions(+), 11 deletions(-)

diff -r 0cf6b75423e9 linux-2.6-xen-sparse/arch/ia64/pci/pci.c
@@ -834,3 +839,31 @@ int pci_vector_resources(int last, int n
 
 	return count;
 }
+
+#ifdef CONFIG_XEN
+void __devinit xen_add_resource(struct pci_controller *controller,
+				unsigned int domain, unsigned int bus,
+				struct acpi_resource *resource)
+{
+	struct pci_root_info info;
+	char *name;
+
+	name = kmalloc(16, GFP_KERNEL);
+	if (!name)
+		return;
+
+	sprintf(name, "PCI Bus %04x:%02x", domain, bus);
+	info.controller = controller;
+	info.name = name;
+
+	add_window(resource, &info);
+}
+EXPORT_SYMBOL(xen_add_resource);
+
+void __devinit xen_pcibios_setup_root_windows(struct pci_bus *bus,
+					      struct pci_controller *controller)
+{
+	pcibios_setup_root_windows(bus, controller);
+}
+EXPORT_SYMBOL(xen_pcibios_setup_root_windows);
+#endif

diff -r 0cf6b75423e9 linux-2.6-xen-sparse/drivers/xen/Kconfig
@@ -109,7 +109,8 @@ choice
 choice
 	prompt "PCI Backend Mode"
 	depends on XEN_PCIDEV_BACKEND
-	default XEN_PCIDEV_BACKEND_VPCI
+	default XEN_PCIDEV_BACKEND_VPCI if !IA64
+	default XEN_PCIDEV_BACKEND_CONTROLLER if IA64
 
 config XEN_PCIDEV_BACKEND_VPCI
 	bool "Virtual PCI"

@@ -138,6 +139,21 @@ config XEN_PCIDEV_BACKEND_SLOT
 	  For example, a device at 03:05.2 will be re-assigned to 00:00.0. A
 	  second device at 02:1a.1 will be re-assigned to 00:01.0.
 
+config XEN_PCIDEV_BACKEND_CONTROLLER
+	bool "Controller"
+	depends on IA64
+	---help---
+	  This PCI backend virtualizes the PCI bus topology by providing a
+	  virtual bus per PCI root device.  Devices which are physically under
+	  the same root bus will appear on the same virtual bus.  For systems
+	  with complex I/O addressing, this is the only backend which supports
+	  extended I/O port spaces and MMIO translation offsets.  This backend
+	  also supports slot virtualization.  For example, a device at
+	  0000:01:02.1 will be re-assigned to 0000:00:00.0.  A second device
+	  at 0000:02:05.0 (behind a P2P bridge on bus 0000:01) will be
+	  re-assigned to 0000:00:01.0.  A third device at 0000:16:05.0 (under
+	  a different PCI root bus) will be re-assigned to 0000:01:00.0.
+
 endchoice
 
 config XEN_PCIDEV_BE_DEBUG

diff -r 0cf6b75423e9 linux-2.6-xen-sparse/drivers/xen/pciback/Makefile
@@ -9,6 +9,7 @@ pciback-$(CONFIG_XEN_PCIDEV_BACKEND_VPCI
 pciback-$(CONFIG_XEN_PCIDEV_BACKEND_VPCI) += vpci.o
 pciback-$(CONFIG_XEN_PCIDEV_BACKEND_SLOT) += slot.o
 pciback-$(CONFIG_XEN_PCIDEV_BACKEND_PASS) += passthrough.o
+pciback-$(CONFIG_XEN_PCIDEV_BACKEND_CONTROLLER) += controller.o
 
 ifeq ($(CONFIG_XEN_PCIDEV_BE_DEBUG),y)
 EXTRA_CFLAGS += -DDEBUG

diff -r 0cf6b75423e9 linux-2.6-xen-sparse/drivers/xen/pcifront/pci_op.c
@@ -207,7 +207,7 @@ int pcifront_scan_root(struct pcifront_d
 		err = -ENOMEM;
 		goto err_out;
 	}
-	pcifront_init_sd(sd, domain, pdev);
+	pcifront_init_sd(sd, domain, bus, pdev);
 
 	b = pci_scan_bus_parented(&pdev->xdev->dev, bus,
 				  &pcifront_bus_ops, sd);

@@ -217,6 +217,8 @@ int pcifront_scan_root(struct pcifront_d
 		err = -ENOMEM;
 		goto err_out;
 	}
+
+	pcifront_setup_root_resources(b, sd);
 	bus_entry->bus = b;
 
 	list_add(&bus_entry->list, &pdev->root_buses);

diff -r 0cf6b75423e9 linux-2.6-xen-sparse/drivers/xen/pcifront/pcifront.h
@@ -10,7 +10,6 @@
 #include <linux/pci.h>
 #include <xen/xenbus.h>
 #include <xen/interface/io/pciif.h>
-#include <xen/pcifront.h>
 
 struct pci_bus_entry {
 	struct list_head list;

@@ -30,6 +29,8 @@ struct pcifront_device {
 	struct xen_pci_sharedinfo *sh_info;
 };
 
+#include <xen/pcifront.h>
+
 int pcifront_connect(struct pcifront_device *pdev);
 void pcifront_disconnect(struct pcifront_device *pdev);
 

It would be nice to have just one backend type, with relevant bits of its code made arch-specific. I'm sure x86 doesn't need vpci, passthru *and* slot.

This is probably fine for now; it doesn't preclude merging or dropping other backends down the line. Only thing I spotted is that you make an enormous inline function in pcifront.h. That function will have to be moved elsewhere into a .c file if it's getting that big.

-- Keir

Hi Keir,

Thanks for looking at the patch.

I agree, but I wasn't ready to tackle that problem.

Right, I forgot about that. It is rather unwieldy for a static
inline. I guess pcifront/pci_op.c is the most appropriate place for it. I'll tuck it in a #ifdef there unless you have a better suggestion. Thanks,

Alex

Alex Williamson                             HP Open Source & Linux Org.