Skip to contentSkip to conversation

PCI bridge driver rewrite (rev 02)

RFC PATCH In review Adam Belay 4 participants 7 messages Mar 7, 2005 – Mar 8, 2005 PCI

lore
Share & tools
lore

Adam Belay

author
#0 lore raw

RFC PATCH 1/1 PCI bridge driver rewrite (rev 02)

Hi all,

This is the second release of my PCI bridge driver rewrite. It is still in early development. Before the code can compile and be functional, a number of resource management changes need to be made. Also the PCI<->PCI bridge driver needs to be finished and the root bridge driver needs to be written.

With that in mind, this release makes significant progress in refining the PCI bus class API. Also it adds a few important features. Any Comments or suggestions would be appreciated.

I did some experimentation with ACPI, and plan to use it to enumerate the root bridges. Also, I plan to handle the ISA_ENABLE and VGA_ENABLE bits in a future release.

Thanks,
Adam

Changelog:

revision 02
===========
(me):
* Minor reorganization of the file structure in "/driver/pci". Moved probe.c into "/drivers/pci/bus" where it logically belongs. Created "bus-res.c" for resource management and "bus-num.c" for assigning bus numbers.
* Added some skeleton code to "bus-num.c"
* Use "struct resource" to describe bus numbers. It occurred to me that "resource.c" could be used as a general purpose library for assigning ranges. This will prevent code duplication.
* Added "struct pci_dev_bridge" to contain data about bridges when it is available. (i.e. pci bridge and carbus headers) Other capabilities, such as the PCI power management information could also be handled this way.
* Updated probe code to fill in "struct pci_dev_bridge".
* Removed "pci_derive_parent", as it was incorrect and not needed.
* Added functions to "struct pci_bus". Every bus driver should implement these. They are as follows:

int (*setup_windows) (struct pci_bus *bus);/* allocates resources */ int (*enable) (struct pci_bus *bus);/* enables the bridge */ int (*disable) (struct pci_bus *bus);/* disables the bridge */
int (*set_busnr) (struct pci_bus *bus); /* records bus number config */

Diffstat:

 include/linux/pci.h                   |   35 -
 linux-pci/drivers/pci/Makefile        |    8
 linux-pci/drivers/pci/bus/Makefile    |    5
 linux-pci/drivers/pci/bus/bus-class.c |  208 +++++++
 linux-pci/drivers/pci/bus/bus-num.c   |   44 +
 linux-pci/drivers/pci/bus/bus-probe.c |  601 +++++++++++++++++++++
 linux-pci/drivers/pci/bus/bus-res.c   |    1
 linux-pci/drivers/pci/bus/p2p.c       |  134 ++++
 linux-pci/drivers/pci/bus/root.c      |    5
 linux-pci/drivers/pci/device.c        |  143 +++++
 linux-pci/drivers/pci/pci.h           |    4
 linux-pci/drivers/pci/probe.c         |  944 ----------------------------------
 linux-pci/drivers/pci/remove.c        |  126 ----
 13 files changed, 1177 insertions(+), 1081 deletions(-)

TODO:
1. root bridge driver
2. cardbus bridge driver (YENTA)
3. update p2p bridge driver
4. cleanup pci.h and export PCI bus class functions
5. redesign resource management
6. handle bus number assignment
7. export more bus information to sysfs
8. adapt the API for PCI hotplug
9. fix "PCI_LEGACY"
10. tune registration order for optimal resource assignment
11. sanitize locking
12. improve debugging

Patch:

drivers/pci/bus/bus-class.c +208 -0
diff -urN linux/drivers/pci/bus/bus-class.c linux-pci/drivers/pci/bus/bus-class.c
@@ -0,0 +1,208 @@
+/*
+ * bus-class.c - the core of the PCI bus class driver
+ *
+ */
+
+#include <linux/init.h>
+#include <linux/pci.h>
+#include <linux/slab.h>
+#include <linux/module.h>
+
+#undef DEBUG
+
+#ifdef DEBUG
+#define DBG(x...) printk(x)
+#else
+#define DBG(x...)
+#endif
+
+
+/*
+ * PCI Bus Class
+ */
+
+#ifdef HAVE_PCI_LEGACY
+/**
+ * pci_create_legacy_files - create legacy I/O port and memory files
+ * @b: bus to create files under
+ *
+ * Some platforms allow access to legacy I/O port and ISA memory space on
+ * a per-bus basis.  This routine creates the files and ties them into
+ * their associated read, write and mmap files from pci-sysfs.c
+ */
+static void pci_create_legacy_files(struct pci_bus *b)
+{
+	b->legacy_io = kmalloc(sizeof(struct bin_attribute) * 2,
+			       GFP_ATOMIC);
+	if (b->legacy_io) {
+		memset(b->legacy_io, 0, sizeof(struct bin_attribute) * 2);
+		b->legacy_io->attr.name = "legacy_io";
+		b->legacy_io->size = 0xffff;
+		b->legacy_io->attr.mode = S_IRUSR | S_IWUSR;
+		b->legacy_io->attr.owner = THIS_MODULE;
+		b->legacy_io->read = pci_read_legacy_io;
+		b->legacy_io->write = pci_write_legacy_io;
+		class_device_create_bin_file(&b->class_dev, b->legacy_io);
+
+		/* Allocated above after the legacy_io struct */
+		b->legacy_mem = b->legacy_io + 1;
+		b->legacy_mem->attr.name = "legacy_mem";
+		b->legacy_mem->size = 1024*1024;
+		b->legacy_mem->attr.mode = S_IRUSR | S_IWUSR;
+		b->legacy_mem->attr.owner = THIS_MODULE;
+		b->legacy_mem->mmap = pci_mmap_legacy_mem;
+		class_device_create_bin_file(&b->class_dev, b->legacy_mem);
+	}
+}
+
+void pci_remove_legacy_files(struct pci_bus *b)
+{
+	class_device_remove_bin_file(&b->class_dev, b->legacy_io);
+	class_device_remove_bin_file(&b->class_dev, b->legacy_mem);
+	kfree(b->legacy_io); /* both are allocated here */
+}
+#else /* !HAVE_PCI_LEGACY */
+static inline void pci_create_legacy_files(struct pci_bus *bus) { return; }
+void pci_remove_legacy_files(struct pci_bus *bus) { return; }
+#endif /* HAVE_PCI_LEGACY */
+
+static ssize_t pci_bus_show_cpuaffinity(struct class_device *class_dev, char *buf)
+{
+	cpumask_t cpumask = pcibus_to_cpumask((to_pci_bus(class_dev))->number);
+	int ret;
+
+	ret = cpumask_scnprintf(buf, PAGE_SIZE, cpumask);
+	if (ret < PAGE_SIZE)
+		buf[ret++] = '\n';
+	return ret;
+}
+CLASS_DEVICE_ATTR(cpuaffinity, S_IRUGO, pci_bus_show_cpuaffinity, NULL);
+
+ 
+static void release_pcibus_dev(struct class_device *class_dev)
+{
+	struct pci_bus *pci_bus = to_pci_bus(class_dev);
+
+	if (pci_bus->bridge)
+		put_device(pci_bus->bridge);
+	kfree(pci_bus);
+}
+
+static struct class pcibus_class = {
+	.name		= "pci_bus",
+	.release	= &release_pcibus_dev,
+};
+
+static int __init pcibus_class_init(void)
+{
+	return class_register(&pcibus_class);
+}
+
+postcore_initcall(pcibus_class_init);
+
+
+/*
+ * Registration
+ */
+
+/**
+ *	pci_alloc_bus - allocates a "pci_bus" structure
+ */
+struct pci_bus * pci_alloc_bus(void)
+{
+	struct pci_bus *b;
+
+	b = kmalloc(sizeof(*b), GFP_KERNEL);
+	if (b) {
+		memset(b, 0, sizeof(*b));
+		INIT_LIST_HEAD(&b->node);
+		INIT_LIST_HEAD(&b->children);
+		INIT_LIST_HEAD(&b->devices);
+	}
+	return b;
+}
+
+EXPORT_SYMBOL(pci_alloc_bus);
+
+/**
+ *	pci_register_buses - registers bus or cardbus pci devices
+ *	@bus: the bus
+ */
+static void pci_register_buses(struct pci_bus *bus)
+{
+	struct pci_dev *dev;
+
+	list_for_each_entry(dev, &bus->devices, bus_list) {
+		if (dev->hdr_type == PCI_HEADER_TYPE_BRIDGE ||
+		    dev->hdr_type == PCI_HEADER_TYPE_CARDBUS)
+			pci_add_device(dev);
+	}
+}
+
+/**
+ * 	pci_register_devices - registers normal pci devices
+ * 	@bus:	the bus
+ */
+static void pci_register_devices(struct pci_bus *bus)
+{
+	struct pci_dev *dev;
+
+	list_for_each_entry(dev, &bus->devices, bus_list) {
+		if (dev->hdr_type == PCI_HEADER_TYPE_NORMAL)
+			pci_add_device(dev);
+	}
+}
+
+/**
+ *	pci_add_bus - registers a bus with the pci bus class
+ *	@bus:		the bus
+ *
+ *	This function binds a pci bus device to the pci bus class.  Every pci
+ *	bus driver is expected to register its devices with the pci subsystem
+ *	via this function.
+ */
+int pci_add_bus(struct pci_bus *bus)
+{
+	int ret;
+
+	bus->class_dev.class = &pcibus_class;
+	sprintf(bus->class_dev.class_id, "%04x:%02x", pci_domain_nr(bus), bus->primary);
+	ret = class_device_register(&bus->class_dev);
+	if (ret)
+		return ret;
+
+	class_device_create_file(&bus->class_dev, &class_device_attr_cpuaffinity);
+
+	pci_detect_children(bus);
+	pci_register_buses(bus);
+	pci_register_devices(bus);
+
+	return 0;
+}
+
+EXPORT_SYMBOL(pci_add_bus);
+
+/**
+ *	pci_remove_bus - unregisters a bus with the pci bus class
+ *	@bus:	the bus
+ *
+ *	This function removes a pci bus device from the pci bus class.  Every
+ *	pci bus driver is expected to unregister it's devices with the pci
+ *	subsystem using this function.
+ */
+void pci_remove_bus(struct pci_bus *bus)
+{
+	pci_proc_detach_bus(bus);
+
+	spin_lock(&pci_bus_lock);
+	list_del(&bus->node);
+	spin_unlock(&pci_bus_lock);
+	pci_remove_legacy_files(bus);
+	class_device_remove_file(&bus->class_dev,
+		&class_device_attr_cpuaffinity);
+	sysfs_remove_link(&bus->class_dev.kobj, "bridge");
+	class_device_unregister(&bus->class_dev);
+}
+
+EXPORT_SYMBOL(pci_remove_bus);
+
drivers/pci/bus/bus-num.c +44 -0

44 lines not on this page · raw message

drivers/pci/bus/bus-probe.c +601 -0

601 lines not on this page · raw message

drivers/pci/bus/bus-res.c +1 -0
diff -urN linux/drivers/pci/bus/bus-res.c linux-pci/drivers/pci/bus/bus-res.c
@@ -0,0 +1 @@
+
drivers/pci/bus/Makefile +5 -0
diff -urN linux/drivers/pci/bus/Makefile linux-pci/drivers/pci/bus/Makefile
@@ -0,0 +1,5 @@
+#
+# Makefile for the PCI bus class
+#
+
+obj-y :=  bus-class.o bus-num.o bus-probe.o bus-res.o p2p.o root.o
drivers/pci/bus/p2p.c +134 -0

134 lines not on this page · raw message

drivers/pci/bus/root.c +5 -0
diff -urN linux/drivers/pci/bus/root.c linux-pci/drivers/pci/bus/root.c
@@ -0,0 +1,5 @@
+/*
+ * p2p.c - a PCI bus driver for root bridges
+ *
+ */
+  
drivers/pci/device.c +143 -0

143 lines not on this page · raw message

drivers/pci/Makefile +5 -3
diff -urN linux/drivers/pci/Makefile linux-pci/drivers/pci/Makefile
@@ -2,9 +2,8 @@
 # Makefile for the PCI bus specific drivers.
 #
 
-obj-y		+= access.o bus.o probe.o remove.o pci.o quirks.o \
-			names.o pci-driver.o search.o pci-sysfs.o \
-			rom.o
+obj-y		+= access.o bus.o bus-class.o device.o pci.o quirks.o \
+		   names.o pci-driver.o search.o pci-sysfs.o rom.o
 obj-$(CONFIG_PROC_FS) += proc.o
 
 ifndef CONFIG_SPARC64
@@ -16,6 +15,9 @@
 # Build the PCI Hotplug drivers if we were asked to
 obj-$(CONFIG_HOTPLUG_PCI) += hotplug/
 
+# Build the PCI Bus class
+obj-y += bus/
+
 #
 # Some architectures use the generic PCI setup functions
 #
drivers/pci/pci.h +4 -0
diff -urN linux/drivers/pci/pci.h linux-pci/drivers/pci/pci.h
@@ -11,6 +11,10 @@
 				  void (*alignf)(void *, struct resource *,
 					  	 unsigned long, unsigned long),
 				  void *alignf_data);
+
+extern int pci_detect_children(struct pci_bus *bus);
+extern int pci_add_device(struct pci_dev *dev);
+
 /* PCI /proc functions */
 #ifdef CONFIG_PROC_FS
 extern int pci_proc_attach_device(struct pci_dev *dev);
drivers/pci/probe.c +0 -944

944 lines not on this page · raw message

drivers/pci/remove.c +0 -126

126 lines not on this page · raw message

include/linux/pci.h +27 -8
@@ -502,6 +502,16 @@
 #define PCI_D3hot	((pci_power_t __force) 3)
 #define PCI_D3cold	((pci_power_t __force) 4)
 
+struct pci_dev_bridge {
+	unsigned char	primary;	/* number of primary bridge */
+	unsigned char	secondary;	/* number of secondary bridge */
+	unsigned char	subordinate;	/* max number of subordinate buses */
+	unsigned short	bridge_ctl;	/* manage NO_ISA/FBB/et al behaviors */
+
+	unsigned int	transparent:1;	/* Transparent PCI bridge */
+	unsigned int	configured:1;	/* bridge configured (by BIOS or OS) */
+};
+
 /*
  * The pci_dev structure is used to describe PCI devices.
  */
@@ -536,6 +546,9 @@
 
 	struct	device	dev;		/* Generic device interface */
 
+	/* optional capabilities */
+	struct pci_dev_bridge *bridge_data;
+
 	/* device is compatible with these IDs */
 	unsigned short vendor_compatible[DEVICE_COUNT_COMPATIBLE];
 	unsigned short device_compatible[DEVICE_COUNT_COMPATIBLE];
@@ -552,7 +565,6 @@
 	char *		slot_name;	/* pointer to dev.bus_id */
 
 	/* These fields are used by common fixups */
-	unsigned int	transparent:1;	/* Transparent PCI bridge */
 	unsigned int	multifunction:1;/* Part of multi-function device */
 	/* keep track of device state */
 	unsigned int	is_enabled:1;	/* pci_enable_device has been called */
@@ -605,24 +617,31 @@
 	void		*sysdata;	/* hook for sys-specific extension */
 	struct proc_dir_entry *procdir;	/* directory entry in /proc/bus/pci */
 
-	unsigned char	number;		/* bus number */
-	unsigned char	primary;	/* number of primary bridge */
-	unsigned char	secondary;	/* number of secondary bridge */
-	unsigned char	subordinate;	/* max number of subordinate buses */
+	struct resource number;		/* secondary bus number */
+	struct resource busnr_range;	/* range from secondary to subordinate */
 
 	char		name[48];
 
-	unsigned short  bridge_ctl;	/* manage NO_ISA/FBB/et al behaviors */
-	unsigned short  pad2;
 	struct device		*bridge;
 	struct class_device	class_dev;
 	struct bin_attribute	*legacy_io; /* legacy I/O for this bus */
 	struct bin_attribute	*legacy_mem; /* legacy mem */
+
+	int (*setup_windows) (struct pci_bus *bus);	/* allocates resources */
+	int (*enable) (struct pci_bus *bus);		/* enables the bridge */
+	int (*disable) (struct pci_bus *bus);		/* disables the bridge */
+	int (*set_busnr) (struct pci_bus *bus);		/* records bus number config */
 };
 
 #define pci_bus_b(n)	list_entry(n, struct pci_bus, node)
 #define to_pci_bus(n)	container_of(n, struct pci_bus, class_dev)
 
+static inline unsigned char pci_bus_number(struct pci_bus *bus)
+{
+	return (unsigned char) bus->number.start;
+}
+
+
 /*
  * Error values that may be returned by PCI functions.
  */
@@ -968,7 +987,7 @@
 static inline int pci_domain_nr(struct pci_bus *bus) { return 0; }
 static inline int pci_name_bus(char *name, struct pci_bus *bus)
 {
-	sprintf(name, "%02x", bus->number);
+	sprintf(name, "%02x", pci_bus_number(bus));
 	return 0;
 }
 #endif
#1 lore raw

What about a bridge driver for ISA LPC bridges? That would also provide a logical place to hang serial ports, floppy, parallel port, ps2 port, etc. Things in /sys/bus/platform are really attached to the LPC bridge.

Signature · 2 lines
Jon Smirl
[email protected]
author
#2 lore raw
Quoted 5 lines · Jon Smirl

What about a bridge driver for ISA LPC bridges? That would also provide a logical place to hang serial ports, floppy, parallel port, ps2 port, etc. Things in /sys/bus/platform are really attached to the LPC bridge.

I agree that /sys/bus/platform isn't really the right place. It doesn't show the correct parent device, among other issues. I've done some work on this issue in the past, and it turns out to be a very complicated problem.

Basically there are many protocols that feed into the pool of devices known as *legacy hardware*. The include the following:

1.) ACPI
* provides resource information
* provides identification information
* the only protocol that accurately describes topology
* often provides power management features
* sometimes a few device specific methods (e.g. floppy drives)
2.) PnPBIOS
* outdated by ACPI, generally useful for x86 boxes before 2000
* provides resource information
* provides identification information, and a class code not found in ACPI
* all devices reported are considered root devices, no sense of true topology
* In theory could handle some hotplugging of these devices (e.g. docking stations)
3.) ISAPnP
* Even more outdated.
* Provides resource information.
* provides identification, including a card id not found in PnPBIOS or ACPI (obviously).
* Only used for ISA expansion cards
4.) SuperIO drivers
* In theory it is possible to determine configuration information from the SuperIO directly.
* Some, but very limited, work has been done in this area.
* ACPI generally handles this because there is little standardization at this level.
5.) Legacy Probing
* Driver attempts to find the hardware directly by reading various ports.
* Can be dangerous.
* Drivers of this type encourage vendors to include legacy compatibility (which in the long run holds us back).
* Very difficult to integrate with the driver model.
6.) Open Firmware
* I don't know much about it, but I believe it does do similar things to ACPI.
* Hopefully it uses EISA ids, but not really sure. If not, it wouldn't be included.

So basically we have to handle all (or most) of these. The question becomes should driver developers have to write code for all 6 of these interfaces (which seems a little overwhelming), or should they share a common layer. If so, the driver model would need a way to represent this. One idea I had was to make "buses" a special type of "class". And then allow classes to be layered. So it would look something like
ISA/LPC ->ACPI

	->PnPBIOS
	->ISAPnP
	->SuperIO
	->legacy
	->Open Firmware

Where each of the 6 classes inherit characteristics from "ISA/LPC".

A driver could then choose to bind to the more general "ISA/LPC" interfaces, or if necessary a more specific interface like "ACPI". "ISA/LPC" would be sort of a least common denominator.

Of course this would require big changes to the driver model, so it would have to be really worth it. I look forward to any comments or suggestions for alternative approaches.

Thanks,
Adam

#3 lore raw
Quoted 5 lines · Adam Belay

6.) Open Firmware
* I don't know much about it, but I believe it does do similar things to ACPI.
* Hopefully it uses EISA ids, but not really sure. If not, it wouldn't be included.

OpenFirmware is very similar to ACPI, except OpenFirmware interprets the FORTH programs in the firmware itself, whereas ACPI does it in the OS. The reason for the difference is that on Intel/x86 there is no compatible way to call directly into the firmware from 32-bit code. You won't find the two on the same system, and they basically provide the same services: device tree, power management, etc.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------

#4 lore raw
In reply to Adam Belay

How is sys/bus/platform/* going to work for IA64 machine line SGI SVN? SVN supports multiple simultaneously active legacy spaces, that means that there can be multiple floppy, serial, ps/2, etc controllers. Should these devices be hung off from the bridge they are on?

Signature · 2 lines
Jon Smirl
[email protected]
#5 lore raw
Quoted 4 lines · Jon Smirl

How is sys/bus/platform/* going to work for IA64 machine line SGI SVN? SVN supports multiple simultaneously active legacy spaces, that means that there can be multiple floppy, serial, ps/2, etc controllers. Should these devices be hung off from the bridge they are on?

Probably, though no one in their right mind is going to put anything like that on these machines (sn2 btw) :). VGA cards will hopefully be the only devices of this type that we'll have installed.

Jesse

author
#6 lore raw
Quoted 11 lines · Jesse Barnes
Quoted 4 lines · Jon Smirl

How is sys/bus/platform/* going to work for IA64 machine line SGI SVN? SVN supports multiple simultaneously active legacy spaces, that means that there can be multiple floppy, serial, ps/2, etc controllers. Should these devices be hung off from the bridge they are on?

Probably, though no one in their right mind is going to put anything like that on these machines (sn2 btw) :). VGA cards will hopefully be the only devices of this type that we'll have installed.

Jesse

Well, if the system supports ACPI, then in theory it could have any number/configuration of legacy devices, and we'd know everything about them including exactly where to put them in the device tree. However, I agree that legacy hardware will be less common in this architecture.

Adam