PCI segment support

I'm putting this patch out for comments rather than merging, since I'm aware that it won't apply to Linus' tree any more.

- Add segment to pci_bus.
- Change the sysfs name of each device to include a 16-bit segment ID.

diff -urpNX dontdiff linux-2.5.66/drivers/pci/probe.c linux-2.5.66-laptop/drivers/pci/probe.c
@@ -510,7 +510,7 @@ pci_scan_device(struct pci_bus *bus, int
 	pci_name_device(dev);
 
 	/* now put in global tree */
-	strcpy(dev->dev.bus_id,dev->slot_name);
+	sprintf(dev->dev.bus_id, "%04x:%s", bus->segment, dev->slot_name);
 	dev->dev.dma_mask = &dev->dma_mask;
 
 	return dev;

diff -urpNX dontdiff linux-2.5.66/drivers/pcmcia/cardbus.c linux-2.5.66-laptop/drivers/pcmcia/cardbus.c
@@ -282,7 +282,8 @@ int cb_alloc(socket_info_t * s)
 
 		pci_setup_device(dev);
 
-		strcpy(dev->dev.bus_id, dev->slot_name);
+		sprintf(dev->dev.bus_id, "%04x:%s", bus->segment,
+				dev->slot_name);
 
 		/* We need to assign resources for expansion ROM. */
 		for (r = 0; r < 7; r++) {

diff -urpNX dontdiff linux-2.5.66/include/linux/pci.h linux-2.5.66-laptop/include/linux/pci.h
@@ -451,6 +451,7 @@ struct pci_bus {
 	void		*sysdata;	/* hook for sys-specific extension */
 	struct proc_dir_entry *procdir;	/* directory entry in /proc/bus/pci */
 
+	unsigned short	segment;	/* domain/segment number */
 	unsigned char	number;		/* bus number */
 	unsigned char	primary;	/* number of primary bridge */
 	unsigned char	secondary;	/* number of secondary bridge */

"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

First of all, the "segment" name is extremely misleading. PCI spec assumes everywhere that "segment" is a group of devices sitting on the same wires (ie primary and secondary buses of the PCI-to-PCI bridge are *different* segments).
Let's call it "configuration space domain" instead.

Second, why not

- strcpy(dev->dev.bus_id,dev->slot_name);
+ sprintf(dev->dev.bus_id, "%04x:%s", pci_controller_num(dev),
+ dev->slot_name);

?

Ivan.

I don't mind changing it to domain', I prefer the term myself. ACPI calls it _SEG' so I went with segment.

Because it's possible to have multiple pci root bridges in the same pci domain. This is true on at least HP's ia64 & parisc boxes.

"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

I believe each PCI controller on these ia64/parisc boxes has its own config space, and can support up to 256 bridged PCI buses, right? Whether or not these PCI controllers share the same IO or MEM space is irrelevant (because it's entirely implementation specific).

Ivan.

I think hardware _could_ work like that, but it's never set up to work like that in practice.

"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

...

yes - at least "HP Designed" chipsets do.

I interpret "practice" as equivalent to "implementation" in this context.

I hedge by saying "HP designed" because HP tried to sell Intel itanic boxes until current mckinley boxes became available. And SGI, IBM, and other vendors are designing their own "high end" ia64 chipsets. (eg NEC Azusa - 16 CPU itanic).

parisc firmware (both Legacy and PAT PDC) and current IA64 firmware number the PCI busses. I didn't want the parisc-linux implementation to reprogram PCI Bus numbers because:
o lack of PCI Segment support (which I'm glad to see willy is fixing)
o the "view" of devices from both firmware and linux should match.

card-mode dino (PCI controller on GSC HBA) is the only exception where firmware has no clue about the PCI controller.

hth,
grant

Picking up this loose thread, here's a reimplementation of PCI domain support.

- Use pci_domain_nr() macro to determine which domain a bus or device is in.
- Default implementation for architectures which don't support PCI domains.
- Implementation for ia64.

I envisage ia64 will always turn on CONFIG_PCI_DOMAINS but x86 might well have it as a user question. I suspect most architectures would never turn it on (yeah, I'm going to design an embedded ARM box with multiple PCI domains. sure.)

Index: arch/ia64/Kconfig
===================================================================
RCS file: /var/cvs/linux-2.5/arch/ia64/Kconfig,v
retrieving revision 1.12

diff -u -p -r1.12 Kconfig
@@ -543,6 +543,10 @@ config PCI
 	  information about which PCI hardware does work under Linux and which
 	  doesn't.
 
+config PCI_DOMAINS
+	bool
+	default PCI
+
 source "drivers/pci/Kconfig"
 
 config HOTPLUG

Index: arch/ia64/hp/common/sba_iommu.c
===================================================================
RCS file: /var/cvs/linux-2.5/arch/ia64/hp/common/sba_iommu.c,v retrieving revision 1.7

diff -u -p -r1.7 sba_iommu.c
@@ -1889,7 +1889,7 @@ sba_connect_bus(struct pci_bus *bus)
 		handle = parent;
 	} while (ACPI_SUCCESS(status));
 
-	printk(KERN_WARNING "No IOC for PCI Bus %02x:%02x in ACPI\n", PCI_SEGMENT(bus), bus->number);
+	printk(KERN_WARNING "No IOC for PCI Bus %04x:%02x in ACPI\n", pci_domain_nr(bus), bus->number);
 }
 
 static int __init

Index: arch/ia64/pci/pci.c
===================================================================
RCS file: /var/cvs/linux-2.5/arch/ia64/pci/pci.c,v retrieving revision 1.7

diff -u -p -r1.7 pci.c
@@ -87,14 +87,14 @@ __pci_sal_write (int seg, int bus, int d
 static int
 pci_sal_read (struct pci_bus *bus, unsigned int devfn, int where, int size, u32 *value)
 {
-	return __pci_sal_read(PCI_SEGMENT(bus), bus->number, PCI_SLOT(devfn), PCI_FUNC(devfn),
+	return __pci_sal_read(pci_domain_nr(bus), bus->number, PCI_SLOT(devfn), PCI_FUNC(devfn),
 			      where, size, value);
 }
 
 static int
 pci_sal_write (struct pci_bus *bus, unsigned int devfn, int where, int size, u32 value)
 {
-	return __pci_sal_write(PCI_SEGMENT(bus), bus->number, PCI_SLOT(devfn), PCI_FUNC(devfn),
+	return __pci_sal_write(pci_domain_nr(bus), bus->number, PCI_SLOT(devfn), PCI_FUNC(devfn),
 			       where, size, value);
 }
 

Index: drivers/pci/probe.c
===================================================================
RCS file: /var/cvs/linux-2.5/drivers/pci/probe.c,v retrieving revision 1.14

diff -u -p -r1.14 probe.c
@@ -528,7 +528,8 @@ pci_scan_device(struct pci_bus *bus, int
 	pci_name_device(dev);
 
 	/* now put in global tree */
-	strcpy(dev->dev.bus_id,dev->slot_name);
+	sprintf(dev->dev.bus_id, "%04x:%s", pci_domain_nr(bus),
+			dev->slot_name);
 	dev->dev.dma_mask = &dev->dma_mask;
 
 	return dev;

Index: include/asm-ia64/pci.h
===================================================================
RCS file: /var/cvs/linux-2.5/include/asm-ia64/pci.h,v retrieving revision 1.7

diff -u -p -r1.7 pci.h
@@ -95,7 +95,7 @@ struct pci_controller {
 };
 
 #define PCI_CONTROLLER(busdev) ((struct pci_controller *) busdev->sysdata)
-#define PCI_SEGMENT(busdev)    (PCI_CONTROLLER(busdev)->segment)
+#define pci_domain_nr(busdev)    (PCI_CONTROLLER(busdev)->segment)
 
 /* generic pci stuff */
 #include <asm-generic/pci.h>

Index: include/linux/pci.h
===================================================================
RCS file: /var/cvs/linux-2.5/include/linux/pci.h,v retrieving revision 1.16

diff -u -p -r1.16 pci.h
@@ -868,5 +868,15 @@ extern int pci_pci_problems;
 #define PCIPCI_VSFX		16
 #define PCIPCI_ALIMAGIK		32
 
+/*
+ * PCI domain support.  Sometimes called PCI segment (eg by ACPI),
+ * a PCI domain is defined to be a set of PCI busses which share
+ * configuration space.
+ */
+
+#ifndef CONFIG_PCI_DOMAINS
+#define pci_domain_nr(pdev)	0
+#endif
+
 #endif /* __KERNEL__ */
 #endif /* LINUX_PCI_H */

"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Don't be so sure. There's already ARM implementations where there are multiple PCI buses hanging off the host bridge - mostly stuff from Intel though.

Russell King ([email protected])                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

And x86 also, although its normally multiple pci host bridges hanging off an internal faster bus with a large collection of magic hardware wizardry to make it look like one

Looks good, but shouldn't we pass 'struct pci_bus *' instead of pci_dev to pci_domain_nr()?
Because another place where the domain number must be checked on is pci_bus_exists().

Ivan.

I don't think it matters, but someone may find a useful use of having the exact device available, who knows...

We could just pass the bus self device in this case.

David S. Miller <[email protected]>

Hmm. Actually the patch *does* use pci_bus. What got me confused is definition in include/linux/pci.h:

+#ifndef CONFIG_PCI_DOMAINS
+#define pci_domain_nr(pdev) 0

		       ~~~~

+#endif

I think it should be changed to 'pbus' to avoid confusion.

Root buses often do not have the self device, e.g. on alpha.

Ivan.

From: Ivan Kokshaysky <[email protected]>
Date: Mon, 9 Jun 2003 14:42:42 +0400

Root buses often do not have the self device, e.g. on alpha.

How can people make PCI config space accesses to them?

The root level controllers itself are not accessible from PCI config space (unlike x86 host bridges). They have dedicated control registers somewhere in the IO space.

Ivan.

From: Ivan Kokshaysky <[email protected]>
Date: Mon, 9 Jun 2003 15:00:18 +0400

The root level controllers itself are not accessible from PCI config space (unlike x86 host bridges). They have dedicated control registers somewhere in the IO space.

This sounds more like a PCI host controller, not the root of the actual PCI bus.

hmm, yes, well. There's a certain amount of sloppiness allowed with it being a macro, in that bus->sysdata and dev->sysdata have the same value so it works both ways. Of course, this prohibits any architecture from implementing it as a function, so we really should make up our minds which it is to be. It sounds like bus is more generally useful than dev, so I'll make that explicit.

"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Well, it's true for many architectures, but not for all. IIRC, bus->sysdata != dev->sysdata on sparc and parisc.

Great.

Ivan.

Certainly not true for parisc. It might be true for sparc; I'm not quite sure what arch/sparc/kernel/pcic.c is up to -- it seems a little bitrotten.

"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

BTW, you can include this in the next version of the patch. :-)

Ivan.

@@ -195,6 +195,9 @@ extern void
 pcibios_resource_to_bus(struct pci_dev *dev, struct pci_bus_region *region,
 			 struct resource *res);
 
+#define pci_domain_nr(pbus) ({ struct pci_controller *_hose_ = pbus->sysdata; \
+			       _hose_->index; })
+
 #endif /* __KERNEL__ */
 
 /* Values for the `which' argument to sys_pciconfig_iobase.  */

@@ -295,6 +295,10 @@ config PCI
 	  information about which PCI hardware does work under Linux and which
 	  doesn't.
 
+config PCI_DOMAINS
+	bool
+	default PCI
+
 config ALPHA_CORE_AGP
 	bool
 	depends on ALPHA_GENERIC || ALPHA_TITAN || ALPHA_MARVEL