Skip to contentSkip to conversation

pcibus_to_node() addition to topology infrastructure

PATCH In review Matthew Dobson 2 participants 6 messages Oct 30, 2002 – Oct 31, 2002 Generic include/asm header files

lore
Share & tools
lore

Matthew Dobson

author
#0 lore raw

PATCH 1/1 pcibus_to_node() addition to topology infrastructure

Linus,
Here's a patch that adds PCI busses to the list of basic topology
elements (incl. CPUs, MemBlks, & Nodes).

pcibus_to_node-2.5.44.patch

This patch adds a new topology macro: pcibus_to_node(). This will be useful to allow I/O bound processes to bind themselves to CPUs/Nodes close to the PCI busses they are communicating over.

1) Adds pcibus_to_node() macro to asm-generic/topology.h
2) Makes small modifications to NUMA-Q PCI code, mostly modifying macros.
3) Uses the macros from #2 to implement pcibus_to_node() in asm-i386/topology.h

[mcd@arrakis patches]$ diffstat api_patches/pcibus_to_node-2.5.44.patch

 arch/i386/pci/numa.c           |   33 +++++++++++++++------------------
 include/asm-generic/topology.h |    3 +++
 include/asm-i386/topology.h    |    3 +++
 3 files changed, 21 insertions(+), 18 deletions(-)

Cheers!

-Matt

author
#1 lore raw

Whoops! A patch would be nice...

Cheers!

-Matt

Quoted 13 lines · Matthew Dobson

Linus,
Here's a patch that adds PCI busses to the list of basic topology
elements (incl. CPUs, MemBlks, & Nodes).

pcibus_to_node-2.5.44.patch

This patch adds a new topology macro: pcibus_to_node(). This will be useful to allow I/O bound processes to bind themselves to CPUs/Nodes close to the PCI busses they are communicating over.

1) Adds pcibus_to_node() macro to asm-generic/topology.h
2) Makes small modifications to NUMA-Q PCI code, mostly modifying macros.
3) Uses the macros from #2 to implement pcibus_to_node() in asm-i386/topology.h

[mcd@arrakis patches]$ diffstat api_patches/pcibus_to_node-2.5.44.patch

Quoted 9 lines
 arch/i386/pci/numa.c           |   33 +++++++++++++++------------------
 include/asm-generic/topology.h |    3 +++
 include/asm-i386/topology.h    |    3 +++
 3 files changed, 21 insertions(+), 18 deletions(-)

Cheers!

-Matt

arch/i386/pci/numa.c +15 -18
diff -Nur --exclude-from=/usr/src/.dontdiff linux-2.5.44-vanilla/arch/i386/pci/numa.c linux-2.5.44-pcibus_to_node/arch/i386/pci/numa.c
@@ -5,13 +5,10 @@
 #include <linux/pci.h>
 #include <linux/init.h>
 #include "pci.h"
-
-#define BUS2QUAD(global) (mp_bus_id_to_node[global])
-#define BUS2LOCAL(global) (mp_bus_id_to_local[global])
-#define QUADLOCAL2BUS(quad,local) (quad_local_to_mp_bus_id[quad][local])
+#include <asm/topology.h>
 
 #define PCI_CONF1_MQ_ADDRESS(bus, dev, fn, reg) \
-	(0x80000000 | (BUS2LOCAL(bus) << 16) | (dev << 11) | (fn << 8) | (reg & ~3))
+	(0x80000000 | mp_bus_id_to_local[bus] << 16) | (dev << 11) | (fn << 8) | (reg & ~3))
 
 static int __pci_conf1_mq_read (int seg, int bus, int dev, int fn, int reg, int len, u32 *value)
 {

@@ -22,17 +19,17 @@
 
 	spin_lock_irqsave(&pci_config_lock, flags);
 
-	outl_quad(PCI_CONF1_MQ_ADDRESS(bus, dev, fn, reg), 0xCF8, BUS2QUAD(bus));
+	outl_quad(PCI_CONF1_MQ_ADDRESS(bus, dev, fn, reg), 0xCF8, __pcibus_to_node(bus));
 
 	switch (len) {
 	case 1:
-		*value = inb_quad(0xCFC + (reg & 3), BUS2QUAD(bus));
+		*value = inb_quad(0xCFC + (reg & 3), __pcibus_to_node(bus));
 		break;
 	case 2:
-		*value = inw_quad(0xCFC + (reg & 2), BUS2QUAD(bus));
+		*value = inw_quad(0xCFC + (reg & 2), __pcibus_to_node(bus));
 		break;
 	case 4:
-		*value = inl_quad(0xCFC, BUS2QUAD(bus));
+		*value = inl_quad(0xCFC, __pcibus_to_node(bus));
 		break;
 	}
 

@@ -50,17 +47,17 @@
 
 	spin_lock_irqsave(&pci_config_lock, flags);
 
-	outl_quad(PCI_CONF1_MQ_ADDRESS(bus, dev, fn, reg), 0xCF8, BUS2QUAD(bus));
+	outl_quad(PCI_CONF1_MQ_ADDRESS(bus, dev, fn, reg), 0xCF8, __pcibus_to_node(bus));
 
 	switch (len) {
 	case 1:
-		outb_quad((u8)value, 0xCFC + (reg & 3), BUS2QUAD(bus));
+		outb_quad((u8)value, 0xCFC + (reg & 3), __pcibus_to_node(bus));
 		break;
 	case 2:
-		outw_quad((u16)value, 0xCFC + (reg & 2), BUS2QUAD(bus));
+		outw_quad((u16)value, 0xCFC + (reg & 2), __pcibus_to_node(bus));
 		break;
 	case 4:
-		outl_quad((u32)value, 0xCFC, BUS2QUAD(bus));
+		outl_quad((u32)value, 0xCFC, __pcibus_to_node(bus));
 		break;
 	}
 

@@ -96,7 +93,7 @@
 	 */
 	int pxb, reg;
 	u8 busno, suba, subb;
-	int quad = BUS2QUAD(d->bus->number);
+	int quad = __pcibus_to_node(d->bus->number);
 
 	printk("PCI: Searching for i450NX host bridges on %s\n", d->slot_name);
 	reg = 0xd0;

@@ -106,9 +103,9 @@
 		pci_read_config_byte(d, reg++, &subb);
 		DBG("i450NX PXB %d: %02x/%02x/%02x\n", pxb, busno, suba, subb);
 		if (busno)
-			pci_scan_bus(QUADLOCAL2BUS(quad,busno), pci_root_ops, NULL);	/* Bus A */
+			pci_scan_bus(quad_local_to_mp_bus_id[quad][busno], pci_root_ops, NULL);	/* Bus A */
 		if (suba < subb)
-			pci_scan_bus(QUADLOCAL2BUS(quad,suba+1), pci_root_ops, NULL);	/* Bus B */
+			pci_scan_bus(quad_local_to_mp_bus_id[quad][suba+1], pci_root_ops, NULL);	/* Bus B */
 	}
 	pcibios_last_bus = -1;
 }

@@ -130,8 +127,8 @@
 	if (clustered_apic_mode && (numnodes > 1)) {
 		for (quad = 1; quad < numnodes; ++quad) {
 			printk("Scanning PCI bus %d for quad %d\n", 
-				QUADLOCAL2BUS(quad,0), quad);
-			pci_scan_bus(QUADLOCAL2BUS(quad,0), 
+				quad_local_to_mp_bus_id[quad][0], quad);
+			pci_scan_bus(quad_local_to_mp_bus_id[quad][0], 
 				pci_root_ops, NULL);
 		}
 	}
include/asm-generic/topology.h +3 -0
diff -Nur --exclude-from=/usr/src/.dontdiff linux-2.5.44-vanilla/include/asm-generic/topology.h linux-2.5.44-pcibus_to_node/include/asm-generic/topology.h
@@ -47,5 +47,8 @@
 #ifndef __node_to_memblk
 #define __node_to_memblk(node)		(0)
 #endif
+#ifndef __pcibus_to_node
+#define __pcibus_to_node(bus)		(0)
+#endif
 
 #endif /* _ASM_GENERIC_TOPOLOGY_H */
include/asm-i386/topology.h +3 -0
diff -Nur --exclude-from=/usr/src/.dontdiff linux-2.5.44-vanilla/include/asm-i386/topology.h linux-2.5.44-pcibus_to_node/include/asm-i386/topology.h
@@ -83,6 +83,9 @@
 /* Returns the number of the first MemBlk on Node 'node' */
 #define __node_to_memblk(node) (node)
 
+/* Returns the number of the node containing PCI bus 'bus' */
+#define __pcibus_to_node(bus) (mp_bus_id_to_node[bus])
+
 #else /* !CONFIG_X86_NUMAQ */
 /*
  * Other i386 platforms should define their own version of the
#2 lore raw
In reply to Matthew Dobson

This is a nice addition, but just FYI there are SGI systems that have PCI busses attached to more than one node. I guess for now we can just round-robin through the attached nodes for the return value.

Thanks,
Jesse

Quoted 30 lines · Matthew Dobson

Linus,
Here's a patch that adds PCI busses to the list of basic topology
elements (incl. CPUs, MemBlks, & Nodes).

pcibus_to_node-2.5.44.patch

This patch adds a new topology macro: pcibus_to_node(). This will be useful to allow I/O bound processes to bind themselves to CPUs/Nodes close to the PCI busses they are communicating over.

1) Adds pcibus_to_node() macro to asm-generic/topology.h
2) Makes small modifications to NUMA-Q PCI code, mostly modifying macros.
3) Uses the macros from #2 to implement pcibus_to_node() in asm-i386/topology.h

[mcd@arrakis patches]$ diffstat api_patches/pcibus_to_node-2.5.44.patch

 arch/i386/pci/numa.c           |   33 +++++++++++++++------------------
 include/asm-generic/topology.h |    3 +++
 include/asm-i386/topology.h    |    3 +++
 3 files changed, 21 insertions(+), 18 deletions(-)

Cheers!

(quoted diff) +0 -2
-Matt
 
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

author
#3 lore raw
Quoted 6 lines · Jesse Barnes

This is a nice addition, but just FYI there are SGI systems that have PCI busses attached to more than one node. I guess for now we can just round-robin through the attached nodes for the return value.

Thanks,
Jesse

Ah, yes... The p-bricks, i-bricks, etc. right?

Yes, I suppose a round-robin return for the SGI version of the macro would work... Certainly not ideal, but it would work. The problem is if you assume that binding several processes to the node PCI bus X is on will ensure that those processes are on the same node. In this case (SGI), it would be a false assumption. I suppose that's what you get for assuming, though? ;)

Cheers!

-Matt

#4 lore raw
Quoted 1 line · Matthew Dobson

Ah, yes... The p-bricks, i-bricks, etc. right?

Yup.

Quoted 2 lines

Yes, I suppose a round-robin return for the SGI version of the macro would work... Certainly not ideal, but it would work. The problem is

Can you think of any better way to do it? Perhaps make pcibus_to_node return a list of nodes?

Thanks,
Jesse

author
#5 lore raw
Quoted 5 lines · Jesse Barnes
Quoted 1 line · Matthew Dobson

Ah, yes... The p-bricks, i-bricks, etc. right?

Quoted 8 lines

Yup.

Quoted 1 line

Yes, I suppose a round-robin return for the SGI version of the macro would work... Certainly not ideal, but it would work. The problem is

Can you think of any better way to do it? Perhaps make pcibus_to_node return a list of nodes?

Yeah... I was thinking about having it return a bitmask... It makes it a bit of a pain to iterate through the mask, only to find that in most cases there is only one bit set in the mask. :(

It's a bit late in the day... I'll mull this over and see if I can come up with something good.

Cheers!

-Matt