Fix northbridge quirk to assign correct NUMA node

For systems with multiple servers and routed fabric, all northbridges get assigned to the first server. Fix this by also using the node reported from the PCI bus. For single-fabric systems, the northbriges are on PCI bus 0 by definition, which are on NUMA node 0 by definition, so this is invarient on most systems.

Tested on fam10h and fam15h single and multi-fabric systems and candidate for stable.

Quoted 5 lines · Daniel J Blueman

For systems with multiple servers and routed fabric, all northbridges get assigned to the first server. Fix this by also using the node reported from the PCI bus. For single-fabric systems, the northbriges are on PCI bus 0 by definition, which are on NUMA node 0 by definition, so this is invarient on most systems.

Yeah, I think this is of very low risk for !Numascale setups. :-) So

Acked-by Borislav Petkov <[email protected]>

Quoted 2 lines

Tested on fam10h and fam15h single and multi-fabric systems and candidate for stable.

I'm not sure about it - this is only reporting the wrong node, right? Does anything depend on that node setting being correct and breaks due to this?

For systems with multiple servers and routed fabric, all northbridges get assigned to the first server. Fix this by also using the node reported from the PCI bus. For single-fabric systems, the northbriges are on PCI bus 0 by definition, which are on NUMA node 0 by definition, so this is invarient on most systems.

Tested on fam10h and fam15h single and multi-fabric systems and candidate for stable.

Signed-off-by Daniel J Blueman <[email protected]>

Acked-by Steffen Persvold <[email protected]>

---
 arch/x86/kernel/quirks.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

arch/x86/kernel/quirks.c +1 -2

index 04ee1e2..52dbf1e 100644
@@ -529,7 +529,7 @@ static void quirk_amd_nb_node(struct pci_dev *dev)
                return;
 
        pci_read_config_dword(nb_ht, 0x60, &val);
-       node = val & 7;
+       node = pcibus_to_node(dev->bus) | (val & 7);
        /*
         * Some hardware may return an invalid node ID,
         * so check it first:
--

1.8.3.2

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

[+cc linux-pci, Myron, Suravee, Kim, Aravind]

Quoted 6 lines · Daniel J Blueman

For systems with multiple servers and routed fabric, all northbridges get assigned to the first server. Fix this by also using the node reported from the PCI bus. For single-fabric systems, the northbriges are on PCI bus 0 by definition, which are on NUMA node 0 by definition, so this is invarient on most systems.

Quoted 2 lines

Tested on fam10h and fam15h single and multi-fabric systems and candidate for stable.

I wish this had been cc'd to linux-pci. We're talking about a related change by Suravee there. In fact, we were hoping this quirk could be removed altogether.

I don't understand what this quirk is doing. Normally we discover the NUMA node for a PCI host bridge via the ACPI _PXM method. The way _PXM works is that every PCI device in the hierarchy below the bridge inherits the same node number as the host bridge. I first thought this might be a workaround for a system that lacks _PXM, but I don't think that can be right, because you're only changing the node for a few devices, not the whole hierarchy.

So I suspect the problem is more complicated, and maybe _PXM is insufficient to describe the topology? Are there subtrees that should have nodes different from the host bridge?

I know this patch is already in v3.14-rc7, but I'd still like to understand it so we can do the right thing with Suravee's patch.

Quoted 11 lines · Bjorn Helgaas

Quoted 9 lines · Daniel J Blueman

For systems with multiple servers and routed fabric, all northbridges get assigned to the first server. Fix this by also using the node reported from
the PCI bus. For single-fabric systems, the northbriges are on PCI bus 0 by definition, which are on NUMA node 0 by definition, so this is invarient
on most systems.

Quoted 6 lines

Quoted 2 lines

Tested on fam10h and fam15h single and multi-fabric systems and candidate for stable.

So I suspect the problem is more complicated, and maybe _PXM is insufficient to describe the topology? Are there subtrees that should have nodes different from the host bridge?

Yes; see below.
...
The _PXM method associates each northbridge with the first NUMA node, 0 in single-fabric systems, and eg 4 for the second server in a multi-fabric system with 2 dual-module Opterons (with 2 NUMA nodes internally) etc, since the northbridges appear in the PCI tree, under the host bridge, not above it [1].

With _PXM, the rest of the PCI bus hierarchy has the right NUMA node associated, but the northbridge PCI devices should be associated with their actual NUMA node, 0, 1, 2, 3 for the first server in this example. The quirk fixes this up; irqbalance at least uses this NUMA data exposed in /sys.

The alternative to the quirk may be to explicitly express the northbridge PCI devices in the AML with their own _PXM methods. If it's valid, it may be the honest approach, though the quirk may be needed for most BIOSs; I can check the AML on a few servers to confirm if helpful.

[1] http://quora.org/2014/lspci.txt

[+cc Rafael, linux-acpi for _PXM questions]

Quoted 27 lines · Daniel J Blueman

Quoted 11 lines · Bjorn Helgaas

Quoted 9 lines · Daniel J Blueman

For systems with multiple servers and routed fabric, all northbridges get assigned to the first server. Fix this by also using the node reported from
the PCI bus. For single-fabric systems, the northbriges are on PCI bus 0 by definition, which are on NUMA node 0 by definition, so this is invarient
on most systems.

Quoted 6 lines

Quoted 2 lines

Tested on fam10h and fam15h single and multi-fabric systems and candidate for stable.

So I suspect the problem is more complicated, and maybe _PXM is insufficient to describe the topology? Are there subtrees that should have nodes different from the host bridge?

Yes; see below.
...
The _PXM method associates each northbridge with the first NUMA node, 0 in single-fabric systems, and eg 4 for the second server in a multi-fabric system with 2 dual-module Opterons (with 2 NUMA nodes internally) etc, since the northbridges appear in the PCI tree, under the host bridge, not above it [1].

Quoted 4 lines

With _PXM, the rest of the PCI bus hierarchy has the right NUMA node associated, but the northbridge PCI devices should be associated with their actual NUMA node, 0, 1, 2, 3 for the first server in this example. The quirk fixes this up; irqbalance at least uses this NUMA data exposed in /sys.

I'm confused about which devices we're talking about. We currently look at _PXM for PNP0A08 (and PNP0A03) ACPI devices. The resulting node is associated with every PCI device we enumerate below the PNP0A08 bridge. This association is made in pci_device_add().

When you say "northbridge PCI devices should be associated with their actual NUMA node," I assume you mean the 00:18.x and 00:19.x devices ("AMD Family 10h Processor ..."), since those seem to be what the quirk applies to. You are *not* talking about 00:00.0 ("ATI RD890 Northbridge"), right?

You mention irqbalance; is the NUMA node information for the 00:18.x and 00:19.x devices important because you get a lot of interrupts from those devices? Or is the issue with actual I/O devices (NICs, SCSI adapters, etc.)? If so, I don't see how this quirk would affect those, because the node information for them comes from the PNP0A08 bridge (in pci_device_add()), not from the 00:00.0, 00:18.x, or 00:19.x devices.

Quoted 4 lines

The alternative to the quirk may be to explicitly express the northbridge PCI devices in the AML with their own _PXM methods. If it's valid, it may be the honest approach, though the quirk may be needed for most BIOSs; I can check the AML on a few servers to confirm if helpful.

ACPI allows _PXM for any device, so this might be a possible approach.
However, it looks like Linux only pays attention to _PXM for
PNP0A08/03, CPUs, memory and IOAPICs (which seems like a Linux defect to me).

I'm really worried about the approach here:

         pci_read_config_dword(nb_ht, 0x60, &val);
         node = pcibus_to_node(dev->bus) | (val & 7);

because the pcibus_to_node() information comes indirectly from _PXM, and the "val" part comes from the hardware, and I don't think these are the same node number space. If I understand correctly, the BIOS can synthesize whatever numbers it wants for _PXM, which returns a "proximity domain," and then Linux can make up its own mapping of "proximity domain" to "logical Linux node." So I don't see why we can assume that it's valid to OR in the bits from a PCI config register to this logical Linux node number.

Quoted 9 lines · Bjorn Helgaas

[+cc linux-pci, Myron, Suravee, Kim, Aravind]

Quoted 6 lines · Daniel J Blueman

For systems with multiple servers and routed fabric, all northbridges get assigned to the first server. Fix this by also using the node reported from the PCI bus. For single-fabric systems, the northbriges are on PCI bus 0 by definition, which are on NUMA node 0 by definition, so this is invarient on most systems.

Quoted 6 lines

Quoted 2 lines

Tested on fam10h and fam15h single and multi-fabric systems and candidate for stable.

I wish this had been cc'd to linux-pci. We're talking about a related change by Suravee there. In fact, we were hoping this quirk could be removed altogether.

Noted.

Quoted 11 lines

I don't understand what this quirk is doing. Normally we discover the NUMA node for a PCI host bridge via the ACPI _PXM method. The way _PXM works is that every PCI device in the hierarchy below the bridge inherits the same node number as the host bridge. I first thought this might be a workaround for a system that lacks _PXM, but I don't think that can be right, because you're only changing the node for a few devices, not the whole hierarchy.

So I suspect the problem is more complicated, and maybe _PXM is insufficient to describe the topology? Are there subtrees that should have nodes different from the host bridge?

Yes; see below.

Quoted 2 lines

I know this patch is already in v3.14-rc7, but I'd still like to understand it so we can do the right thing with Suravee's patch.

The _PXM method associates each northbridge with the first NUMA node, 0 in single-fabric systems, and eg 4 for the second server in a multi-fabric system with 2 dual-module Opterons (with 2 NUMA nodes internally) etc, since the northbridges appear in the PCI tree, under the host bridge, not above it [1].

With _PXM, the rest of the PCI bus hierarchy has the right NUMA node associated, but the northbridge PCI devices should be associated with their actual NUMA node, 0, 1, 2, 3 for the first server in this example. The quirk fixes this up; irqbalance at least uses this NUMA data exposed in /sys.

The alternative to the quirk may be to explicitly express the northbridge PCI devices in the AML with their own _PXM methods. If it's valid, it may be the honest approach, though the quirk may be needed for most BIOSs; I can check the AML on a few servers to confirm if helpful.

Thanks,
Daniel

[1] http://quora.org/2014/lspci.txt

Quoted 39 lines · Daniel J Blueman

Quoted 14 lines · Bjorn Helgaas

[+cc linux-pci, Myron, Suravee, Kim, Aravind]

Quoted 10 lines · Daniel J Blueman

For systems with multiple servers and routed fabric, all northbridges get
assigned to the first server. Fix this by also using the node reported from
the PCI bus. For single-fabric systems, the northbriges are on PCI bus 0
by definition, which are on NUMA node 0 by definition, so this is invarient
on most systems.

Quoted 7 lines

Quoted 3 lines

Tested on fam10h and fam15h single and multi-fabric systems and candidate
for stable.

I wish this had been cc'd to linux-pci. We're talking about a related change by Suravee there. In fact, we were hoping this quirk could be removed altogether.

Noted.

Quoted 11 lines

I don't understand what this quirk is doing. Normally we discover the NUMA node for a PCI host bridge via the ACPI _PXM method. The way _PXM works is that every PCI device in the hierarchy below the bridge inherits the same node number as the host bridge. I first thought this might be a workaround for a system that lacks _PXM, but I don't think that can be right, because you're only changing the node for a few devices, not the whole hierarchy.

So I suspect the problem is more complicated, and maybe _PXM is insufficient to describe the topology? Are there subtrees that should have nodes different from the host bridge?

Yes; see below.

Quoted 8 lines

Quoted 2 lines

I know this patch is already in v3.14-rc7, but I'd still like to understand it so we can do the right thing with Suravee's patch.

The _PXM method associates each northbridge with the first NUMA node, 0 in single-fabric systems, and eg 4 for the second server in a multi-fabric system with 2 dual-module Opterons (with 2 NUMA nodes internally) etc, since the northbridges appear in the
PCI tree, under the host bridge, not above it [1].

Daniel,

That lspci looks interesting, what is the value returned from pci_bus_to_node() on your system for each fabric?

[+cc linux-pci, Myron, Suravee, Kim, Aravind]

Quoted 8 lines · Daniel J Blueman

For systems with multiple servers and routed fabric, all northbridges get assigned to the first server. Fix this by also using the node reported from the PCI bus. For single-fabric systems, the northbriges are on PCI bus 0 by definition, which are on NUMA node 0 by definition, so this is invarient on most systems.

Tested on fam10h and fam15h single and multi-fabric systems and candidate for stable.

I wish this had been cc'd to linux-pci. We're talking about a related change by Suravee there. In fact, we were hoping this quirk could be removed altogether.

I don't understand what this quirk is doing. Normally we discover the NUMA node for a PCI host bridge via the ACPI _PXM method. The way _PXM works is that every PCI device in the hierarchy below the bridge inherits the same node number as the host bridge. I first thought this might be a workaround for a system that lacks _PXM, but I don't think that can be right, because you're only changing the node for a few devices, not the whole hierarchy.

So I suspect the problem is more complicated, and maybe _PXM is insufficient to describe the topology? Are there subtrees that should have nodes different from the host bridge?

I know this patch is already in v3.14-rc7, but I'd still like to understand it so we can do the right thing with Suravee's patch.

Bjorn

Quoted 27 lines

Signed-off-by Daniel J Blueman <[email protected]>

Acked-by Steffen Persvold <[email protected]>

---
  arch/x86/kernel/quirks.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

arch/x86/kernel/quirks.c +1 -2

index 04ee1e2..52dbf1e 100644
@@ -529,7 +529,7 @@ static void quirk_amd_nb_node(struct pci_dev *dev)
                 return;
 
         pci_read_config_dword(nb_ht, 0x60, &val);
-       node = val & 7;
+       node = pcibus_to_node(dev->bus) | (val & 7);
         /*
          * Some hardware may return an invalid node ID,
          * so check it first:
--

1.8.3.2

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Bjorn,

On a typical AMD system, there are two types of host bridges:
* PCI Root Complex Host bridge (e.g. RD890, SR56xx, etc.)
* CPU Host bridge

Here is an example from a 2 sockets system:

$ lspci

The host bridge 00:00.0 is basically the PCI root complex which connects to the actual PCI bus with
PCI devices hanging off of it. However, the host bridge 00:[18,19].x are the CPU host bridges,
each of which represents a CPU node within the system. In system with single root complex,
the root complex is normally connected to node 0 (i.e. 00:18.0) via non-coherent HT (I/O) link.

Even though the CPU host bridge 00:[18,19].x is on the same bus as the PCI root complex, it should
not be using the NUMA information from the PCI root complex host bridge.

Therefore, I don't think we should be using the pcibus_to_node(dev->bus) here.
Only the "val" from pci_read_config_dword(nb_ht, 0x60, &val), should be used here.

Quoted 67 lines

[+cc linux-pci, Myron, Suravee, Kim, Aravind]

Quoted 9 lines · Daniel J Blueman

For systems with multiple servers and routed fabric, all northbridges get
assigned to the first server. Fix this by also using the node reported from
the PCI bus. For single-fabric systems, the northbriges are on PCI bus 0 by definition, which are on NUMA node 0 by definition, so this is invarient
on most systems.

Quoted 3 lines

Tested on fam10h and fam15h single and multi-fabric systems and candidate
for stable.

I wish this had been cc'd to linux-pci. We're talking about a related change by Suravee there. In fact, we were hoping this quirk could be removed altogether.

I don't understand what this quirk is doing. Normally we discover the NUMA node for a PCI host bridge via the ACPI _PXM method. The way _PXM works is that every PCI device in the hierarchy below the bridge inherits the same node number as the host bridge. I first thought this might be a workaround for a system that lacks _PXM, but I don't think that can be right, because you're only changing the node for a few devices, not the whole hierarchy.

So I suspect the problem is more complicated, and maybe _PXM is insufficient to describe the topology? Are there subtrees that should have nodes different from the host bridge?

I know this patch is already in v3.14-rc7, but I'd still like to understand it so we can do the right thing with Suravee's patch.

Bjorn

Quoted 28 lines

Signed-off-by Daniel J Blueman <[email protected]>

Acked-by Steffen Persvold <[email protected]>

---
  arch/x86/kernel/quirks.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

arch/x86/kernel/quirks.c +1 -2

index 04ee1e2..52dbf1e 100644
@@ -529,7 +529,7 @@ static void quirk_amd_nb_node(struct pci_dev *dev)
                 return;
 
         pci_read_config_dword(nb_ht, 0x60, &val);
-       node = val & 7;
+       node = pcibus_to_node(dev->bus) | (val & 7);
         /*
          * Some hardware may return an invalid node ID,
          * so check it first:
--

1.8.3.2

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/