Skip to contentSkip to conversation

PCI: hv: Fix NUMA node assignment when kernel boots with custom NUMA topology

PATCH In review [email protected] 4 participants 5 messages Jan 27, 2022 – Feb 3, 2022

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 PCI: hv: Fix NUMA node assignment when kernel boots with custom NUMA topology

From: Long Li <[email protected]>

When kernel boots with a NUMA topology with some NUMA nodes offline, the PCI driver should only set an online NUMA node on the device. This can happen during KDUMP where some NUMA nodes are not made online by the KDUMP kernel.

This patch also fixes the case where kernel is booting with "numa=off".

Fixes 999dd956d838 ("PCI: hv: Add support for protocol 1.3 and support PCI_BUS_RELATIONS2")

---
Change log:
v2: use numa_map_to_online_node() to assign a node to device (suggested by Michael Kelly <[email protected]>)
v3: add "Fixes" and check for num_possible_nodes()
v4: fix commit message format

 drivers/pci/controller/pci-hyperv.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
drivers/pci/controller/pci-hyperv.c +11 -2
index 20ea2ee330b8..ae0bc2fee4ca 100644
@@ -2155,8 +2155,17 @@ static void hv_pci_assign_numa_node(struct hv_pcibus_device *hbus)
 		if (!hv_dev)
 			continue;
 
-		if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY)
-			set_dev_node(&dev->dev, hv_dev->desc.virtual_numa_node);
+		if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY &&
+		    hv_dev->desc.virtual_numa_node < num_possible_nodes())
+			/*
+			 * The kernel may boot with some NUMA nodes offline
+			 * (e.g. in a KDUMP kernel) or with NUMA disabled via
+			 * "numa=off". In those cases, adjust the host provided
+			 * NUMA node to a valid NUMA node used by the kernel.
+			 */
+			set_dev_node(&dev->dev,
+				     numa_map_to_online_node(
+					     hv_dev->desc.virtual_numa_node));
 
 		put_pcichild(hv_dev);
 	}
Signature · 1 line
2.25.1
Reviewed-by Michael Kelley Tested-by on v4 1/1
#1 lore raw

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, January 27, 2022 7:14 AM
To: [email protected]; [email protected]; [email protected]; Purna Pavan Chandra Aekkaladevi <[email protected]>
Cc: Long Li <[email protected]>
Subject: [EXTERNAL] [Patch v4] PCI: hv: Fix NUMA node assignment when kernel boots with custom NUMA topology

[You don't often get email from [email protected]. Learn why this is important at http://aka.ms/LearnAboutSenderIdentification.]

From: Long Li <[email protected]>

When kernel boots with a NUMA topology with some NUMA nodes offline, the PCI driver should only set an online NUMA node on the device. This can happen during KDUMP where some NUMA nodes are not made online by the KDUMP kernel.

This patch also fixes the case where kernel is booting with "numa=off".

Fixes 999dd956d838 ("PCI: hv: Add support for protocol 1.3 and support PCI_BUS_RELATIONS2")

---
Change log:
v2: use numa_map_to_online_node() to assign a node to device (suggested by Michael Kelly <[email protected]>)
v3: add "Fixes" and check for num_possible_nodes()
v4: fix commit message format

 drivers/pci/controller/pci-hyperv.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
drivers/pci/controller/pci-hyperv.c +12 -2
index 20ea2ee330b8..ae0bc2fee4ca 100644
@@ -2155,8 +2155,17 @@ static void hv_pci_assign_numa_node(struct hv_pcibus_device *hbus)
                if (!hv_dev)
                        continue;
 
-               if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY)
-                       set_dev_node(&dev->dev, hv_dev->desc.virtual_numa_node);
+               if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY &&
+                   hv_dev->desc.virtual_numa_node < num_possible_nodes())
+                       /*
+                        * The kernel may boot with some NUMA nodes offline
+                        * (e.g. in a KDUMP kernel) or with NUMA disabled via
+                        * "numa=off". In those cases, adjust the host provided
+                        * NUMA node to a valid NUMA node used by the kernel.
+                        */
+                       set_dev_node(&dev->dev,
+                                    numa_map_to_online_node(
+                                            
+ hv_dev->desc.virtual_numa_node));
 
                put_pcichild(hv_dev);
        }

--
2.25.1

maintainer
#2 lore raw
In reply to [email protected]
Quoted 11 lines · On Wed, Jan 26, 2022 at 05:43:34PM -0800, [email protected] wrote:

From: Long Li <[email protected]>

When kernel boots with a NUMA topology with some NUMA nodes offline, the PCI driver should only set an online NUMA node on the device. This can happen during KDUMP where some NUMA nodes are not made online by the KDUMP kernel.

This patch also fixes the case where kernel is booting with "numa=off".

Fixes 999dd956d838 ("PCI: hv: Add support for protocol 1.3 and support PCI_BUS_RELATIONS2")

CC PCI maintainers.

Hi Bjorn and Lorenzo

Are you going to pick this up? This is a fix we would like to see upstream sooner rather than later.

I can pick this up via hyperv-fixes if that suits you, since this only affect pci-hyperv.c.

Thanks,
Wei.

Quoted 37 lines

---
Change log:
v2: use numa_map_to_online_node() to assign a node to device (suggested by Michael Kelly <[email protected]>)
v3: add "Fixes" and check for num_possible_nodes()
v4: fix commit message format

 drivers/pci/controller/pci-hyperv.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
drivers/pci/controller/pci-hyperv.c +11 -2
index 20ea2ee330b8..ae0bc2fee4ca 100644
@@ -2155,8 +2155,17 @@ static void hv_pci_assign_numa_node(struct hv_pcibus_device *hbus)
 		if (!hv_dev)
 			continue;
 
-		if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY)
-			set_dev_node(&dev->dev, hv_dev->desc.virtual_numa_node);
+		if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY &&
+		    hv_dev->desc.virtual_numa_node < num_possible_nodes())
+			/*
+			 * The kernel may boot with some NUMA nodes offline
+			 * (e.g. in a KDUMP kernel) or with NUMA disabled via
+			 * "numa=off". In those cases, adjust the host provided
+			 * NUMA node to a valid NUMA node used by the kernel.
+			 */
+			set_dev_node(&dev->dev,
+				     numa_map_to_online_node(
+					     hv_dev->desc.virtual_numa_node));
 
 		put_pcichild(hv_dev);
 	}
Signature · 1 line
2.25.1
Acked-by on v4 1/1
#3 lore raw
In reply to [email protected]
Quoted 18 lines · On Wed, Jan 26, 2022 at 05:43:34PM -0800, [email protected] wrote:

From: Long Li <[email protected]>

When kernel boots with a NUMA topology with some NUMA nodes offline, the PCI driver should only set an online NUMA node on the device. This can happen during KDUMP where some NUMA nodes are not made online by the KDUMP kernel.

This patch also fixes the case where kernel is booting with "numa=off".

Fixes 999dd956d838 ("PCI: hv: Add support for protocol 1.3 and support PCI_BUS_RELATIONS2")

---
Change log:
v2: use numa_map_to_online_node() to assign a node to device (suggested by Michael Kelly <[email protected]>)
v3: add "Fixes" and check for num_possible_nodes()
v4: fix commit message format

Quoted 2 lines
 drivers/pci/controller/pci-hyperv.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

Feel free to pick it up:

Quoted 27 lines
drivers/pci/controller/pci-hyperv.c +11 -2
index 20ea2ee330b8..ae0bc2fee4ca 100644
@@ -2155,8 +2155,17 @@ static void hv_pci_assign_numa_node(struct hv_pcibus_device *hbus)
 		if (!hv_dev)
 			continue;
 
-		if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY)
-			set_dev_node(&dev->dev, hv_dev->desc.virtual_numa_node);
+		if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY &&
+		    hv_dev->desc.virtual_numa_node < num_possible_nodes())
+			/*
+			 * The kernel may boot with some NUMA nodes offline
+			 * (e.g. in a KDUMP kernel) or with NUMA disabled via
+			 * "numa=off". In those cases, adjust the host provided
+			 * NUMA node to a valid NUMA node used by the kernel.
+			 */
+			set_dev_node(&dev->dev,
+				     numa_map_to_online_node(
+					     hv_dev->desc.virtual_numa_node));
 
 		put_pcichild(hv_dev);
 	}
Signature · 1 line
2.25.1
maintainer
#4 lore raw
Quoted 19 lines · Lorenzo Pieralisi
Quoted 18 lines · On Wed, Jan 26, 2022 at 05:43:34PM -0800, [email protected] wrote:

From: Long Li <[email protected]>

When kernel boots with a NUMA topology with some NUMA nodes offline, the PCI driver should only set an online NUMA node on the device. This can happen during KDUMP where some NUMA nodes are not made online by the KDUMP kernel.

This patch also fixes the case where kernel is booting with "numa=off".

Fixes 999dd956d838 ("PCI: hv: Add support for protocol 1.3 and support PCI_BUS_RELATIONS2")

---
Change log:
v2: use numa_map_to_online_node() to assign a node to device (suggested by Michael Kelly <[email protected]>)
v3: add "Fixes" and check for num_possible_nodes()
v4: fix commit message format

Quoted 6 lines
Quoted 2 lines
 drivers/pci/controller/pci-hyperv.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

Feel free to pick it up:

Thanks Lorenzo.