Skip to contentSkip to conversation

PCI: Avoid unnecessary cpu_hotplug_disable() for non-NUMA devices

PATCH Superseded weipengliang 3 participants 11 messages May 22 PCI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI: Avoid unnecessary cpu_hotplug_disable() for non-NUMA devices

When a PCI device is on an invalid/offline NUMA node or is a VF being probed from PF context, the code path calls local_pci_probe() directly without needing work_on_cpu(). In these cases, the cpu_hotplug_disable() / cpu_hotplug_enable() round-trip is unnecessary overhead.

Move cpu_hotplug_disable() into the NUMA-valid else branch so non-NUMA devices skip it entirely, returning early after local_pci_probe().

---
 drivers/pci/pci-driver.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/pci/pci-driver.c +4 -1
index e3f59001785a..d3df90f58cf4 100644
@@ -384,7 +384,6 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 	node = dev_to_node(&dev->dev);
 	dev->is_probed = 1;
 
-	cpu_hotplug_disable();
 	/*
 	 * Prevent nesting work_on_cpu() for the case where a Virtual Function
 	 * device is probed from work_on_cpu() of the Physical device.
@@ -392,9 +391,13 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 	if (node < 0 || node >= MAX_NUMNODES || !node_online(node) ||
 	    pci_physfn_is_probed(dev)) {
 		error = local_pci_probe(&ddi);
+		dev->is_probed = 0;
+		return error;
 	} else {
 		struct pci_probe_arg arg = { .ddi = &ddi };
 
+		cpu_hotplug_disable();
+
 		INIT_WORK_ONSTACK(&arg.work, local_pci_probe_callback);
 		/*
 		 * The target election and the enqueue of the work must be within
Signature · 1 line
2.47.1.windows.1
#1 lore raw
Quoted 7 lines · weipengliang

When a PCI device is on an invalid/offline NUMA node or is a VF being probed from PF context, the code path calls local_pci_probe() directly without needing work_on_cpu(). In these cases, the cpu_hotplug_disable() / cpu_hotplug_enable() round-trip is unnecessary overhead.

Move cpu_hotplug_disable() into the NUMA-valid else branch so non-NUMA devices skip it entirely, returning early after local_pci_probe().

Hm, this will retain the cpu_hotplug_enable() outside the else-branch, which isn't very readable IMHO. How about moving cpu_hotplug_enable() directly below each rcu_read_unlock() call?

Quoted 4 lines
drivers/pci/pci-driver.c +0 -0
@@ -392,9 +391,13 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 	if (node < 0 || node >= MAX_NUMNODES || !node_online(node) ||
 	    pci_physfn_is_probed(dev)) {
 		error = local_pci_probe(&ddi);

drivers/pci/pci-driver.c static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev, · line 403

Quoted 10 lines
drivers/pci/pci-driver.c +4 -0
+		dev->is_probed = 0;
+		return error;
 	} else {
 		struct pci_probe_arg arg = { .ddi = &ddi };
 
+		cpu_hotplug_disable();
+
 		INIT_WORK_ONSTACK(&arg.work, local_pci_probe_callback);
 		/*
 		 * The target election and the enqueue of the work must be within

I'd also move the cpu_hotplug_disable() below INIT_WORK_ONSTACK(). For one it minimizes the critical section, and also INIT_WORK_ONSTACK() initializes a portion of "arg" declared immediately above, so it's more readable to keep declaration + initialization together.

Thanks,

Lukas

#2 lore raw

PATCH v2 1/1 PCI: Avoid unnecessary cpu_hotplug_disable() for non-NUMA devices

When a PCI device is on an invalid/offline NUMA node or is a VF being probed from PF context, the code path calls local_pci_probe() directly without needing work_on_cpu(). In these cases, the cpu_hotplug_disable() / cpu_hotplug_enable() round-trip is unnecessary overhead.

Move cpu_hotplug_disable() into the NUMA-valid else branch so non-NUMA devices skip it entirely, returning early after local_pci_probe(). Also move cpu_hotplug_enable() directly below each rcu_read_unlock() call to keep the disable/enable symmetrical and improve readability.

Changes since v1:
- Moved cpu_hotplug_enable() directly below each rcu_read_unlock() (Lukas Wunner)
- Moved cpu_hotplug_disable() below INIT_WORK_ONSTACK() to minimize critical section and keep declaration + initialization together (Lukas Wunner)

---
 drivers/pci/pci-driver.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/pci/pci-driver.c +5 -2
index e3f59001785a..ea213555fb6c 100644
@@ -384,7 +384,6 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 	node = dev_to_node(&dev->dev);
 	dev->is_probed = 1;
 
-	cpu_hotplug_disable();
 	/*
 	 * Prevent nesting work_on_cpu() for the case where a Virtual Function
 	 * device is probed from work_on_cpu() of the Physical device.
@@ -392,10 +391,13 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 	if (node < 0 || node >= MAX_NUMNODES || !node_online(node) ||
 	    pci_physfn_is_probed(dev)) {
 		error = local_pci_probe(&ddi);
+		dev->is_probed = 0;
+		return error;
 	} else {
 		struct pci_probe_arg arg = { .ddi = &ddi };
 
 		INIT_WORK_ONSTACK(&arg.work, local_pci_probe_callback);
+		cpu_hotplug_disable();
 		/*
 		 * The target election and the enqueue of the work must be within
 		 * the same RCU read side section so that when the workqueue pool
@@ -414,10 +416,12 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 				wq = system_percpu_wq;
 			queue_work_on(cpu, wq, &arg.work);
 			rcu_read_unlock();
+			cpu_hotplug_enable();
 			flush_work(&arg.work);
 			error = arg.ret;
 		} else {
 			rcu_read_unlock();
+			cpu_hotplug_enable();
 			error = local_pci_probe(&ddi);
 		}
 
@@ -425,7 +429,6 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 	}
 
 	dev->is_probed = 0;
-	cpu_hotplug_enable();
 	return error;
 }
 
Signature · 1 line
2.47.1.windows.1
#3 lore raw

drivers/pci/pci-driver.c static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev, · line 396

Quoted 7 lines · weipengliang
drivers/pci/pci-driver.c +2 -0
@@ -392,10 +391,13 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 	if (node < 0 || node >= MAX_NUMNODES || !node_online(node) ||
 	    pci_physfn_is_probed(dev)) {
 		error = local_pci_probe(&ddi);
+		dev->is_probed = 0;
+		return error;
 	} else {

Hm, looks like these two newly added lines are unnecessary. Am I missing something?

Thanks,

Lukas

#4 lore raw
Quoted 2 lines · Lukas Wunner

Hm, looks like these two newly added lines are unnecessary. Am I missing something?

You're right. Since cpu_hotplug_enable() has been moved into the branches, the early return is no longer needed -- both paths now fall through to the common cleanup at the end.

I'll drop these two lines in v3.

Thanks,
weipengliang

#5 lore raw
In reply to Lukas Wunner

Hi Lukas,

Regarding cpu_hotplug_enable() placement in the else block: would you prefer it right after each rcu_read_unlock(), or a single call before destroy_work_on_stack() at the end? The latter keeps the code simpler while the critical section stays minimal -- flush_work() and destroy_work_on_stack() are both short and don't depend on CPU affinity.

Thanks,
weipengliang

#6 lore raw
In reply to Lukas Wunner

Hi Lukas,

I just received AI review feedback from sashiko.dev on this patch:
https://sashiko.dev/#/patchset/[email protected]?part=1

The reviewer raised a concern about enabling CPU hotplug before flush_work():

Does enabling CPU hotplug before flush_work() defeat the protection for the actual probe execution?

If the targeted CPU goes offline while flush_work() is waiting, the workqueue core might unbind the worker pool and migrate the pending probe work. Because unbound workers fall back to cpu_possible_mask, the probe might execute on an incorrect NUMA node or an isolated CPU, violating the housekeeping_cpumask constraints evaluated earlier.

Given this, I think it's better to place cpu_hotplug_enable() after destroy_work_on_stack() at the end of the else block. This keeps hotplug disabled during both flush_work() and local_pci_probe(), while the extra time during destroy_work_on_stack() is negligible. I'll update v3 accordingly.

Thanks,
weipengliang

#7 lore raw

PATCH v3 1/1 PCI: Avoid unnecessary cpu_hotplug_disable() for non-NUMA devices

When a PCI device is on an invalid/offline NUMA node or is a VF being probed from PF context, the code path calls local_pci_probe() directly without needing work_on_cpu(). In these cases, the cpu_hotplug_disable() / cpu_hotplug_enable() round-trip is unnecessary overhead.

Move cpu_hotplug_disable() into the NUMA-valid else branch so non-NUMA devices skip it entirely. Place a single cpu_hotplug_enable() after destroy_work_on_stack() to keep hotplug disabled during flush_work() and local_pci_probe(), preventing CPU offline from migrating the probe work to an incorrect NUMA node.

---
 drivers/pci/pci-driver.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/pci/pci-driver.c +2 -2
index e3f59001785a..27c1cc7c247c 100644
@@ -384,7 +384,6 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 	node = dev_to_node(&dev->dev);
 	dev->is_probed = 1;
 
-	cpu_hotplug_disable();
 	/*
 	 * Prevent nesting work_on_cpu() for the case where a Virtual Function
 	 * device is probed from work_on_cpu() of the Physical device.
@@ -396,6 +395,7 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 		struct pci_probe_arg arg = { .ddi = &ddi };
 
 		INIT_WORK_ONSTACK(&arg.work, local_pci_probe_callback);
+		cpu_hotplug_disable();
 		/*
 		 * The target election and the enqueue of the work must be within
 		 * the same RCU read side section so that when the workqueue pool
@@ -421,11 +421,11 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 			error = local_pci_probe(&ddi);
 		}
 
+		cpu_hotplug_enable();
 		destroy_work_on_stack(&arg.work);
 	}
 
 	dev->is_probed = 0;
-	cpu_hotplug_enable();
 	return error;
 }
 
Signature · 1 line
2.47.1.windows.1
#8 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Commit message discrepancy regarding the placement of cpu_hotplug_enable(). --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Commit message discrepancy regarding the placement of cpu_hotplug_enable(). --

commit 9acd77cab9fa6f681c21a1bfe522a33f79ec6605
Author: weipengliang <[email protected]>

PCI: Avoid unnecessary cpu_hotplug_disable() for non-NUMA devices

This commit limits cpu_hotplug_disable() and cpu_hotplug_enable() calls to the NUMA-valid code path. This removes overhead for non-NUMA devices where local_pci_probe() can be called directly.

Quoted 9 lines

Move cpu_hotplug_disable() into the NUMA-valid else branch so non-NUMA devices skip it entirely. Place a single cpu_hotplug_enable() after destroy_work_on_stack() to keep hotplug disabled during flush_work() and local_pci_probe(), preventing CPU offline from migrating the probe work to an incorrect NUMA node.

drivers/pci/pci-driver.c +0 -0

[ ... ]

drivers/pci/pci-driver.c static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev, · line 418

Quoted 6 lines
drivers/pci/pci-driver.c +1 -0
@@ -414,11 +414,11 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 			error = local_pci_probe(&ddi);
 		}
 
+		cpu_hotplug_enable();
 		destroy_work_on_stack(&arg.work);

[Severity: Low]
This isn't a bug, but does the commit message accurately describe the code sequence here?

The commit message mentions placing cpu_hotplug_enable() after destroy_work_on_stack(), but the code actually places it immediately before.

Could the commit message be updated to match the code, or should the code be adjusted to match the intent described in the commit message?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#9 lore raw

PATCH v4 1/1 PCI: Avoid unnecessary cpu_hotplug_disable() for non-NUMA devices

In reply to weipengliang

When a PCI device is on an invalid/offline NUMA node or is a VF being probed from PF context, the code path calls local_pci_probe() directly without needing work_on_cpu(). In these cases, the cpu_hotplug_disable() / cpu_hotplug_enable() round-trip is unnecessary overhead.

Move cpu_hotplug_disable() into the NUMA-valid else branch so non-NUMA devices skip it entirely. Place a single cpu_hotplug_enable() before destroy_work_on_stack() to keep hotplug disabled during flush_work() and local_pci_probe(), preventing CPU offline from migrating the probe work to an incorrect NUMA node.

---
 drivers/pci/pci-driver.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/pci/pci-driver.c +2 -2
index e3f59001785a..27c1cc7c247c 100644
@@ -384,7 +384,6 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 	node = dev_to_node(&dev->dev);
 	dev->is_probed = 1;
 
-	cpu_hotplug_disable();
 	/*
 	 * Prevent nesting work_on_cpu() for the case where a Virtual Function
 	 * device is probed from work_on_cpu() of the Physical device.
@@ -396,6 +395,7 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 		struct pci_probe_arg arg = { .ddi = &ddi };
 
 		INIT_WORK_ONSTACK(&arg.work, local_pci_probe_callback);
+		cpu_hotplug_disable();
 		/*
 		 * The target election and the enqueue of the work must be within
 		 * the same RCU read side section so that when the workqueue pool
@@ -421,11 +421,11 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 			error = local_pci_probe(&ddi);
 		}
 
+		cpu_hotplug_enable();
 		destroy_work_on_stack(&arg.work);
 	}
 
 	dev->is_probed = 0;
-	cpu_hotplug_enable();
 	return error;
 }
 
Signature · 1 line
2.47.1.windows.1
#10 lore raw
In reply to weipengliang
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] The patch removes CPU hotplug protection during PCI device driver probing, exposing drivers to…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The patch removes CPU hotplug protection during PCI device driver probing, exposing drivers to concurrent hotplug races, breaking NUMA locality, and violating CPU isolation. --

commit d0d4eccba5faab11b523664a86d8feec90710056
Author: weipengliang <[email protected]>

PCI: Avoid unnecessary cpu_hotplug_disable() for non-NUMA devices

This commit avoids taking the CPU hotplug lock when probing non-NUMA PCI devices or VFs being probed from a PF context. It also adjusts the scope of the hotplug lock in the NUMA path to cover a smaller region around the target CPU selection and work queueing.

Quoted 4 lines
drivers/pci/pci-driver.c +0 -0
index d10ece0889f0f..8a7da7bf24580 100644

[ ... ]

drivers/pci/pci-driver.c static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev, · line 386

Quoted 12 lines
drivers/pci/pci-driver.c +0 -1
@@ -377,7 +377,6 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 	node = dev_to_node(&dev->dev);
 	dev->is_probed = 1;
 
-	cpu_hotplug_disable();
 	/*
 	 * Prevent nesting work_on_cpu() for the case where a Virtual Function
 	 * device is probed from work_on_cpu() of the Physical device.
@@ -385,10 +384,13 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 	if (node < 0 || node >= MAX_NUMNODES || !node_online(node) ||
 	    pci_physfn_is_probed(dev)) {
 		error = local_pci_probe(&ddi);

[Severity: High]
Does bypassing cpu_hotplug_disable() here expose the driver probe path to concurrent CPU hotplug events?

Historically, cpu_hotplug_disable() wrapped the entirety of PCI probing to prevent hotplug events from causing race conditions or deadlocks inside driver probe functions.

Quoted 7 lines
drivers/pci/pci-driver.c +3 -0
+		dev->is_probed = 0;
+		return error;
 	} else {
 		struct pci_probe_arg arg = { .ddi = &ddi };
 
 		INIT_WORK_ONSTACK(&arg.work, local_pci_probe_callback);
+		cpu_hotplug_disable();

[ ... ]

drivers/pci/pci-driver.c static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev, · line 413

Quoted 6 lines
drivers/pci/pci-driver.c +1 -0
@@ -407,10 +409,12 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
 				wq = system_percpu_wq;
 			queue_work_on(cpu, wq, &arg.work);
 			rcu_read_unlock();
+			cpu_hotplug_enable();
 			flush_work(&arg.work);

[Severity: High]
Does enabling CPU hotplug before flush_work() defeat the protection for the actual probe execution?

If the targeted CPU goes offline while flush_work() is waiting, the workqueue core might unbind the worker pool and migrate the pending probe work. Because unbound workers fall back to cpu_possible_mask, the probe might execute on an incorrect NUMA node or an isolated CPU, violating the housekeeping_cpumask constraints evaluated earlier.

drivers/pci/pci-driver.c static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev, · line 425

Quoted 5 lines
drivers/pci/pci-driver.c +1 -0
 			error = arg.ret;
 		} else {
 			rcu_read_unlock();
+			cpu_hotplug_enable();
 			error = local_pci_probe(&ddi);

[Severity: High]
Similar to the workqueue path, does calling local_pci_probe() after cpu_hotplug_enable() leave the driver initialization unprotected against concurrent CPU hotplug events?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1