Skip to contentSkip to conversation

Drivers: hv: Avoid infinite retry loop in init_vp_index()

PATCH Superseded Waiman Long 3 participants 6 messages Aug 27 – Aug 30

Subsystems (2)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 Drivers: hv: Avoid infinite retry loop in init_vp_index()

There is a retry loop in init_vp_index() where the CPUs from a certain node are stripped out if they have already been in the allocated cpumask or not in HK_TYPE_MANAGED_IRQ housekeeping cpumask. If there is no CPU left, the allocated cpumask is ignored and the process is retried again. However, if the HK_TYPE_MANAGED_IRQ housekeeping cpumask turns out not to contain any CPU in that particular node, that will become an infinite retry loop. This particular problem was reported by sashiko [1]. This should rarely happen, but we still need to guard against this.

Fix this infinite loop problem by also skipping NUMA node that has no housekeeping CPU in the inner while loop of init_vp_index(). As the outer for loop will only be reached if the housekeeping cpumask isn't empty, a NUMA node with housekeeping CPUs will eventually be found.

Fixes 6640b5df1a38 ("Drivers: hv: vmbus: Don't assign VMbus channel interrupts to isolated CPUs")
---
 drivers/hv/channel_mgmt.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/hv/channel_mgmt.c +5 -2
index 89d214dda360..ed121d74d73f 100644
@@ -752,6 +752,7 @@ static void init_vp_index(struct vmbus_channel *channel)
 	u32 i, ncpu = num_online_cpus();
 	cpumask_var_t available_mask;
 	struct cpumask *allocated_mask;
+	const struct cpumask *node_mask;
 	const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ);
 	u32 target_cpu;
 	int numa_node;
@@ -780,14 +781,16 @@ static void init_vp_index(struct vmbus_channel *channel)
 				next_numa_node_id = 0;
 				continue;
 			}
-			if (cpumask_empty(cpumask_of_node(numa_node)))
+			node_mask = cpumask_of_node(numa_node);
+			if (cpumask_empty(node_mask) ||
+			    !cpumask_intersects(node_mask, hk_mask))
 				continue;
 			break;
 		}
 		allocated_mask = &hv_context.hv_numa_map[numa_node];
 
 retry:
-		cpumask_xor(available_mask, allocated_mask, cpumask_of_node(numa_node));
+		cpumask_xor(available_mask, allocated_mask, node_mask);
 		cpumask_and(available_mask, available_mask, hk_mask);
 
 		if (cpumask_empty(available_mask)) {
Signature · 1 line
2.55.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] This problem wasn't introduced by this patch, but the patch attempts to fix an infinite retry loop while relying on a flawed assumption, leaving a kernel hang scenario intact under CPU hotplug. --

commit 357edd9f97074f4d93f3160be466a7aa7d20c7bb
Author: Waiman Long <[email protected]>

Drivers: hv: Avoid infinite retry loop in init_vp_index()

This commit attempts to fix an infinite retry loop in init_vp_index() by skipping NUMA nodes that do not contain any housekeeping CPUs. It aims to ensure that a NUMA node with housekeeping CPUs is eventually found.

Quoted 4 lines
drivers/hv/channel_mgmt.c +0 -0
index a044fd3b3c4e7..2e611d2de6d10 100644

[ ... ]

drivers/hv/channel_mgmt.c static void init_vp_index(struct vmbus_channel *channel) · line 789

Quoted 11 lines
drivers/hv/channel_mgmt.c +3 -1
@@ -780,14 +781,16 @@ static void init_vp_index(struct vmbus_channel *channel)
 				next_numa_node_id = 0;
 				continue;
 			}
-			if (cpumask_empty(cpumask_of_node(numa_node)))
+			node_mask = cpumask_of_node(numa_node);
+			if (cpumask_empty(node_mask) ||
+			    !cpumask_intersects(node_mask, hk_mask))
 				continue;
 			break;
 		}

[Severity: High]
This is a pre-existing issue, but the patch attempts to fix an infinite retry loop while relying on an assumption that leaves a kernel hang scenario intact under CPU hotplug.

Could this while (true) loop in init_vp_index() iterate endlessly without sleeping if an administrator dynamically hot-offlines all managed IRQ housekeeping CPUs?

Since housekeeping_cpumask(HK_TYPE_MANAGED_IRQ) tracks all possible housekeeping CPUs, the early abort check at the start of the function will see hk_mask as non-empty even if those CPUs are currently offline.

However, cpumask_of_node() only returns online CPUs. If the remaining managed IRQ housekeeping CPUs are offlined, cpumask_intersects(node_mask, hk_mask) will evaluate to false for every NUMA node, causing this loop to spin indefinitely while holding channel_mutex and cpus_read_lock().

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Waiman Long

From: Waiman Long <[email protected]> Sent: Thursday, August 27, 2026 12:38 PM

Quoted 35 lines

There is a retry loop in init_vp_index() where the CPUs from a certain node are stripped out if they have already been in the allocated cpumask or not in HK_TYPE_MANAGED_IRQ housekeeping cpumask. If there is no CPU left, the allocated cpumask is ignored and the process is retried again. However, if the HK_TYPE_MANAGED_IRQ housekeeping cpumask turns out not to contain any CPU in that particular node, that will become an infinite retry loop. This particular problem was reported by sashiko [1]. This should rarely happen, but we still need to guard against this.

Fix this infinite loop problem by also skipping NUMA node that has no housekeeping CPU in the inner while loop of init_vp_index(). As the outer for loop will only be reached if the housekeeping cpumask isn't empty, a NUMA node with housekeeping CPUs will eventually be found.

Fixes 6640b5df1a38 ("Drivers: hv: vmbus: Don't assign VMbus channel interrupts to isolated CPUs")
---
 drivers/hv/channel_mgmt.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/hv/channel_mgmt.c +1 -0
index 89d214dda360..ed121d74d73f 100644
@@ -752,6 +752,7 @@ static void init_vp_index(struct vmbus_channel *channel)
 	u32 i, ncpu = num_online_cpus();
 	cpumask_var_t available_mask;
 	struct cpumask *allocated_mask;
+	const struct cpumask *node_mask;
 	const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ);
 	u32 target_cpu;
 	int numa_node;

drivers/hv/channel_mgmt.c static void init_vp_index(struct vmbus_channel *channel) · line 786

Quoted 7 lines
drivers/hv/channel_mgmt.c +3 -1
@@ -780,14 +781,16 @@ static void init_vp_index(struct vmbus_channel *channel)
 				next_numa_node_id = 0;
 				continue;
 			}
-			if (cpumask_empty(cpumask_of_node(numa_node)))
+			node_mask = cpumask_of_node(numa_node);
+			if (cpumask_empty(node_mask) ||
+			    !cpumask_intersects(node_mask, hk_mask))

The cpumask_empty() test looks to be redundant. The cpumask_intersects() test will catch the case where node_mask is empty.

Otherwise, I think this looks good as a solution to the core problem.

Michael

Quoted 14 lines
drivers/hv/channel_mgmt.c +1 -2
 				continue;
 			break;
 		}
 		allocated_mask = &hv_context.hv_numa_map[numa_node];
 
 retry:
-		cpumask_xor(available_mask, allocated_mask, cpumask_of_node(numa_node));
+		cpumask_xor(available_mask, allocated_mask, node_mask);
 		cpumask_and(available_mask, available_mask, hk_mask);
 
 		if (cpumask_empty(available_mask)) {
--

2.55.0

#3 lore raw
Quoted 46 lines · Michael Kelley

From: Waiman Long <[email protected]> Sent: Thursday, August 27, 2026 12:38 PM

Quoted 34 lines

There is a retry loop in init_vp_index() where the CPUs from a certain node are stripped out if they have already been in the allocated cpumask or not in HK_TYPE_MANAGED_IRQ housekeeping cpumask. If there is no CPU left, the allocated cpumask is ignored and the process is retried again. However, if the HK_TYPE_MANAGED_IRQ housekeeping cpumask turns out not to contain any CPU in that particular node, that will become an infinite retry loop. This particular problem was reported by sashiko [1]. This should rarely happen, but we still need to guard against this.

Fix this infinite loop problem by also skipping NUMA node that has no housekeeping CPU in the inner while loop of init_vp_index(). As the outer for loop will only be reached if the housekeeping cpumask isn't empty, a NUMA node with housekeeping CPUs will eventually be found.

Fixes 6640b5df1a38 ("Drivers: hv: vmbus: Don't assign VMbus channel interrupts to isolated CPUs")
---
  drivers/hv/channel_mgmt.c | 7 +++++--
  1 file changed, 5 insertions(+), 2 deletions(-)
drivers/hv/channel_mgmt.c +1 -0
index 89d214dda360..ed121d74d73f 100644
@@ -752,6 +752,7 @@ static void init_vp_index(struct vmbus_channel *channel)
  	u32 i, ncpu = num_online_cpus();
  	cpumask_var_t available_mask;
  	struct cpumask *allocated_mask;
+	const struct cpumask *node_mask;
  	const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ);
  	u32 target_cpu;
  	int numa_node;

drivers/hv/channel_mgmt.c static void init_vp_index(struct vmbus_channel *channel) · line 786

Quoted 7 lines
drivers/hv/channel_mgmt.c +3 -1
@@ -780,14 +781,16 @@ static void init_vp_index(struct vmbus_channel *channel)
  				next_numa_node_id = 0;
  				continue;
  			}
-			if (cpumask_empty(cpumask_of_node(numa_node)))
+			node_mask = cpumask_of_node(numa_node);
+			if (cpumask_empty(node_mask) ||
+			    !cpumask_intersects(node_mask, hk_mask))

The cpumask_empty() test looks to be redundant. The cpumask_intersects() test will catch the case where node_mask is empty.

Quoted 4 lines

Otherwise, I think this looks good as a solution to the core problem.

Michael

Yes, I am aware that cpumask_empty() test is redundant and can be skipped. I keep it just to make it easier to read. I can certainly drop the cpumask_empty() statement.

Cheers,
Longman

Quoted 15 lines
Quoted 15 lines
drivers/hv/channel_mgmt.c +1 -2
  				continue;
  			break;
  		}
  		allocated_mask = &hv_context.hv_numa_map[numa_node];
 
  retry:
-		cpumask_xor(available_mask, allocated_mask, cpumask_of_node(numa_node));
+		cpumask_xor(available_mask, allocated_mask, node_mask);
  		cpumask_and(available_mask, available_mask, hk_mask);
 
  		if (cpumask_empty(available_mask)) {
--

2.55.0

#4 lore raw

From: Waiman Long <[email protected]> Sent: Friday, August 28, 2026 6:22 PM

Quoted 48 lines · Michael Kelley
Quoted 46 lines

From: Waiman Long <[email protected]> Sent: Thursday, August 27, 2026 12:38 PM

Quoted 34 lines

There is a retry loop in init_vp_index() where the CPUs from a certain node are stripped out if they have already been in the allocated cpumask or not in HK_TYPE_MANAGED_IRQ housekeeping cpumask. If there is no CPU left, the allocated cpumask is ignored and the process is retried again. However, if the HK_TYPE_MANAGED_IRQ housekeeping cpumask turns out not to contain any CPU in that particular node, that will become an infinite retry loop. This particular problem was reported by sashiko [1]. This should rarely happen, but we still need to guard against this.

Fix this infinite loop problem by also skipping NUMA node that has no housekeeping CPU in the inner while loop of init_vp_index(). As the outer for loop will only be reached if the housekeeping cpumask isn't empty, a NUMA node with housekeeping CPUs will eventually be found.

Fixes 6640b5df1a38 ("Drivers: hv: vmbus: Don't assign VMbus channel interrupts to isolated CPUs")
---
  drivers/hv/channel_mgmt.c | 7 +++++--
  1 file changed, 5 insertions(+), 2 deletions(-)
drivers/hv/channel_mgmt.c +1 -0
index 89d214dda360..ed121d74d73f 100644
@@ -752,6 +752,7 @@ static void init_vp_index(struct vmbus_channel *channel)
  	u32 i, ncpu = num_online_cpus();
  	cpumask_var_t available_mask;
  	struct cpumask *allocated_mask;
+	const struct cpumask *node_mask;
  	const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ);
  	u32 target_cpu;
  	int numa_node;

drivers/hv/channel_mgmt.c static void init_vp_index(struct vmbus_channel *channel) · line 786

Quoted 7 lines
drivers/hv/channel_mgmt.c +3 -1
@@ -780,14 +781,16 @@ static void init_vp_index(struct vmbus_channel *channel)
  				next_numa_node_id = 0;
  				continue;
  			}
-			if (cpumask_empty(cpumask_of_node(numa_node)))
+			node_mask = cpumask_of_node(numa_node);
+			if (cpumask_empty(node_mask) ||
+			    !cpumask_intersects(node_mask, hk_mask))

The cpumask_empty() test looks to be redundant. The cpumask_intersects() test will catch the case where node_mask is empty.

Quoted 9 lines
Quoted 4 lines

Otherwise, I think this looks good as a solution to the core problem.

Michael

Yes, I am aware that cpumask_empty() test is redundant and can be skipped. I keep it just to make it easier to read. I can certainly drop the cpumask_empty() statement.

The flip side is people like me try to figure out "why is this here?", thinking there must be a reason. :-) I'd say drop the code and add a comment like "Also catches an empty node_mask".

Michael

#5 lore raw
Quoted 51 lines · Michael Kelley

From: Waiman Long <[email protected]> Sent: Friday, August 28, 2026 6:22 PM

Quoted 50 lines · Michael Kelley
Quoted 49 lines

From: Waiman Long <[email protected]> Sent: Thursday, August 27, 2026 12:38 PM

Quoted 34 lines

There is a retry loop in init_vp_index() where the CPUs from a certain node are stripped out if they have already been in the allocated cpumask or not in HK_TYPE_MANAGED_IRQ housekeeping cpumask. If there is no CPU left, the allocated cpumask is ignored and the process is retried again. However, if the HK_TYPE_MANAGED_IRQ housekeeping cpumask turns out not to contain any CPU in that particular node, that will become an infinite retry loop. This particular problem was reported by sashiko [1]. This should rarely happen, but we still need to guard against this.

Fix this infinite loop problem by also skipping NUMA node that has no housekeeping CPU in the inner while loop of init_vp_index(). As the outer for loop will only be reached if the housekeeping cpumask isn't empty, a NUMA node with housekeeping CPUs will eventually be found.

Fixes 6640b5df1a38 ("Drivers: hv: vmbus: Don't assign VMbus channel interrupts to isolated CPUs")
---
   drivers/hv/channel_mgmt.c | 7 +++++--
   1 file changed, 5 insertions(+), 2 deletions(-)
drivers/hv/channel_mgmt.c +1 -0
index 89d214dda360..ed121d74d73f 100644
@@ -752,6 +752,7 @@ static void init_vp_index(struct vmbus_channel *channel)
   	u32 i, ncpu = num_online_cpus();
   	cpumask_var_t available_mask;
   	struct cpumask *allocated_mask;
+	const struct cpumask *node_mask;
   	const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ);
   	u32 target_cpu;
   	int numa_node;

drivers/hv/channel_mgmt.c static void init_vp_index(struct vmbus_channel *channel) · line 786

Quoted 7 lines
drivers/hv/channel_mgmt.c +3 -1
@@ -780,14 +781,16 @@ static void init_vp_index(struct vmbus_channel *channel)
   				next_numa_node_id = 0;
   				continue;
   			}
-			if (cpumask_empty(cpumask_of_node(numa_node)))
+			node_mask = cpumask_of_node(numa_node);
+			if (cpumask_empty(node_mask) ||
+			    !cpumask_intersects(node_mask, hk_mask))

The cpumask_empty() test looks to be redundant. The cpumask_intersects() test will catch the case where node_mask is empty.

Otherwise, I think this looks good as a solution to the core problem.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Michael

Yes, I am aware that cpumask_empty() test is redundant and can be skipped. I keep it just to make it easier to read. I can certainly drop the cpumask_empty() statement.

The flip side is people like me try to figure out "why is this here?", thinking there must be a reason. :-) I'd say drop the code and add a comment like "Also catches an empty node_mask".

Fair, will do that.

Thanks,
Longman

Quoted 1 line

Michael