Skip to contentSkip to conversation

genirq: Fix software resend lockup and nested resend

PATCH Applied Johan Hovold 3 participants 3 messages Aug 26, 2023 IRQ

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 genirq: Fix software resend lockup and nested resend

The switch to using hlist for managing software resend of interrupts broke resend in at least two ways:

First, unconditionally adding interrupt descriptors to the resend list can corrupt the list when the descriptor in question has already been added. This causes the resend tasklet to loop indefinitely with interrupts disabled as was recently reported with the Lenovo ThinkPad X13s after threaded NAPI was disabled in the ath11k WiFi driver. [1]

This bug is easily fixed by restoring the old semantics of irq_sw_resend() so that it can be called also for descriptors that have already been marked for resend.

Second, the offending commit also broke software resend of nested interrupts by simply discarding the code that made sure that such interrupts are retriggered using the parent interrupt.

Add back the corresponding code that adds the parent descriptor to the resend list. Note that this bit is untested, but I decided to include it to avoid having to revert the offending commit and the maple tree conversion that depends on it.

[1] https://lore.kernel.org/lkml/[email protected]/

Fixes bc06a9e08742 ("genirq: Use hlist for managing resend handlers")
Cc Shanker Donthineni <[email protected]>

---

Hi Thomas and Marc,

This patch fixes a severe regression in the resend code in 6.5-rc1 that breaks machines like the Lenovo X13s and which ideally should be addressed before 6.5 is released tomorrow.

I hesitated about including the fix for nested interrupts as I've not had time to test this bit, but I ultimately decided to include it to avoid having to suggest a revert of the maple tree conversion. Let me know if you prefer to go this route and I'll post a (prepared) revert series instead.

Johan

 kernel/irq/resend.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/irq/resend.c +6 -1
index edec335c0a7a..5f2c66860ac6 100644
@@ -68,11 +68,16 @@ static int irq_sw_resend(struct irq_desc *desc)
 		 */
 		if (!desc->parent_irq)
 			return -EINVAL;
+
+		desc = irq_to_desc(desc->parent_irq);
+		if (!desc)
+			return -EINVAL;
 	}
 
 	/* Add to resend_list and activate the softirq: */
 	raw_spin_lock(&irq_resend_lock);
-	hlist_add_head(&desc->resend_node, &irq_resend_list);
+	if (hlist_unhashed(&desc->resend_node))
+		hlist_add_head(&desc->resend_node, &irq_resend_list);
 	raw_spin_unlock(&irq_resend_lock);
 	tasklet_schedule(&resend_tasklet);
 	return 0;
Signature · 1 line
2.41.0
Reviewed-by on v1 1/1
#1 lore raw

Hi Johan,

Quoted 9 lines · Johan Hovold

The switch to using hlist for managing software resend of interrupts broke resend in at least two ways:

First, unconditionally adding interrupt descriptors to the resend list can corrupt the list when the descriptor in question has already been added. This causes the resend tasklet to loop indefinitely with interrupts disabled as was recently reported with the Lenovo ThinkPad X13s after threaded NAPI was disabled in the ath11k WiFi driver. [1]

Gah, of course. Making the descriptor pending again isn't an idempotent operation anymore (while setting an already set bit in the bitmap was). You'll need the right timing, but it looks like you've managed to achieve that reliably.

Quoted 5 lines

This bug is easily fixed by restoring the old semantics of irq_sw_resend() so that it can be called also for descriptors that have already been marked for resend.

Quoted 8 lines

Second, the offending commit also broke software resend of nested interrupts by simply discarding the code that made sure that such interrupts are retriggered using the parent interrupt.

Add back the corresponding code that adds the parent descriptor to the resend list. Note that this bit is untested, but I decided to include it to avoid having to revert the offending commit and the maple tree conversion that depends on it.

Indeed, and that one is not timing related, but 100% broken.

Quoted 6 lines

[1] https://lore.kernel.org/lkml/[email protected]/

Fixes bc06a9e08742 ("genirq: Use hlist for managing resend handlers")
Cc Shanker Donthineni <[email protected]>
Quoted 7 lines

---

Hi Thomas and Marc,

This patch fixes a severe regression in the resend code in 6.5-rc1 that breaks machines like the Lenovo X13s and which ideally should be addressed before 6.5 is released tomorrow.

This relies on Thomas seeing this email before tomorrow. Unless you directly reach out to Linus to try and intercept the release.

Quoted 6 lines

I hesitated about including the fix for nested interrupts as I've not had time to test this bit, but I ultimately decided to include it to avoid having to suggest a revert of the maple tree conversion. Let me know if you prefer to go this route and I'll post a (prepared) revert series instead.

I think it is too late for that revert to happen, and we might as well plough along and take the fix.

Quoted 19 lines

Johan

 kernel/irq/resend.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/irq/resend.c +4 -0
index edec335c0a7a..5f2c66860ac6 100644
@@ -68,11 +68,16 @@ static int irq_sw_resend(struct irq_desc *desc)
 		 */
 		if (!desc->parent_irq)
 			return -EINVAL;
+
+		desc = irq_to_desc(desc->parent_irq);
+		if (!desc)
+			return -EINVAL;

kernel/irq/resend.c static int irq_sw_resend(struct irq_desc *desc) · line 83

Quoted 10 lines
kernel/irq/resend.c +2 -1
 	}
 
 	/* Add to resend_list and activate the softirq: */
 	raw_spin_lock(&irq_resend_lock);
-	hlist_add_head(&desc->resend_node, &irq_resend_list);
+	if (hlist_unhashed(&desc->resend_node))
+		hlist_add_head(&desc->resend_node, &irq_resend_list);
 	raw_spin_unlock(&irq_resend_lock);
 	tasklet_schedule(&resend_tasklet);
 	return 0;

FWIW:

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
merge bot Applied
#2 lore raw
In reply to Johan Hovold
The following commit has been merged into the irq/urgent branch of tip:

The following commit has been merged into the irq/urgent branch of tip:

Commit-ID:     9f5deb551655a4cff04b21ecffdcdab75112da3a
Gitweb:        https://git.kernel.org/tip/9f5deb551655a4cff04b21ecffdcdab75112da3a
Author:        Johan Hovold <[email protected]>
AuthorDate:    Sat, 26 Aug 2023 17:40:04 +02:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Sat, 26 Aug 2023 19:14:31 +02:00

genirq: Fix software resend lockup and nested resend

The switch to using hlist for managing software resend of interrupts broke resend in at least two ways:

First, unconditionally adding interrupt descriptors to the resend list can corrupt the list when the descriptor in question has already been added. This causes the resend tasklet to loop indefinitely with interrupts disabled as was recently reported with the Lenovo ThinkPad X13s after threaded NAPI was disabled in the ath11k WiFi driver.

This bug is easily fixed by restoring the old semantics of irq_sw_resend() so that it can be called also for descriptors that have already been marked for resend.

Second, the offending commit also broke software resend of nested interrupts by simply discarding the code that made sure that such interrupts are retriggered using the parent interrupt.

Add back the corresponding code that adds the parent descriptor to the resend list.

---
 kernel/irq/resend.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/irq/resend.c +6 -1
index edec335..5f2c668 100644
@@ -68,11 +68,16 @@ static int irq_sw_resend(struct irq_desc *desc)
 		 */
 		if (!desc->parent_irq)
 			return -EINVAL;
+
+		desc = irq_to_desc(desc->parent_irq);
+		if (!desc)
+			return -EINVAL;
 	}
 
 	/* Add to resend_list and activate the softirq: */
 	raw_spin_lock(&irq_resend_lock);
-	hlist_add_head(&desc->resend_node, &irq_resend_list);
+	if (hlist_unhashed(&desc->resend_node))
+		hlist_add_head(&desc->resend_node, &irq_resend_list);
 	raw_spin_unlock(&irq_resend_lock);
 	tasklet_schedule(&resend_tasklet);
 	return 0;