Skip to contentSkip to conversation

Fix state corruption on fault migration, cleanups

PATCH In review Jan Kiszka 2 participants 5 messages Jun 7, 2021

Subsystems (3)
lore
Share & tools
lore

Jan Kiszka

author
#0 lore raw

v3 cover Fix state corruption on fault migration, cleanups

Changes in v3:
- do not track oob stalling on iryentry
- add check that oob is unstalled on irqentry

Jan

Jan Kiszka (3):
irq_pipeline: Clean up stage_info field and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when calling irqentry_enter with oob stalled

 arch/x86/kernel/irq_pipeline.c |  2 +-
 arch/x86/mm/fault.c            |  2 +-
 include/linux/entry-common.h   | 11 ++++++++---
 kernel/entry/common.c          | 17 ++++++++++-------
 4 files changed, 20 insertions(+), 12 deletions(-)
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH v3 1/3 irq_pipeline: Clean up stage_info field and users

From: Jan Kiszka <[email protected]>

This field represents mutually exclusive states, namely

- IRQENTRY_INBAND_UNSTALLED
- IRQENTRY_INBAND_STALLED
- IRQENTRY_OOB

Encodes them as enum and test against them, rather than against state bits that suggest they could be combined.

Also flip the inverted naming of INBAND_STALLED vs. INBAND_UNSTALLED:
Only when we entered under INBAND_UNSTALLED, certain actions need to be taken on exit.

Finally, document the stage_info field of irqentry_state.

---
 arch/x86/kernel/irq_pipeline.c |  2 +-
 arch/x86/mm/fault.c            |  2 +-
 include/linux/entry-common.h   | 11 ++++++++---
 kernel/entry/common.c          | 12 ++++++------
 4 files changed, 16 insertions(+), 11 deletions(-)
arch/x86/kernel/irq_pipeline.c +1 -1
index 7ff5c03386e8..6944f9e4f852 100644
@@ -64,7 +64,7 @@ static irqentry_state_t pipeline_enter_rcu(void)
 {
 	irqentry_state_t state = {
 		.exit_rcu = false,
-		.stage_info = 0,
+		.stage_info = IRQENTRY_INBAND_UNSTALLED,
 	};
 
 	if (!IS_ENABLED(CONFIG_TINY_RCU) && is_idle_task(current)) {
arch/x86/mm/fault.c +1 -1
index dcc3e77fed2b..a242ac40a71d 100644
@@ -813,7 +813,7 @@ static inline void cond_reenable_irqs_kernel(irqentry_state_t state,
 					struct pt_regs *regs)
 {
 	if (regs->flags & X86_EFLAGS_IF) {
-		if (state.stage_info & IRQENTRY_INBAND_STALLED)
+		if (state.stage_info == IRQENTRY_INBAND_UNSTALLED)
 			unstall_inband_nocheck();
 		hard_local_irq_enable();
 	}
include/linux/entry-common.h +8 -3
index 0b6639ff313f..2a08b00632b9 100644
@@ -399,6 +399,12 @@ void irqentry_enter_from_user_mode(struct pt_regs *regs);
  */
 void irqentry_exit_to_user_mode(struct pt_regs *regs);
 
+enum irqentry_info {
+	IRQENTRY_INBAND_UNSTALLED = 0,
+	IRQENTRY_INBAND_STALLED,
+	IRQENTRY_OOB,
+};
+
 #ifndef irqentry_state
 /**
  * struct irqentry_state - Opaque object for exception state storage
@@ -406,6 +412,7 @@ void irqentry_exit_to_user_mode(struct pt_regs *regs);
  *            exit path has to invoke rcu_irq_exit().
  * @lockdep: Used exclusively in the irqentry_nmi_*() calls; ensures that
  *           lockdep state is restored correctly on exit from nmi.
+ * @stage_info: Information about pipeline state and current stage on IRQ entry.
  *
  * This opaque object is filled in by the irqentry_*_enter() functions and
  * must be passed back into the corresponding irqentry_*_exit() functions
@@ -421,9 +428,7 @@ typedef struct irqentry_state {
 		bool	lockdep;
 	};
 #ifdef CONFIG_IRQ_PIPELINE
-#define IRQENTRY_INBAND_STALLED	BIT(0)
-#define IRQENTRY_OOB_ENTRY	BIT(1)
-	int	stage_info;
+	enum irqentry_info stage_info;
 #endif
 } irqentry_state_t;
 #endif
kernel/entry/common.c +6 -6
index e8f274523217..5a669f87ec3f 100644
@@ -404,13 +404,13 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)
 	irqentry_state_t ret = {
 		.exit_rcu = false,
 #ifdef CONFIG_IRQ_PIPELINE
-		.stage_info = 0,
+		.stage_info = IRQENTRY_INBAND_STALLED,
 #endif
 	};
 
 #ifdef CONFIG_IRQ_PIPELINE
 	if (running_oob()) {
-		ret.stage_info = IRQENTRY_OOB_ENTRY;
+		ret.stage_info = IRQENTRY_OOB;
 		return ret;
 	}
 #endif
@@ -419,7 +419,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)
 #ifdef CONFIG_IRQ_PIPELINE
 		WARN_ON_ONCE(irq_pipeline_debug() && irqs_disabled());
 		stall_inband_nocheck();
-		ret.stage_info = IRQENTRY_INBAND_STALLED;
+		ret.stage_info = IRQENTRY_INBAND_UNSTALLED;
 #endif
 		irqentry_enter_from_user_mode(regs);
 		return ret;
@@ -432,7 +432,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)
 	 * irqs are off on entry, we have to stall the in-band stage.
 	 */
 	if (!test_and_stall_inband_nocheck())
-		ret.stage_info = IRQENTRY_INBAND_STALLED;
+		ret.stage_info = IRQENTRY_INBAND_UNSTALLED;
 #endif
 
 	/*
@@ -510,7 +510,7 @@ static inline
 bool irqexit_may_preempt_schedule(irqentry_state_t state,
 				struct pt_regs *regs)
 {
-	return !!(state.stage_info & IRQENTRY_INBAND_STALLED);
+	return state.stage_info == IRQENTRY_INBAND_UNSTALLED;
 }
 
 #else
@@ -579,7 +579,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)
 	 * irqentry_enter() raised it in order to mirror the hardware
 	 * state.
 	 */
-	if (state.stage_info & IRQENTRY_INBAND_STALLED)
+	if (state.stage_info == IRQENTRY_INBAND_UNSTALLED)
 		unstall_inband();
 #endif
 	return;
Signature · 1 line
2.26.2
author
#2 lore raw

PATCH v3 2/3 irq_pipeline: Account for stage migration across faults

In reply to Jan Kiszka

From: Jan Kiszka <[email protected]>

We need to unstall the inband stage when we entered for a fault over OOB and then migrated to inband. So far we kept the inband stage stalled, causing a state corruption this way.

---
 kernel/entry/common.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/entry/common.c +4 -2
index 5a669f87ec3f..35a6f6a4a1ba 100644
@@ -577,9 +577,11 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)
 	/*
 	 * If pipelining interrupts, clear the in-band stall bit if
 	 * irqentry_enter() raised it in order to mirror the hardware
-	 * state.
+	 * state. Also clear it when we where coming from oob, thus went
+	 * through a migration that was caused by taking, e.g., a fault.
 	 */
-	if (state.stage_info == IRQENTRY_INBAND_UNSTALLED)
+	if (state.stage_info == IRQENTRY_INBAND_UNSTALLED ||
+	    state.stage_info == IRQENTRY_OOB)
 		unstall_inband();
 #endif
 	return;
Signature · 1 line
2.26.2
author
#3 lore raw

PATCH v3 3/3 irq_pipeline: Warn when calling irqentry_enter with oob stalled

In reply to Jan Kiszka

From: Jan Kiszka <[email protected]>

Something must have went wrong if entering for an IRQ or an exception over oob and with this stage stalled. Warn when debugging

---
 kernel/entry/common.c | 1 +
 1 file changed, 1 insertion(+)
kernel/entry/common.c +1 -0
index 35a6f6a4a1ba..a6c0236850ad 100644
@@ -410,6 +410,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)
 
 #ifdef CONFIG_IRQ_PIPELINE
 	if (running_oob()) {
+		WARN_ON_ONCE(irq_pipeline_debug() && oob_irqs_disabled());
 		ret.stage_info = IRQENTRY_OOB;
 		return ret;
 	}
Signature · 1 line
2.26.2
#4 lore raw
In reply to Jan Kiszka
Quoted 11 lines · Jan Kiszka via Xenomai

Changes in v3:
- do not track oob stalling on iryentry
- add check that oob is unstalled on irqentry

Jan

Jan Kiszka (3):
irq_pipeline: Clean up stage_info field and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when calling irqentry_enter with oob stalled

Quoted 6 lines
 arch/x86/kernel/irq_pipeline.c |  2 +-
 arch/x86/mm/fault.c            |  2 +-
 include/linux/entry-common.h   | 11 ++++++++---
 kernel/entry/common.c          | 17 ++++++++++-------
 4 files changed, 20 insertions(+), 12 deletions(-)

Tested and Reported-by: Florian Bezdeka <[email protected]>

I stressed my system for a few hours with the stress-ng workload that was triggering the complete system hang. Normally the system was dead after 40-50 minutes, it survived for at least a couple of hours now.

Signature · 2 lines
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux