Skip to contentSkip to conversation

rqspinlock: Fix order in raw_res_spin_(un)lock_irq to allow schedule

PATCH In review Gabriele Monaco 6 participants 9 messages Jun 9 – Jun 10

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rqspinlock: Fix order in raw_res_spin_(un)lock_irq to allow schedule

raw_res_spin_unlock_irqrestore() calls raw_res_spin_unlock() and then restores interrupts, this means preemption is enabled when interrupts are still disabled (as part of raw_res_spin_unlock()) so this cannot trigger an actual preemption.
This is inconsistent with other spinlock implementations (raw_spin_unlock_irqrestore() and bpf_res_spin_unlock_irqrestore() itself).

Adjust the macro to ensure interrupts are enabled before enabling preemption, allowing to schedule at that point. Make the same modification in the error path of raw_res_spin_lock_irqsave().

Fixes 101acd2e78b1 ("rqspinlock: Add macros for rqspinlock usage")
---
 include/asm-generic/rqspinlock.h | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
include/asm-generic/rqspinlock.h +11 -3
index 151d267a49..4d46643f46 100644
@@ -243,12 +243,20 @@ static __always_inline void res_spin_unlock(rqspinlock_t *lock)
 	({                                        \
 		int __ret;                        \
 		local_irq_save(flags);            \
-		__ret = raw_res_spin_lock(lock);  \
-		if (__ret)                        \
+		preempt_disable();                \
+		__ret = res_spin_lock(lock);      \
+		if (__ret) {                      \
 			local_irq_restore(flags); \
+			preempt_enable();         \
+		}                                 \
 		__ret;                            \
 	})
 
-#define raw_res_spin_unlock_irqrestore(lock, flags) ({ raw_res_spin_unlock(lock); local_irq_restore(flags); })
+#define raw_res_spin_unlock_irqrestore(lock, flags) \
+	({                                          \
+		res_spin_unlock(lock);              \
+		local_irq_restore(flags);           \
+		preempt_enable();                   \
+	})
 
 #endif /* __ASM_GENERIC_RQSPINLOCK_H */

base-commit: e43ffb69e0438cddd72aaa30898b4dc446f664f8

Signature · 1 line
2.54.0
maintainer Acked-by on v1 1/1
#1 lore raw
Quoted 8 lines · Gabriele Monaco

raw_res_spin_unlock_irqrestore() calls raw_res_spin_unlock() and then restores interrupts, this means preemption is enabled when interrupts are still disabled (as part of raw_res_spin_unlock()) so this cannot trigger an actual preemption.
This is inconsistent with other spinlock implementations (raw_spin_unlock_irqrestore() and bpf_res_spin_unlock_irqrestore() itself).

Quoted 5 lines

Adjust the macro to ensure interrupts are enabled before enabling preemption, allowing to schedule at that point. Make the same modification in the error path of raw_res_spin_lock_irqsave().

Fixes 101acd2e78b1 ("rqspinlock: Add macros for rqspinlock usage")

Should this be Cc:[email protected] to get backported?

Did you see this cause measurable performance problems, or did you find it through inspection?

Quoted 1 line

This should probably get merged through the BPF tree, but I've added the kernel/locking maintainers to Cc as well, since I feel it's more useful to have them look at it than me.

Maybe it would be good to update (as a separate patch) the MAINTAINERS file so the locking subsystem also includes the headers currently missing:

arch/*/include/asm/*spinlock*.h
arch/*/include/asm/*rwlock*.h
include/asm-generic/*spinlock*.h
include/asm-generic/*rwlock*.h

Arnd

(full patch quoted below)

Quoted 39 lines
---
 include/asm-generic/rqspinlock.h | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
include/asm-generic/rqspinlock.h +11 -3
index 151d267a49..4d46643f46 100644
@@ -243,12 +243,20 @@ static __always_inline void 
 res_spin_unlock(rqspinlock_t *lock)
 	({                                        \
 		int __ret;                        \
 		local_irq_save(flags);            \
-		__ret = raw_res_spin_lock(lock);  \
-		if (__ret)                        \
+		preempt_disable();                \
+		__ret = res_spin_lock(lock);      \
+		if (__ret) {                      \
 			local_irq_restore(flags); \
+			preempt_enable();         \
+		}                                 \
 		__ret;                            \
 	})
 
-#define raw_res_spin_unlock_irqrestore(lock, flags) ({ 
 raw_res_spin_unlock(lock); local_irq_restore(flags); })
+#define raw_res_spin_unlock_irqrestore(lock, flags) \
+	({                                          \
+		res_spin_unlock(lock);              \
+		local_irq_restore(flags);           \
+		preempt_enable();                   \
+	})

#endif /* __ASM_GENERIC_RQSPINLOCK_H */

base-commit: e43ffb69e0438cddd72aaa30898b4dc446f664f8

Signature · 1 line
2.54.0
#2 lore raw
Quoted 1 line · Arnd Bergmann

Should this be Cc:[email protected] to get backported?

Not sure if the Fixes: is enough to trigger the automation, I rarely remember to Cc:[email protected] and they're usually picked.

In case I guess I'd need to re-submit the patch right?

Quoted 2 lines

Did you see this cause measurable performance problems, or did you find it through inspection?

I noticed it while debugging an ENOMEM issue in the test_maps BPF selftest on PREEMPT_RT and this was an obvious cuplrit (irq_work not scheduled during a stress run). Turns out the problem is still there after this fix though.

Quoted 4 lines
Quoted 2 lines

Thanks,
Gabriele

Quoted 57 lines

This should probably get merged through the BPF tree, but I've added the kernel/locking maintainers to Cc as well, since I feel it's more useful to have them look at it than me.

Maybe it would be good to update (as a separate patch) the MAINTAINERS file so the locking subsystem also includes the headers currently missing:

arch/*/include/asm/*spinlock*.h
arch/*/include/asm/*rwlock*.h
include/asm-generic/*spinlock*.h
include/asm-generic/*rwlock*.h

       Arnd

(full patch quoted below)

Quoted 39 lines

---
 include/asm-generic/rqspinlock.h | 14 +++++++++++---

 1 file changed, 11 insertions(+), 3 deletions(-)
include/asm-generic/rqspinlock.h +11 -3
index 151d267a49..4d46643f46 100644
@@ -243,12 +243,20 @@ static __always_inline void 
 res_spin_unlock(rqspinlock_t *lock)
  	({                                        \
  		int __ret;                        \
  		local_irq_save(flags);            \
-		__ret = raw_res_spin_lock(lock);  \
-		if (__ret)                        \
+		preempt_disable();                \
+		__ret = res_spin_lock(lock);      \
+		if (__ret) {                      \
  			local_irq_restore(flags); \
+			preempt_enable();         \
+		}                                 \
  		__ret;                            \
  	})
 
-#define raw_res_spin_unlock_irqrestore(lock, flags) ({ 
 raw_res_spin_unlock(lock); local_irq_restore(flags); })
+#define raw_res_spin_unlock_irqrestore(lock, flags) \
+	({                                          \
+		res_spin_unlock(lock);              \
+		local_irq_restore(flags);           \
+		preempt_enable();                   \
+	})

 #endif /* __ASM_GENERIC_RQSPINLOCK_H */

base-commit: e43ffb69e0438cddd72aaa30898b4dc446f664f8

Signature · 1 line
2.54.0
maintainer
#3 lore raw
Quoted 5 lines · Gabriele Monaco
Quoted 1 line · Arnd Bergmann

Should this be Cc:[email protected] to get backported?

Not sure if the Fixes: is enough to trigger the automation, I rarely remember to Cc:[email protected] and they're usually picked.

There is always human interaction. If you just have 'Fixes', this means someone will have to look at the patch carefully and make a decision, since a lot of bugfix patches either don't apply to old kernels or don't fall under the rules for stable backports.

If the patch gets tagged Cc:, this means it is expected to be backported and needs less manual work.

Quoted 1 line

In case I guess I'd need to re-submit the patch right?

It can be added by whoever picks up the patch.

Quoted 7 lines
Quoted 2 lines

Did you see this cause measurable performance problems, or did you find it through inspection?

I noticed it while debugging an ENOMEM issue in the test_maps BPF selftest on PREEMPT_RT and this was an obvious cuplrit (irq_work not scheduled during a stress run). Turns out the problem is still there after this fix though.

Ok

Arnd

maintainer Acked-by
#4 lore raw
In reply to Gabriele Monaco
Quoted 8 lines · Gabriele Monaco
Quoted 1 line · Arnd Bergmann

Should this be Cc:[email protected] to get backported?

Not sure if the Fixes: is enough to trigger the automation, I rarely remember to Cc:[email protected] and they're usually picked.

In case I guess I'd need to re-submit the patch right?

Quoted 7 lines
Quoted 2 lines

Did you see this cause measurable performance problems, or did you find it through inspection?

I noticed it while debugging an ENOMEM issue in the test_maps BPF selftest on PREEMPT_RT and this was an obvious cuplrit (irq_work not scheduled during a stress run). Turns out the problem is still there after this fix though.

I would imagine this to be the least of your problems, I think there's a bunch of blockers for complete selftests passing with PREEMPT_RT support in BPF.

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

The patch makes sense to me as well.

#5 lore raw
Quoted 6 lines · Kumar Kartikeya Dwivedi
Quoted 5 lines · Gabriele Monaco
Quoted 2 lines · Arnd Bergmann

Did you see this cause measurable performance problems, or did you find it through inspection?

Quoted 9 lines
Quoted 4 lines

I noticed it while debugging an ENOMEM issue in the test_maps BPF selftest on PREEMPT_RT and this was an obvious cuplrit (irq_work not scheduled during a stress run). Turns out the problem is still there after this fix though.

I would imagine this to be the least of your problems, I think there's a bunch of blockers for complete selftests passing with PREEMPT_RT support in BPF.

Well, I'm starting to believe that too..
At the moment on well tuned machines we are only observing ENOMEM issues in the test_maps and only when it's literally hogging the allocator (preallocation is off and 100 treads do updates in parallel).

What else are you expecting to fail under PREEMPT_RT?

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

The patch makes sense to me as well.

Thanks,
Gabriele

#6 lore raw
In reply to Gabriele Monaco
Quoted 7 lines · Gabriele Monaco
Quoted 1 line · Arnd Bergmann

Should this be Cc:[email protected] to get backported?

Not sure if the Fixes: is enough to trigger the automation, I rarely remember to Cc:[email protected] and they're usually picked.

In case I guess I'd need to re-submit the patch right?

Yes. For whatever reason the patch didn't reach the patchwork.

Please resubmit with [PATCH bpf-next] subject, so that CI can test it properly. And collect Acks.

#7 lore raw
In reply to Arnd Bergmann
Quoted 9 lines · Arnd Bergmann
Quoted 8 lines · Gabriele Monaco

raw_res_spin_unlock_irqrestore() calls raw_res_spin_unlock() and then restores interrupts, this means preemption is enabled when interrupts are still disabled (as part of raw_res_spin_unlock()) so this cannot trigger an actual preemption.
This is inconsistent with other spinlock implementations (raw_spin_unlock_irqrestore() and bpf_res_spin_unlock_irqrestore() itself).

Quoted 5 lines
Quoted 5 lines

Adjust the macro to ensure interrupts are enabled before enabling preemption, allowing to schedule at that point. Make the same modification in the error path of raw_res_spin_lock_irqsave().

Fixes 101acd2e78b1 ("rqspinlock: Add macros for rqspinlock usage")

Yeah, this is right. spinlocks always get one preempt_disable, in addition they might also get irq or bh disable.

Acked-by
#8 lore raw
In reply to Arnd Bergmann
Quoted 73 lines · Arnd Bergmann
Quoted 8 lines · Gabriele Monaco

raw_res_spin_unlock_irqrestore() calls raw_res_spin_unlock() and then restores interrupts, this means preemption is enabled when interrupts are still disabled (as part of raw_res_spin_unlock()) so this cannot trigger an actual preemption.
This is inconsistent with other spinlock implementations (raw_spin_unlock_irqrestore() and bpf_res_spin_unlock_irqrestore() itself).

Quoted 5 lines

Adjust the macro to ensure interrupts are enabled before enabling preemption, allowing to schedule at that point. Make the same modification in the error path of raw_res_spin_lock_irqsave().

Fixes 101acd2e78b1 ("rqspinlock: Add macros for rqspinlock usage")

Should this be Cc:[email protected] to get backported?

Did you see this cause measurable performance problems, or did you find it through inspection?

Quoted 1 line

This should probably get merged through the BPF tree, but I've added the kernel/locking maintainers to Cc as well, since I feel it's more useful to have them look at it than me.

Maybe it would be good to update (as a separate patch) the MAINTAINERS file so the locking subsystem also includes the headers currently missing:

arch/*/include/asm/*spinlock*.h
arch/*/include/asm/*rwlock*.h
include/asm-generic/*spinlock*.h
include/asm-generic/*rwlock*.h

Arnd

(full patch quoted below)

Quoted 34 lines
---
  include/asm-generic/rqspinlock.h | 14 +++++++++++---
  1 file changed, 11 insertions(+), 3 deletions(-)
include/asm-generic/rqspinlock.h +11 -3
index 151d267a49..4d46643f46 100644
@@ -243,12 +243,20 @@ static __always_inline void
 res_spin_unlock(rqspinlock_t *lock)
  	({                                        \
  		int __ret;                        \
  		local_irq_save(flags);            \
-		__ret = raw_res_spin_lock(lock);  \
-		if (__ret)                        \
+		preempt_disable();                \
+		__ret = res_spin_lock(lock);      \
+		if (__ret) {                      \
  			local_irq_restore(flags); \
+			preempt_enable();         \
+		}                                 \
  		__ret;                            \
  	})
 
-#define raw_res_spin_unlock_irqrestore(lock, flags) ({
 raw_res_spin_unlock(lock); local_irq_restore(flags); })
+#define raw_res_spin_unlock_irqrestore(lock, flags) \
+	({                                          \
+		res_spin_unlock(lock);              \
+		local_irq_restore(flags);           \
+		preempt_enable();                   \
+	})
Quoted 5 lines
Quoted 5 lines
  #endif /* __ASM_GENERIC_RQSPINLOCK_H */

base-commit: e43ffb69e0438cddd72aaa30898b4dc446f664f8

Signature · 1 line
2.54.0

Disabling interrupt will also disable preemption. However, the possible side effect of scheduler preemption when preemption is enabled will make these res_spin_lock APIs behave more like regular spinlock. So