osq_lock on arm64
Commit f5bfdc8e3947 change the osq_lock (optimistic spin queue) code to use smp_cond_load_relaxed() to wait for another cpu to relase the lock.
It waits for 'VAL || need_resched()' with a comment that the IPI that sets need_resched() will break any monitor/wait code. However arm64 implements it as (03e3c2b7edbe and 1cfc63b5ae6):
#define smp_cond_load_acquire(ptr, cond_expr) \
({ \
typeof(ptr) __PTR = (ptr); \
typeof(*ptr) VAL; \
for (;;) { \
VAL = smp_load_acquire(__PTR); \
if (cond_expr) \
break; \
__cmpwait_relaxed(__PTR, VAL); \
} \
VAL; \})
#define __CMPWAIT_CASE(w, sz, name) \
static inline void __cmpwait_case_##name(volatile void *ptr, \
unsigned long val) \
{ \
unsigned long tmp; \
\
asm volatile( \
" sevl\n" \
" wfe\n" \
" ldxr" #sz "\t%" #w "[tmp], %[v]\n" \
" eor %" #w "[tmp], %" #w "[tmp], %" #w "[val]\n" \
" cbnz %" #w "[tmp], 1f\n" \
" wfe\n" \
"1:" \
: [tmp] "=&r" (tmp), [v] "+Q" (*(unsigned long *)ptr) \
: [val] "r" (val)); \
}The "eor" and "cbnz" seem to be there to skip the "wfe" if the value has already changed, but this only works if the condition only depends on VAL. This means it can miss the IPI that makes need_resched() return non-zero.
To work I think it would need the cmp_load_acquire() to be "ldxr" with a "wfe" instead of __cmpwait_relaxed().
But that wouldn't work if 'cond_expr' issued another "ldx".
The documentation I found for the arm64 instruction set doesn't mention that losing the exclusive access requested by ldx will wake wfe, I assume that is an omission from the docs?
David
On Sun, 15 Mar 2026 20:11:53 +0000,
David Laight <[email protected]> wrote:
[...]
Quoted 3 lines
The documentation I found for the arm64 instruction set doesn't mention that losing the exclusive access requested by ldx will wake wfe, I assume that is an omission from the docs?
R_XRZRK explicitly calls for "The clearing of the global monitor for the PE." as a something that sets the event register, and therefore acts as a wake-up event from WFE.
M.
Signature · 1 line
Jazz isn't dead. It just smells funny.