Various rwsem ACQUIRE fixes
Subsystems (2)
Peter Zijlstra
maintainercover Various rwsem ACQUIRE fixes
These are the patches I ended up with after we started with Jan's patch (edited).
Peter Zijlstra
maintainerPATCH 1/4 locking/rwsem: Add missing ACQUIRE to read_slowpath exit when queue is empty
From: Jan Stancek <[email protected]>
LTP mtest06 has been observed to occasionally hit "still mapped when deleted" and following BUG_ON on arm64.
The extra mapcount originated from pagefault handler, which handled pagefault for vma that has already been detached. vma is detached under mmap_sem write lock by detach_vmas_to_be_unmapped(), which also invalidates vmacache.
When the pagefault handler (under mmap_sem read lock) calls find_vma(), vmacache_valid() wrongly reports vmacache as valid.
After rwsem down_read() returns via 'queue empty' path (as of v5.2), it does so without an ACQUIRE on sem->count:
down_read()
__down_read()
rwsem_down_read_failed()
__rwsem_down_read_failed_common()
raw_spin_lock_irq(&sem->wait_lock);
if (list_empty(&sem->wait_list)) {
if (atomic_long_read(&sem->count) >= 0) {
raw_spin_unlock_irq(&sem->wait_lock);
return sem;The problem can be reproduced by running LTP mtest06 in a loop and building the kernel (-j $NCPUS) in parallel. It does reproduces since v4.20 on arm64 HPE Apollo 70 (224 CPUs, 256GB RAM, 2 nodes). It triggers reliably in about an hour.
The patched kernel ran fine for 10+ hours.
--- kernel/locking/rwsem.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/locking/rwsem.c +2 -0
@@ -1032,6 +1032,8 @@ rwsem_down_read_slowpath(struct rw_semap*/ if (adjustment && !(atomic_long_read (&sem->count) & (RWSEM_WRITER_MASK |RWSEM_FLAG_HANDOFF ))) { +/* Provide lock ACQUIRE */ +smp_acquire__after_ctrl_dep ();raw_spin_unlock_irq (&sem->wait_lock);rwsem_set_reader_owned (sem);lockevent_inc (rwsem_rlock_fast);
Re: [PATCH 1/4] locking/rwsem: Add missing ACQUIRE to read_slowpath exit when queue is empty
kernel/locking/rwsem.c rwsem_down_read_slowpath(struct rw_semap · line 1036
Quoted 6 lines
kernel/locking/rwsem.c +2
-0@@ -1032,6 +1032,8 @@ rwsem_down_read_slowpath(struct rw_semap*/ if (adjustment && !(atomic_long_read (&sem->count) & (RWSEM_WRITER_MASK |RWSEM_FLAG_HANDOFF ))) { +/* Provide lock ACQUIRE */ +smp_acquire__after_ctrl_dep ();
Does this also make the lock RCtso? Or maybe RCtso was already guaranteed (and I'm failing to see why)?
Thanks,
Andrea
Peter Zijlstra
maintainerRe: [PATCH 1/4] locking/rwsem: Add missing ACQUIRE to read_slowpath exit when queue is empty
Quoted 9 lines · Andrea Parri
kernel/locking/rwsem.c rwsem_down_read_slowpath(struct rw_semap · line 1036
Quoted 6 lines
kernel/locking/rwsem.c +2
-0@@ -1032,6 +1032,8 @@ rwsem_down_read_slowpath(struct rw_semap*/ if (adjustment && !(atomic_long_read (&sem->count) & (RWSEM_WRITER_MASK |RWSEM_FLAG_HANDOFF ))) { +/* Provide lock ACQUIRE */ +smp_acquire__after_ctrl_dep ();Does this also make the lock RCtso? Or maybe RCtso was already guaranteed (and I'm failing to see why)?
I didn't specifically look for that, but I suspect not, esp given the next patch.
Peter Zijlstra
maintainerPATCH 2/4 rwsem: Add missing ACQUIRE to read_slowpath sleep loop
In reply to Peter ZijlstraWhile reviewing another read_slowpath patch, both Will and I noticed another missing ACQUIRE, namely:
X = 0;
CPU0 CPU1
rwsem_down_read()
for (;;) {
set_current_state(TASK_UNINTERRUPTIBLE); X = 1;
rwsem_up_write();
rwsem_mark_wake()
atomic_long_add(adjustment, &sem->count);
smp_store_release(&waiter->task, NULL);
if (!waiter.task)
break;
...
}r = X;
Allows 'r == 0'.
--- kernel/locking/rwsem.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/locking/rwsem.c +3 -1
@@ -1069,8 +1069,10 @@ rwsem_down_read_slowpath(struct rw_semap/* wait to be given the lock */ while (true) {set_current_state (state); -if (!waiter.task) +if (!smp_load_acquire (&waiter.task)) { +/* Orders against rwsem_mark_wake()'s smp_store_release() */ break ; + }if (signal_pending_state (state, current)) {raw_spin_lock_irq (&sem->wait_lock);if (waiter.task)
Re: [PATCH 2/4] rwsem: Add missing ACQUIRE to read_slowpath sleep loop
Quoted 42 lines · Peter Zijlstra
While reviewing another
read_slowpathpatch, both Will and I noticed another missing ACQUIRE, namely:X = 0;CPU0 CPU1
rwsem_down_read() for (;;) { set_current_state(TASK_UNINTERRUPTIBLE);X = 1; rwsem_up_write(); rwsem_mark_wake() atomic_long_add(adjustment, &sem->count); smp_store_release(&waiter->task, NULL); if (!waiter.task) break; ... }r = X;Allows 'r == 0'.
Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")Reported-by Peter Zijlstra (Intel) <[email protected]>Reported-by Will Deacon <[email protected]>Acked-by Will Deacon <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/locking/rwsem.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)kernel/locking/rwsem.c +0
-0@@ -1069,8 +1069,10 @@ rwsem_down_read_slowpath(struct rw_semap/* wait to be given the lock */
kernel/locking/rwsem.c rwsem_down_read_slowpath(struct rw_semap · line 1078
Quoted 10 lines
kernel/locking/rwsem.c +3
-1while (true) {set_current_state (state); -if (!waiter.task) +if (!smp_load_acquire (&waiter.task)) { +/* Orders against rwsem_mark_wake()'s smp_store_release() */ break ; + }if (signal_pending_state (state, current)) {raw_spin_lock_irq (&sem->wait_lock);if (waiter.task)
Thanks for fixing this old problem.
This problem does not apply to x86 and we do our testing mostly on x86. That is why we seldom notice this kind of issue. Maybe we should be doing more testing on ARM64 and PPC.
Cheers,
Longman
Peter Zijlstra
maintainerPATCH 3/4 tty/ldsem: Add missing ACQUIRE to read_failed sleep loop
In reply to Peter ZijlstraWhile reviewing rwsem down_slowpath, Will noticed ldsem had a copy of a bug we just found for rwsem.
X = 0;
CPU0 CPU1
rwsem_down_read()
for (;;) {
set_current_state(TASK_UNINTERRUPTIBLE); X = 1;
rwsem_up_write();
rwsem_mark_wake()
atomic_long_add(adjustment, &sem->count);
smp_store_release(&waiter->task, NULL);
if (!waiter.task)
break;
...
}r = X;
Allows 'r == 0'.
--- drivers/tty/tty_ldsem.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/tty/tty_ldsem.c +2 -3
@@ -93,8 +93,7 @@ static void __ldsem_wake_readers(structlist_for_each_entry_safe (waiter, next, &sem->read_wait, list) { tsk = waiter->task; -smp_mb (); - waiter->task =NULL ; +smp_store_release (&waiter->task,NULL );wake_up_process (tsk);put_task_struct (tsk); }
@@ -194,7 +193,7 @@ down_read_failed(struct ld_semaphore *sefor (;;) {set_current_state (TASK_UNINTERRUPTIBLE ); -if (!waiter.task) +if (!smp_load_acquire (&waiter.task))break ;if (!timeout)break ;
Peter Zijlstra
maintainerPATCH 4/4 rwsem: Add ACQUIRE comments
In reply to Peter ZijlstraSince we just reviewed read_slowpath for ACQUIRE correctness, add a few coments to retain our findings.
--- kernel/locking/rwsem.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
kernel/locking/rwsem.c +13 -5
@@ -1000,6 +1000,7 @@ rwsem_down_read_slowpath(struct rw_semapatomic_long_add (-RWSEM_READER_BIAS , &sem->count); adjustment =0 ;if (rwsem_optimistic_spin (sem, false)) { +/* rwsem_optimistic_spin() implies ACQUIRE on success */ /* * Wake up other readers in the wait list if the front * waiter is a reader.
@@ -1014,6 +1015,7 @@ rwsem_down_read_slowpath(struct rw_semap }return sem; }else if (rwsem_reader_phase_trylock (sem, waiter.last_rowner)) { +/* rwsem_reader_phase_trylock() implies ACQUIRE on success */ return sem; }
@@ -1067,10 +1069,10 @@ rwsem_down_read_slowpath(struct rw_semapwake_up_q (&wake_q);/* wait to be given the lock */ -while (true) { +for (;;) {set_current_state (state);if (!smp_load_acquire (&waiter.task)) { -/* Orders against rwsem_mark_wake()'s smp_store_release() */ +/* Matches rwsem_mark_wake()'s smp_store_release(). */ break ; }if (signal_pending_state (state, current)) {
@@ -1078,6 +1080,7 @@ rwsem_down_read_slowpath(struct rw_semapif (waiter.task) goto out_nolock;raw_spin_unlock_irq (&sem->wait_lock); +/* Ordered by sem->wait_lock against rwsem_mark_wake(). */ break ; }schedule ();
@@ -1087,6 +1090,7 @@ rwsem_down_read_slowpath(struct rw_semap__set_current_state (TASK_RUNNING );lockevent_inc (rwsem_rlock);return sem; + out_nolock:list_del (&waiter.list);if (list_empty (&sem->wait_list)) {
@@ -1127,8 +1131,10 @@ rwsem_down_write_slowpath(struct rw_sema/* do optimistic spinning and steal lock if possible */ if (rwsem_can_spin_on_owner (sem,RWSEM_WR_NONSPINNABLE ) && -rwsem_optimistic_spin (sem, true)) +rwsem_optimistic_spin (sem, true)) { +/* rwsem_optimistic_spin() implies ACQUIRE on success */ return sem; + }/* * Disable reader optimistic spinning for this rwsem after
@@ -1188,9 +1194,11 @@ rwsem_down_write_slowpath(struct rw_sema wait:/* wait until we successfully acquire the lock */ set_current_state (state); -while (true) { -if (rwsem_try_write_lock (sem, wstate)) +for (;;) { +if (rwsem_try_write_lock (sem, wstate)) { +/* rwsem_try_write_lock() implies ACQUIRE on success */ break ; + }raw_spin_unlock_irq (&sem->wait_lock);
Quoted 2 lines · Peter Zijlstra
These are the patches I ended up with after we started with Jan's patch (edited).
This series looks good to me.