Skip to contentSkip to conversation

Various rwsem ACQUIRE fixes

PATCH In review Peter Zijlstra 4 participants 9 messages Jul 18, 2019 – Jul 29, 2019

Subsystems (2)
lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

cover Various rwsem ACQUIRE fixes

These are the patches I ended up with after we started with Jan's patch (edited).

author maintainer
#1 lore raw

PATCH 1/4 locking/rwsem: Add missing ACQUIRE to read_slowpath exit when queue is empty

From: Jan Stancek <[email protected]>

LTP mtest06 has been observed to occasionally hit "still mapped when deleted" and following BUG_ON on arm64.

The extra mapcount originated from pagefault handler, which handled pagefault for vma that has already been detached. vma is detached under mmap_sem write lock by detach_vmas_to_be_unmapped(), which also invalidates vmacache.

When the pagefault handler (under mmap_sem read lock) calls find_vma(), vmacache_valid() wrongly reports vmacache as valid.

After rwsem down_read() returns via 'queue empty' path (as of v5.2), it does so without an ACQUIRE on sem->count:

  down_read()
    __down_read()
      rwsem_down_read_failed()
        __rwsem_down_read_failed_common()
          raw_spin_lock_irq(&sem->wait_lock);
          if (list_empty(&sem->wait_list)) {
            if (atomic_long_read(&sem->count) >= 0) {
              raw_spin_unlock_irq(&sem->wait_lock);
              return sem;

The problem can be reproduced by running LTP mtest06 in a loop and building the kernel (-j $NCPUS) in parallel. It does reproduces since v4.20 on arm64 HPE Apollo 70 (224 CPUs, 256GB RAM, 2 nodes). It triggers reliably in about an hour.

The patched kernel ran fine for 10+ hours.

---
 kernel/locking/rwsem.c |    2 ++
 1 file changed, 2 insertions(+)
kernel/locking/rwsem.c +2 -0
@@ -1032,6 +1032,8 @@ rwsem_down_read_slowpath(struct rw_semap
 		 */
 		if (adjustment && !(atomic_long_read(&sem->count) &
 		     (RWSEM_WRITER_MASK | RWSEM_FLAG_HANDOFF))) {
+			/* Provide lock ACQUIRE */
+			smp_acquire__after_ctrl_dep();
 			raw_spin_unlock_irq(&sem->wait_lock);
 			rwsem_set_reader_owned(sem);
 			lockevent_inc(rwsem_rlock_fast);
#2 lore raw

Re: [PATCH 1/4] locking/rwsem: Add missing ACQUIRE to read_slowpath exit when queue is empty

kernel/locking/rwsem.c rwsem_down_read_slowpath(struct rw_semap · line 1036

Quoted 6 lines
kernel/locking/rwsem.c +2 -0
@@ -1032,6 +1032,8 @@ rwsem_down_read_slowpath(struct rw_semap
 		 */
 		if (adjustment && !(atomic_long_read(&sem->count) &
 		     (RWSEM_WRITER_MASK | RWSEM_FLAG_HANDOFF))) {
+			/* Provide lock ACQUIRE */
+			smp_acquire__after_ctrl_dep();

Does this also make the lock RCtso? Or maybe RCtso was already guaranteed (and I'm failing to see why)?

Thanks,
Andrea

author maintainer
#3 lore raw

Re: [PATCH 1/4] locking/rwsem: Add missing ACQUIRE to read_slowpath exit when queue is empty

Quoted 9 lines · Andrea Parri

kernel/locking/rwsem.c rwsem_down_read_slowpath(struct rw_semap · line 1036

Quoted 6 lines
kernel/locking/rwsem.c +2 -0
@@ -1032,6 +1032,8 @@ rwsem_down_read_slowpath(struct rw_semap
 		 */
 		if (adjustment && !(atomic_long_read(&sem->count) &
 		     (RWSEM_WRITER_MASK | RWSEM_FLAG_HANDOFF))) {
+			/* Provide lock ACQUIRE */
+			smp_acquire__after_ctrl_dep();

Does this also make the lock RCtso? Or maybe RCtso was already guaranteed (and I'm failing to see why)?

I didn't specifically look for that, but I suspect not, esp given the next patch.

author maintainer
#4 lore raw

PATCH 2/4 rwsem: Add missing ACQUIRE to read_slowpath sleep loop

In reply to Peter Zijlstra

While reviewing another read_slowpath patch, both Will and I noticed another missing ACQUIRE, namely:

  X = 0;

CPU0 CPU1

  rwsem_down_read()
    for (;;) {
      set_current_state(TASK_UNINTERRUPTIBLE);
                        X = 1;
                        rwsem_up_write();
                          rwsem_mark_wake()
                            atomic_long_add(adjustment, &sem->count);
                            smp_store_release(&waiter->task, NULL);

      if (!waiter.task)
        break;

      ...
    }
  r = X;

Allows 'r == 0'.

---
 kernel/locking/rwsem.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/locking/rwsem.c +3 -1
@@ -1069,8 +1069,10 @@ rwsem_down_read_slowpath(struct rw_semap
 	/* wait to be given the lock */
 	while (true) {
 		set_current_state(state);
-		if (!waiter.task)
+		if (!smp_load_acquire(&waiter.task)) {
+			/* Orders against rwsem_mark_wake()'s smp_store_release() */
 			break;
+		}
 		if (signal_pending_state(state, current)) {
 			raw_spin_lock_irq(&sem->wait_lock);
 			if (waiter.task)
reviewer Acked-by on v1 2/4
#5 lore raw

Re: [PATCH 2/4] rwsem: Add missing ACQUIRE to read_slowpath sleep loop

Quoted 42 lines · Peter Zijlstra

While reviewing another read_slowpath patch, both Will and I noticed another missing ACQUIRE, namely:

  X = 0;

CPU0 CPU1

  rwsem_down_read()
    for (;;) {
      set_current_state(TASK_UNINTERRUPTIBLE);
                        X = 1;
                        rwsem_up_write();
                          rwsem_mark_wake()
                            atomic_long_add(adjustment, &sem->count);
                            smp_store_release(&waiter->task, NULL);

      if (!waiter.task)
        break;

      ...
    }
  r = X;

Allows 'r == 0'.

---
 kernel/locking/rwsem.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/locking/rwsem.c +0 -0
@@ -1069,8 +1069,10 @@ rwsem_down_read_slowpath(struct rw_semap
 	/* wait to be given the lock */

kernel/locking/rwsem.c rwsem_down_read_slowpath(struct rw_semap · line 1078

Quoted 10 lines
kernel/locking/rwsem.c +3 -1
 	while (true) {
 		set_current_state(state);
-		if (!waiter.task)
+		if (!smp_load_acquire(&waiter.task)) {
+			/* Orders against rwsem_mark_wake()'s smp_store_release() */
 			break;
+		}
 		if (signal_pending_state(state, current)) {
 			raw_spin_lock_irq(&sem->wait_lock);
 			if (waiter.task)

Thanks for fixing this old problem.

This problem does not apply to x86 and we do our testing mostly on x86. That is why we seldom notice this kind of issue. Maybe we should be doing more testing on ARM64 and PPC.

Cheers,
Longman

author maintainer
#6 lore raw

PATCH 3/4 tty/ldsem: Add missing ACQUIRE to read_failed sleep loop

In reply to Peter Zijlstra

While reviewing rwsem down_slowpath, Will noticed ldsem had a copy of a bug we just found for rwsem.

  X = 0;

CPU0 CPU1

  rwsem_down_read()
    for (;;) {
      set_current_state(TASK_UNINTERRUPTIBLE);
                        X = 1;
                        rwsem_up_write();
                          rwsem_mark_wake()
                            atomic_long_add(adjustment, &sem->count);
                            smp_store_release(&waiter->task, NULL);

      if (!waiter.task)
        break;

      ...
    }
  r = X;

Allows 'r == 0'.

Cc Peter Hurley <[email protected]>
Fixes 4898e640caf0 ("tty: Add timed, writer-prioritized rw semaphore")
---
 drivers/tty/tty_ldsem.c |    5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/tty/tty_ldsem.c +2 -3
@@ -93,8 +93,7 @@ static void __ldsem_wake_readers(struct
 
 	list_for_each_entry_safe(waiter, next, &sem->read_wait, list) {
 		tsk = waiter->task;
-		smp_mb();
-		waiter->task = NULL;
+		smp_store_release(&waiter->task, NULL);
 		wake_up_process(tsk);
 		put_task_struct(tsk);
 	}
@@ -194,7 +193,7 @@ down_read_failed(struct ld_semaphore *se
 	for (;;) {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 
-		if (!waiter.task)
+		if (!smp_load_acquire(&waiter.task))
 			break;
 		if (!timeout)
 			break;
author maintainer
#7 lore raw

PATCH 4/4 rwsem: Add ACQUIRE comments

In reply to Peter Zijlstra

Since we just reviewed read_slowpath for ACQUIRE correctness, add a few coments to retain our findings.

---
 kernel/locking/rwsem.c |   18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)
kernel/locking/rwsem.c +13 -5
@@ -1000,6 +1000,7 @@ rwsem_down_read_slowpath(struct rw_semap
 	atomic_long_add(-RWSEM_READER_BIAS, &sem->count);
 	adjustment = 0;
 	if (rwsem_optimistic_spin(sem, false)) {
+		/* rwsem_optimistic_spin() implies ACQUIRE on success */
 		/*
 		 * Wake up other readers in the wait list if the front
 		 * waiter is a reader.
@@ -1014,6 +1015,7 @@ rwsem_down_read_slowpath(struct rw_semap
 		}
 		return sem;
 	} else if (rwsem_reader_phase_trylock(sem, waiter.last_rowner)) {
+		/* rwsem_reader_phase_trylock() implies ACQUIRE on success */
 		return sem;
 	}
 
@@ -1067,10 +1069,10 @@ rwsem_down_read_slowpath(struct rw_semap
 	wake_up_q(&wake_q);
 
 	/* wait to be given the lock */
-	while (true) {
+	for (;;) {
 		set_current_state(state);
 		if (!smp_load_acquire(&waiter.task)) {
-			/* Orders against rwsem_mark_wake()'s smp_store_release() */
+			/* Matches rwsem_mark_wake()'s smp_store_release(). */
 			break;
 		}
 		if (signal_pending_state(state, current)) {
@@ -1078,6 +1080,7 @@ rwsem_down_read_slowpath(struct rw_semap
 			if (waiter.task)
 				goto out_nolock;
 			raw_spin_unlock_irq(&sem->wait_lock);
+			/* Ordered by sem->wait_lock against rwsem_mark_wake(). */
 			break;
 		}
 		schedule();
@@ -1087,6 +1090,7 @@ rwsem_down_read_slowpath(struct rw_semap
 	__set_current_state(TASK_RUNNING);
 	lockevent_inc(rwsem_rlock);
 	return sem;
+
 out_nolock:
 	list_del(&waiter.list);
 	if (list_empty(&sem->wait_list)) {
@@ -1127,8 +1131,10 @@ rwsem_down_write_slowpath(struct rw_sema
 
 	/* do optimistic spinning and steal lock if possible */
 	if (rwsem_can_spin_on_owner(sem, RWSEM_WR_NONSPINNABLE) &&
-	    rwsem_optimistic_spin(sem, true))
+	    rwsem_optimistic_spin(sem, true)) {
+		/* rwsem_optimistic_spin() implies ACQUIRE on success */
 		return sem;
+	}
 
 	/*
 	 * Disable reader optimistic spinning for this rwsem after
@@ -1188,9 +1194,11 @@ rwsem_down_write_slowpath(struct rw_sema
 wait:
 	/* wait until we successfully acquire the lock */
 	set_current_state(state);
-	while (true) {
-		if (rwsem_try_write_lock(sem, wstate))
+	for (;;) {
+		if (rwsem_try_write_lock(sem, wstate)) {
+			/* rwsem_try_write_lock() implies ACQUIRE on success */
 			break;
+		}
 
 		raw_spin_unlock_irq(&sem->wait_lock);