Skip to contentSkip to conversation

RT: Fix trylock deadlock without msleep() hack

RFC PATCH New Steven Rostedt 1 participant 4 messages Sep 10, 2015

Subsystems (9)
Networking VFS Block layer Scheduler +5 more subsystems
lore
Share & tools
lore

Steven Rostedt

author reviewer
#0 lore raw

v2 cover RT: Fix trylock deadlock without msleep() hack

(Frozen shark deflectors set on hold)

I wont describe the trylock live lock issue again, for that please see v1 of this patch set:

http://lkml.kernel.org/r/[email protected]

My solution there was to add a spin_try_or_boost_lock() call that would temporarily boost the owner of the lock if the lock was held. The boosting will last until the task unlocks any lock, then it would reset that boost.

In the mean time, the caller of the trylock would then call cpu_chill() which was turned into a sched_yield() hoping that if it was blocking the owner the owner (now boosted to the same priority) would get a chance to run.

Thomas Gleixner had issues with this approach (and rightfully so).

#1) if the owner was blocked by an even higher priority task on another
CPU, the trylock spinner would constantly spin and block that CPU from having any other user.

#2) There was a temporary priority leak (although this isn't so bad
because the leak is bounded). Where if the trylock spinner hit a condition to not spin anymore. The owner of the lock would still have the unneeded boost till it released that lock.

#3) There's no connection with the trylock spinner and the owner. If
the spinner was deboosted, or boosted, the owner of the lock would not see that change.

With my new approach (this patch series), the above can be solved. Note, 1 and 2 are solved, but 3 still needs some tweaks with the priority chain. But first lets describe the new approach.

Before describing the new approach, I need to describe the rt_mutex_waiter. This is the descriptor used to hold most of the state of a blocked task in the priority inheritance chain. When a task blocks on a rtmutex, the rt_mutex_waiter (called waiter from now on) is allocated on the stack and initialized. It points to the lock as well as the blocked task. It has hooks to link into the owner's priority list as well as the lock's wait list. The waiter is used extensively in the pi code.

The waiter can be allocated on the stack because a blocked task wont need the pi code once it is woken and is finished with the rtmutex call. But this wont work with the spin_trylock_or_boost() (renamed from the previous spin_try_or_boost_lock()). That is because the trylock_or_boost call will finish and the waiter will still be in use. The scope of this waiter needs to be from the trylock_or_boost() call, all the way to the cpu_chill() call (which must be called after a failed spin_trylock_or_boost()).

To solve the need of a waiter beyond the scope of a function call, a waiter descriptor is added to the task_struct. This "static" waiter can now be used to boost tasks after a failed spin_trylock_or_boost() and keep the state between the trylock spinner and the owner of the lock.

Some caveats are needed though. A task can only boost one other task at a time. If two spin_trylock_or_boost() calls are done and both failed. Only the first one will boost the owner of the task. The second one (which would for now give a warning, because I don't see why two failed attempts would be done), will be ignored.

If the task calls spin_trylock_or_boost() fails, and boosts the owner, and then calls a normal spin_lock() and fails, the boosted task from the trylock_or_boost will be released. That is, the waiter on the task_struct will be removed from the lock and owner's pi list, before the task blocks on the spin_lock. This prevents any strange anomaly with having one task with two waiters. As stated before. A task may only boost one other task at a time, never two or more. That is, it may only be in one pi chain at a time. This keeps the complexity down.

When the task gets around to calling cpu_chill(), the cpu_chill() will check the "static" waiter, and see if there's a lock assigned to it. If there is, it will set a new flag in the waiter stating that it wants to be woken, and will sleep. If there's no lock assigned, that means either the lock was released in the mean time, or the task blocked on another lock and had to release that waiter. In either case, the cpu_chill() simply calls cpu_relax() and continues.

When the owner of the lock releases the lock, if the top_waiter is a static waiter (waiter == &waiter->task.rt_waiter), then the waiter is removed not only from the pi_list of the old owner, but also from the wait_list of the lock itself. The waiter->lock is set to NULL too. This lets the waiter->task know that the lock has been freed. The old owner will only wake up the task if the waiter flag is set to do so. Otherwise it may cause a spurious wake up. If the flag is not set, then the trylock spinner has not made it to the cpu_chill() yet, and with the waiter->lock cleared, the cpu_chill() will not sleep.

The waiter->task->pi_lock is used to synchronize the modifications of the static waiter.

Now, one might notice that we could just keep the task on the lock's wait_list and that way lower priority tasks could not steal that lock. This would be nice, but there's a major problem. And this problem is also the reason for patch 3 (a new patch in the series). That is, because the trylock spinner is not actually blocked on the lock, there's no guarantee that the lock will stay around by the time the spinner calls cpu_chill().

    CPU0		   CPU1
    ----		   ----
  lock(dev->B)
			lock(A)
			trylock(dev->B)
			  [boost and link on lock]
			unlock(A)

unlock(dev->B)
lock(A)
free(dev)
unlock(A)

			cpu_chill()

See the problem?

If the owner kept the trylock spinner on the lock, there's nothing in the logic that prevents that lock from being freed. In fact, in this case it was lock A that protects the lock from being freed. But the trylock spinner releases that lock before calling cpu_chill().

Because of the above, the unlock(dev->B) must remove all top waiters that are there due to a spin_trylock_or_boost(). If there are normal waiters, nothing needs to be done, because the normal waiter would have to be in a path that guarantees the lock stays around (otherwise it would bug on mainline too), and when the normal waiter gets the lock and releases it it will remove any waiters that are below it.

I boot tested this, and ran my simple module to see if it works, and it does. But this code is more complex, and I'm sure there's bugs in it. This is still in RFC, and there's some clean ups that can be done. But I think this is a better solution than the one I supplied before, albeit a bit more complex. But it handles the issues that Thomas brought up. #3 isn't fixed, because that will require the pi chain checking tasks that are not blocked, but boost via its static waiter. This would not be hard to fix, and probably can be fixed by changing cpu_chill() to boost again before it sleeps.

Thoughts?

-- Steve

---

Steven Rostedt (Red Hat) (3):
locking: Add spin_trylock_or_boost() infrastructure
rt: Make cpu_chill() call rt_mutex_wait_for_lock() and add new cpu_rest() as msleep(1)
rtmutex: Wake up all top trylock waiters on unlock

 block/blk-ioc.c                 |    4 
 fs/autofs4/expire.c             |    2 
 fs/dcache.c                     |    6 
 fs/namespace.c                  |    2 
 include/linux/delay.h           |   13 +
 include/linux/rtmutex.h         |    1 
 include/linux/sched.h           |   50 +++++
 include/linux/spinlock.h        |    5 
 include/linux/spinlock_rt.h     |   15 +
 kernel/locking/rtmutex.c        |  376 +++++++++++++++++++++++++++++++++++++++-
 kernel/locking/rtmutex_common.h |   22 --
 kernel/time/hrtimer.c           |   17 +
 kernel/workqueue.c              |    2 
 net/packet/af_packet.c          |    4 
 net/rds/ib_rdma.c               |    2 
 15 files changed, 478 insertions(+), 43 deletions(-)
author reviewer
#1 lore raw

RFC PATCH v2 1/3 locking: Add spin_trylock_or_boost() infrastructure

In mainline Linux, there are cases where locks need to be taken in reverse order. In non PREEMPT_RT code, spinlocks can not be preempted, thus if one needs to take locks in reverse order, if it can't get the second lock, it simply needs to release the first lock (the one that can cause a deadlock) and try again. The owner of the second lock must be running on another CPU and as long as it's not blocked on the held lock of the original task, then progress will be made.

But things change when these locks become preemptive locks.

Simply releasing the first lock when failing to get the second lock wont always work if the locks can be preempted. That's because the task could have preempted the owner, and it can prevent the owner from making forward progress. Just letting go of the first lock and trying again wont change the state of things, as the owner will still be blocked by being preempted by the original task. This is especially true when the original task is a real-time task running a FIFO prio.

<Task 1>
Grab lock A

<preempt to higher priority Task 2>

Grab lock B
try to grab lock A
release lock B
Grab lock B
try to grab lock A
<wash, rinse, repeat> (sorry Fred)

As Task 1 will never run as long as Task 2 is spinning, no forward progress is had, and this becomes a live lock (as suppose to a dead lock).

The solution here is to create add a static rt_mutex_waiter to the task_struct. The rt_mutex_waiter is a structure that is used to connect locks with owners and those that are blocked on the lock in order to propagate priority inheritance. But the rt_mutex_waiter is allocated on the stack when an rtmutex is blocked. This is fine, because the scope of that waiter structure will be finished by the time the rtmutex function returns. For the spin_trylock_or_boost() caller, the waiter's scope needs to go beyond the function call, thus it must not be allocated on the stack. Adding it to the task struct works, but only allows one trylock_or_boost to be called, but that should be fine anyway (a warning is added if it is not).

The spin_trylock_or_boost() will add this new task_struct waiter to the pi lists of the lock, and if necessary (if it is the highest waiter on the lock) to the owner's pi list too. This is exactly the way things work with the current pi inheritance logic.

Some things need to be changed though. As the waiter is not actually waiting on the lock, the owner can't wake it up without prejudice, as the waiter could have set itself in another state before calling cpu_chill(). A "wake_up" field is added to the waiter and is set only when the task is ready to receive the wakeup and it saves the current state.

When the owner of the lock releases the lock and picks this task to take the lock next, it will remove it from the pi lists and clear the waiter->lock. This is done under the task's pi_lock to make sure it synchronizes with the cpu_chill() call.

When cpu_chill() is called, it will look to see if the task's static waiter has a lock. If it does not, the cpu_chill will act like a cpu_relax(), as that means the owner released the lock. The cpu_chill() will then set the "wake_up" flag, save the state, and go to sleep in the TASK_INTERRUPTIBLE state, as anything should be able to wake it up.

When it is woken, it clears the lock (if it hasn't already been done) and removes itself from the pi lists.

Note, the reason that the owner of the lock clears the lock and does not let the task stay on the pi lists, is because nothing prevents the lock from being freed when there is no real owner of the task.

Note 2, if the task blocks on a rtmutex, or tries to call trylock_or_boost again and fails while already boosting a lock, then it must remove itself from the previous lock before it can boost another task. Only one boosting per task is allowed. A task can not boost more than one task.

---
 include/linux/rtmutex.h         |   1 +
 include/linux/sched.h           |  50 +++++++
 include/linux/spinlock.h        |   5 +
 include/linux/spinlock_rt.h     |  15 ++
 kernel/locking/rtmutex.c        | 323 +++++++++++++++++++++++++++++++++++++++-
 kernel/locking/rtmutex_common.h |  22 ---
 6 files changed, 386 insertions(+), 30 deletions(-)
include/linux/rtmutex.h +1 -0
index d5a04ea47a13..4ab8d4bddfbb 100644
@@ -111,6 +111,7 @@ extern int rt_mutex_timed_lock(struct rt_mutex *lock,
 			       struct hrtimer_sleeper *timeout);
 
 extern int rt_mutex_trylock(struct rt_mutex *lock);
+extern int rt_mutex_try_or_boost_lock(struct rt_mutex *lock);
 
 extern void rt_mutex_unlock(struct rt_mutex *lock);
 
include/linux/sched.h +50 -0

62 lines not on this page · raw message

include/linux/spinlock.h +5 -0
index 28f4366fd495..1d15e8c1267f 100644
@@ -330,6 +330,11 @@ static inline int spin_trylock(spinlock_t *lock)
 	return raw_spin_trylock(&lock->rlock);
 }
 
+static inline int spin_trylock_or_boost(spinlock_t *lock)
+{
+	return raw_spin_trylock(&lock->rlock);
+}
+
 #define spin_lock_nested(lock, subclass)			\
 do {								\
 	raw_spin_lock_nested(spinlock_check(lock), subclass);	\
include/linux/spinlock_rt.h +15 -0
index f757096b230c..e65951230e3e 100644
@@ -26,6 +26,7 @@ extern void __lockfunc rt_spin_unlock_wait(spinlock_t *lock);
 extern int __lockfunc rt_spin_trylock_irqsave(spinlock_t *lock, unsigned long *flags);
 extern int __lockfunc rt_spin_trylock_bh(spinlock_t *lock);
 extern int __lockfunc rt_spin_trylock(spinlock_t *lock);
+extern int __lockfunc rt_spin_trylock_or_boost(spinlock_t *lock);
 extern int atomic_dec_and_spin_lock(atomic_t *atomic, spinlock_t *lock);
 
 /*
@@ -53,6 +54,8 @@ extern int __lockfunc __rt_spin_trylock(struct rt_mutex *lock);
 
 #define spin_do_trylock(lock)		__cond_lock(lock, rt_spin_trylock(lock))
 
+#define spin_do_try_or_boost_lock(lock) __cond_lock(lock, rt_spin_trylock_or_boost(lock))
+
 #define spin_trylock(lock)			\
 ({						\
 	int __locked;				\
@@ -63,6 +66,16 @@ extern int __lockfunc __rt_spin_trylock(struct rt_mutex *lock);
 	__locked;				\
 })
 
+#define spin_trylock_or_boost(lock)			\
+({							\
+	int __locked;					\
+	migrate_disable();				\
+	__locked = spin_do_try_or_boost_lock(lock);	\
+	if (!__locked)					\
+		migrate_enable();			\
+	__locked;					\
+})
+
 #ifdef CONFIG_LOCKDEP
 # define spin_lock_nested(lock, subclass)		\
 	do {						\
@@ -153,6 +166,8 @@ static inline unsigned long spin_lock_trace_flags(spinlock_t *lock)
 # define spin_is_contended(lock)	(((void)(lock), 0))
 #endif
 
+void rt_mutex_wait_for_lock(struct task_struct *task);
+
 static inline int spin_can_lock(spinlock_t *lock)
 {
 	return !rt_mutex_is_locked(&lock->lock);
kernel/locking/rtmutex.c +315 -8

414 lines not on this page · raw message

kernel/locking/rtmutex_common.h +0 -20
index 4d317e9a5d0f..338e7fa4bbfa 100644
@@ -37,28 +37,6 @@ extern void schedule_rt_mutex_test(struct rt_mutex *lock);
 #endif
 
 /*
- * This is the control structure for tasks blocked on a rt_mutex,
- * which is allocated on the kernel stack on of the blocked task.
- *
- * @tree_entry:		pi node to enqueue into the mutex waiters tree
- * @pi_tree_entry:	pi node to enqueue into the mutex owner waiters tree
- * @task:		task reference to the blocked task
- */
-struct rt_mutex_waiter {
-	struct rb_node          tree_entry;
-	struct rb_node          pi_tree_entry;
-	struct task_struct	*task;
-	struct rt_mutex		*lock;
-	bool			savestate;
-#ifdef CONFIG_DEBUG_RT_MUTEXES
-	unsigned long		ip;
-	struct pid		*deadlock_task_pid;
-	struct rt_mutex		*deadlock_lock;
-#endif
-	int prio;
-};
author reviewer
#2 lore raw

RFC PATCH v2 2/3 rt: Make cpu_chill() call rt_mutex_wait_for_lock() and add new cpu_rest() as msleep(1)

In reply to Steven Rostedt

Make the cpu_chill() call rt_mutex_wait_for_lock() where it will only sleep if the owner of a lock boosted by spin_trylock_or_boost() still has the lock. The owner will then wake up the caller of cpu_chill().

As there are still locations that use cpu_chill(), as it spins on status events like bits and what not (not locks), where the updater is not known and can not be boosted, add a new cpu_rest() that will take over the msleep(1) action.

Hopefully, we can get rid of the cpu_rest() and find a way to know what tasks need priority boosting, and perhaps make another API that will allow boosting of the updater, and the current task can contine to spin instead of sleep.

Also convert trylock spinners over to spin_try_or_boost_lock(), which will later call cpu_chill().

When trying to take locks in reverse order, it is possible that on PREEMPT_RT that the running task could have preempted the owner and never let it run, creating a live lock. This is because spinlocks in PREEMPT_RT can be preempted.

Currently, this is solved by calling cpu_chill(), which on PREEMPT_RT is converted into a msleep(1), and we just hopen that the owner will have time to release the lock, and nobody else will take in when the task wakes up.

By converting these to spin_trylock_or_boost() which will boost the owners, the cpu_chill() now must be called afterward to sleep if the owner still has the lock, and the owner will do the wake up.

---
 block/blk-ioc.c        |  4 ++--
 fs/autofs4/expire.c    |  2 +-
 fs/dcache.c            |  6 +++---
 fs/namespace.c         |  2 +-
 include/linux/delay.h  | 13 +++++++++++++
 kernel/time/hrtimer.c  | 17 +++++++++++++++--
 kernel/workqueue.c     |  2 +-
 net/packet/af_packet.c |  4 ++--
 net/rds/ib_rdma.c      |  2 +-
 9 files changed, 39 insertions(+), 13 deletions(-)
block/blk-ioc.c +2 -2
index 28f467e636cc..93cf668ca314 100644
@@ -105,7 +105,7 @@ static void ioc_release_fn(struct work_struct *work)
 						struct io_cq, ioc_node);
 		struct request_queue *q = icq->q;
 
-		if (spin_trylock(q->queue_lock)) {
+		if (spin_trylock_or_boost(q->queue_lock)) {
 			ioc_destroy_icq(icq);
 			spin_unlock(q->queue_lock);
 		} else {
@@ -183,7 +183,7 @@ retry:
 	hlist_for_each_entry(icq, &ioc->icq_list, ioc_node) {
 		if (icq->flags & ICQ_EXITED)
 			continue;
-		if (spin_trylock(icq->q->queue_lock)) {
+		if (spin_trylock_or_boost(icq->q->queue_lock)) {
 			ioc_exit_icq(icq);
 			spin_unlock(icq->q->queue_lock);
 		} else {
fs/autofs4/expire.c +1 -1
index d487fa27add5..79eef1e3157e 100644
@@ -148,7 +148,7 @@ again:
 			}
 
 			parent = p->d_parent;
-			if (!spin_trylock(&parent->d_lock)) {
+			if (!spin_trylock_or_boost(&parent->d_lock)) {
 				spin_unlock(&p->d_lock);
 				cpu_chill();
 				goto relock;
fs/dcache.c +3 -3
index c1dad92434d5..151d2db0ded7 100644
@@ -573,12 +573,12 @@ static struct dentry *dentry_kill(struct dentry *dentry)
 	struct inode *inode = dentry->d_inode;
 	struct dentry *parent = NULL;
 
-	if (inode && unlikely(!spin_trylock(&inode->i_lock)))
+	if (inode && unlikely(!spin_trylock_or_boost(&inode->i_lock)))
 		goto failed;
 
 	if (!IS_ROOT(dentry)) {
 		parent = dentry->d_parent;
-		if (unlikely(!spin_trylock(&parent->d_lock))) {
+		if (unlikely(!spin_trylock_or_boost(&parent->d_lock))) {
 			if (inode)
 				spin_unlock(&inode->i_lock);
 			goto failed;
@@ -2394,7 +2394,7 @@ again:
 	inode = dentry->d_inode;
 	isdir = S_ISDIR(inode->i_mode);
 	if (dentry->d_lockref.count == 1) {
-		if (!spin_trylock(&inode->i_lock)) {
+		if (!spin_trylock_or_boost(&inode->i_lock)) {
 			spin_unlock(&dentry->d_lock);
 			cpu_chill();
 			goto again;
fs/namespace.c +1 -1
index 28937028f3a5..24769de44041 100644
@@ -356,7 +356,7 @@ int __mnt_want_write(struct vfsmount *m)
 	smp_mb();
 	while (ACCESS_ONCE(mnt->mnt.mnt_flags) & MNT_WRITE_HOLD) {
 		preempt_enable();
-		cpu_chill();
+		cpu_rest();
 		preempt_disable();
 	}
 	/*
include/linux/delay.h +13 -0
index 37caab306336..b787f4b11243 100644
@@ -53,9 +53,22 @@ static inline void ssleep(unsigned int seconds)
 }
 
 #ifdef CONFIG_PREEMPT_RT_FULL
+/*
+ * Use cpu_chill() after a spin_trylock_or_boost() which will boost the owner
+ * of the lock to the callers priority (if needed), and cpu_chill will
+ * act like a sched_yield() allowing the owner to proceed.
+ */
 extern void cpu_chill(void);
+/*
+ * Use cpu_rest() if there's no way to find out who the owner you are waiting
+ * for (like spinning on a status variable or bit). This is equivalent to
+ * a msleep(1) and you can hope that the status will change by the time
+ * you wake up.
+ */
+extern void cpu_rest(void);
 #else
 # define cpu_chill()	cpu_relax()
+# define cpu_rest()	cpu_relax()
 #endif
 
 #endif /* defined(_LINUX_DELAY_H) */
kernel/time/hrtimer.c +15 -2
index 2c6be169bdc7..6fd780ffa5d8 100644
@@ -1881,7 +1881,7 @@ SYSCALL_DEFINE2(nanosleep, struct timespec __user *, rqtp,
 /*
  * Sleep for 1 ms in hope whoever holds what we want will let it go.
  */
-void cpu_chill(void)
+void cpu_rest(void)
 {
 	struct timespec tu = {
 		.tv_nsec = NSEC_PER_MSEC,
@@ -1894,8 +1894,21 @@ void cpu_chill(void)
 	if (!freeze_flag)
 		current->flags &= ~PF_NOFREEZE;
 }
+EXPORT_SYMBOL(cpu_rest);
+
+/*
+ * Used after a spin_trylock_or_boost(), which should boost the owner
+ * of the lock to the priority of the current task (if needed), and
+ * this will yield the current task to the owner if the owner is on
+ * current's CPU.
+ */
+void cpu_chill(void)
+{
+	rt_mutex_wait_for_lock(current);
+}
 EXPORT_SYMBOL(cpu_chill);
-#endif
+
+#endif /* CONFIG_PREEMPT_RT_FULL */
 
 /*
  * Functions related to boot-time initialization:
kernel/workqueue.c +1 -1
index 21daecdfd86d..32b4d73349dd 100644
@@ -1260,7 +1260,7 @@ fail:
 	local_unlock_irqrestore(pendingb_lock, *flags);
 	if (work_is_canceling(work))
 		return -ENOENT;
-	cpu_chill();
+	cpu_rest();
 	return -EAGAIN;
 }
 
net/packet/af_packet.c +2 -2
index ef1eb20504a7..e906044802c8 100644
@@ -699,7 +699,7 @@ static void prb_retire_rx_blk_timer_expired(unsigned long data)
 	if (BLOCK_NUM_PKTS(pbd)) {
 		while (atomic_read(&pkc->blk_fill_in_prog)) {
 			/* Waiting for skb_copy_bits to finish... */
-			cpu_chill();
+			cpu_rest();
 		}
 	}
 
@@ -961,7 +961,7 @@ static void prb_retire_current_block(struct tpacket_kbdq_core *pkc,
 		if (!(status & TP_STATUS_BLK_TMO)) {
 			while (atomic_read(&pkc->blk_fill_in_prog)) {
 				/* Waiting for skb_copy_bits to finish... */
-				cpu_chill();
+				cpu_rest();
 			}
 		}
 		prb_close_block(pkc, pbd, po, status);
net/rds/ib_rdma.c +1 -1
index c8faaf36423a..31fe3b8b4cde 100644
@@ -287,7 +287,7 @@ static inline void wait_clean_list_grace(void)
 	for_each_online_cpu(cpu) {
 		flag = &per_cpu(clean_list_grace, cpu);
 		while (test_bit(CLEAN_LIST_BUSY_BIT, flag))
-			cpu_chill();
+			cpu_rest();
 	}
 }
 
Signature · 1 line
2.5.1
author reviewer
#3 lore raw

RFC PATCH v2 3/3 rtmutex: Wake up all top trylock waiters on unlock

In reply to Steven Rostedt

A task that boosts an owner of a lock via spin_trylock_or_boost() is not a real waiter of the lock in non PREEMPT_RT code. In non PREEMPT_RT, that task is just spinning. But in PREEMPT_RT the call to cpu_chill() will touch the lock. But there's nothing keeping the lock there.

As the lock is boosted via a trylock, it means something had to be done before we got that lock with another lock held out of reverse order. That means, if there's nothing using that lock, there's a possible code path that can make that lock disappear. Here's a fictitious example:

    CPU0	     CPU1		CPU2
    ----	     ----		----

[task0] [task1] [task2]

lock(dev->A)

		     lock(B)
		     trylock(dev->A)
		     unlock(B)
		     goto again
					lock(B)
					trylock(dev->A)
					unlock(B)
					goto again
  unlock(dev->A)
    wake(task1)
    remove_task1_links

lock(B)
free(dev)
unlock(B)

At this moment, although task1 is running and ready to go, task2 is still on dev->A->wait_list, and that will cause a panic when task2 does a cpu_chill().

Things are fine as long as there's a waiter that is from a rtmutex_lock(). Wake all the top tasks till a task is found that is blocked on the rtmutex itself.

---
 kernel/locking/rtmutex.c | 65 +++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 59 insertions(+), 6 deletions(-)
kernel/locking/rtmutex.c +59 -6
index 843b67f38e20..f26eebe5de87 100644
@@ -254,17 +254,25 @@ rt_mutex_dequeue_pi(struct task_struct *task, struct rt_mutex_waiter *waiter)
 }
 
 #ifdef CONFIG_PREEMPT_RT_FULL
+static void rt_mutex_wake_waiter(struct rt_mutex_waiter *waiter);
 /*
- * Returns true if the task should be woken up, false otherwise.
+ * Returns true if this is a trylock waiter.
  */
 static inline bool rt_mutex_wake_trylock_waiter(struct rt_mutex_waiter *waiter)
 {
-	struct task_struct *task = waiter->task;
+	struct task_struct *task;
+	struct rt_mutex *lock;
 	unsigned long flags;
 	bool wakeup;
+	bool trylock_waiter = false;
+
+again:
+	task = waiter->task;
 
 	if (likely(waiter != &task->rt_waiter))
-		return true;
+		return trylock_waiter;
+
+	trylock_waiter = true;
 
 	/*
 	 * A task boosted current because it is within a trylock spin.
@@ -276,12 +284,57 @@ static inline bool rt_mutex_wake_trylock_waiter(struct rt_mutex_waiter *waiter)
 	 */
 	raw_spin_lock_irqsave(&task->pi_lock, flags);
 	rt_mutex_dequeue(waiter->lock, waiter);
+	lock = waiter->lock;
 	waiter->lock = NULL;
 
 	wakeup = waiter->wake_up;
+	get_task_struct(task);
 	raw_spin_unlock_irqrestore(&task->pi_lock, flags);
 
-	return wakeup;
+	if (wakeup)
+		rt_mutex_wake_waiter(waiter);
+
+	put_task_struct(task);
+
+	/*
+	 * All tasks that are trylock waiters need to be woken up,
+	 * otherwise there's a chance that the lock may go away from
+	 * under them. Here's the scenario:
+	 *
+	 *    CPU0	     CPU1		CPU2
+	 *    ----	     ----		----
+	 *   [task0]	     [task1]		[task2]
+	 *
+	 *  lock(dev->A)
+	 *		     lock(B)
+	 *		     trylock(dev->A)
+	 *		     unlock(B)
+	 *		     goto again
+	 *					lock(B)
+	 *					trylock(dev->A)
+	 *					unlock(B)
+	 *					goto again
+	 *  unlock(dev->A)
+	 *    wake(task1)
+	 *    remove_task1_links
+	 *  lock(B)
+	 *  free(dev)
+	 *  unlock(B)
+	 *
+	 * At this moment, although task1 is running and ready
+	 * to go, task2 is still on dev->wait_list, and that will
+	 * cause a panic when task2 does a cpu_chill().
+	 *
+	 * Things are fine as long as there's a waiter that is
+	 * from a rtmutex_lock(). Keep waking tasks until we find
+	 * a rtmutex_lock() waiter.
+	 */
+
+	if (!rt_mutex_has_waiters(lock))
+		return true;
+
+	waiter = rt_mutex_top_waiter(lock);
+	goto again;
 }
 
 static void __rt_mutex_adjust_prio(struct task_struct *task);
@@ -496,7 +549,7 @@ static inline struct task_struct *trylock_boost_owner(struct rt_mutex *lock)
 }
 static inline bool rt_mutex_wake_trylock_waiter(struct rt_mutex_waiter *waiter)
 {
-	return true;
+	return false;
 }
 static inline bool check_static_waiter(struct task_struct *task,
 				       struct rt_mutex *lock, bool ok)
@@ -1654,7 +1707,7 @@ static void wakeup_next_waiter(struct rt_mutex *lock)
 
 	raw_spin_unlock_irqrestore(&current->pi_lock, flags);
 
-	if (!rt_mutex_wake_trylock_waiter(waiter))
+	if (rt_mutex_wake_trylock_waiter(waiter))
 		return;
 
 	/*
Signature · 1 line
2.5.1