Skip to content

locking, sched: The PREEMPT-RT locking infrastructure

Thomas Gleixner lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest50 patches 25 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/50sched: Split out the wakeup state checkkernel/sched/core.c+18 -60 replies
2/50sched: Introduce TASK_RTLOCK_WAITinclude/linux/sched.h+3 -10 replies
3/50sched: Prepare for RT sleeping spin/rwlocksinclude/linux/sched.h kernel/sched/core.c+103 -04 replies
4/50sched: Rework the __schedule() preempt argumentkernel/sched/core.c+25 -111 reply
5/50sched: Provide schedule point for RT locksinclude/linux/sched.h kernel/sched/core.c+23 -24 replies
6/50sched/wake_q: Provide WAKE_Q_HEAD_INITIALIZERinclude/linux/sched/wake_q.h+5 -20 replies
7/50rtmutex: Convert macros to inlineskernel/locking/rtmutex.c+27 -40 replies
8/50rtmutex: Switch to try_cmpxchg()kernel/locking/rtmutex.c+2 -20 replies
9/50rtmutex: Split API and implementationkernel/locking/Makefile kernel/locking/rtmutex.c kernel/locking/rtmutex_api.c +1 more+514 -4980 replies
10/50locking/rtmutex: Provide rt_mutex_slowlock_locked()kernel/locking/rtmutex.c kernel/locking/rtmutex_api.c kernel/locking/rtmutex_common.h+72 -422 replies
11/50locking/rtmutex: Provide lockdep less variants of rtmutex interfaceskernel/locking/rtmutex_api.c kernel/locking/rtmutex_common.h+53 -00 replies
12/50locking: Add base code for RT rw_semaphore and rwlockinclude/linux/rwbase_rt.h kernel/locking/rwbase_rt.c+300 -00 replies
13/50locking/rwsem: Add rtmutex based R/W semaphore implementationinclude/linux/rwsem.h kernel/locking/rwsem.c+166 -00 replies
14/50locking/rtmutex: Add wake_state to rt_mutex_waiterkernel/locking/rtmutex.c kernel/locking/rtmutex_common.h+10 -12 replies
15/50locking/rtmutex: Provide rt_mutex_wake_q and helperskernel/locking/rtmutex.c kernel/locking/rtmutex_common.h+29 -00 replies
16/50locking/rtmutex: Use rt_mutex_wake_q_headkernel/futex.c kernel/locking/rtmutex.c kernel/locking/rtmutex_api.c +1 more+19 -222 replies
17/50locking/rtmutex: Prepare RT rt_mutex_wake_q for RT lockskernel/locking/rtmutex.c kernel/locking/rtmutex_common.h+17 -22 replies
18/50locking/rtmutex: Guard regular sleeping locks specific functionskernel/locking/rtmutex.c kernel/locking/rtmutex_api.c+132 -1230 replies
19/50locking/spinlock: Split the lock types headerinclude/linux/rwlock_types.h include/linux/spinlock.h include/linux/spinlock_types.h +1 more+74 -180 replies
20/50locking/rtmutex: Prevent future include recursion hellinclude/linux/rtmutex.h+1 -10 replies
21/50locking/lockdep: Reduce includes in debug_locks.hinclude/linux/debug_locks.h+1 -20 replies
22/50rbtree: Split out the rbtree type definitionsinclude/linux/rbtree.h include/linux/rbtree_types.h+35 -292 replies
23/50locking/rtmutex: Include only rbtree typesinclude/linux/rtmutex.h+1 -10 replies
24/50locking/spinlock: Provide RT specific spinlock typeinclude/linux/spinlock_types.h+26 -02 replies
25/50locking/spinlock: Provide RT variant headerinclude/linux/spinlock.h include/linux/spinlock_api_smp.h include/linux/spinlock_rt.h+162 -10 replies
26/50locking/rtmutex: Provide the spin/rwlock core lock functionkernel/locking/rtmutex.c kernel/locking/rtmutex_common.h+61 -11 reply
27/50locking/spinlock: Provide RT variantkernel/locking/Makefile kernel/locking/spinlock_rt.c+129 -01 reply
28/50locking/rwlock: Provide RT variantinclude/linux/rwlock_rt.h include/linux/rwlock_types.h include/linux/spinlock_rt.h +4 more+317 -30 replies
29/50locking/mutex: Consolidate core headerskernel/locking/mutex-debug.c kernel/locking/mutex-debug.h kernel/locking/mutex.c +1 more+19 -400 replies
30/50locking/mutex: Move waiter to core headerinclude/linux/mutex.h kernel/locking/mutex.h+13 -130 replies
31/50locking/ww_mutex: Move ww_mutex declarations into ww_mutex.hinclude/linux/mutex.h include/linux/ww_mutex.h+8 -110 replies
32/50locking/mutex: Make mutex::wait_lock rawinclude/linux/mutex.h kernel/locking/mutex.c+13 -130 replies
33/50locking/mutex: Introduce _mutex_tinclude/linux/mutex.h+8 -21 reply
34/50locking/mutex: Rename the ww_mutex relevant functionsinclude/linux/mutex.h include/linux/ww_mutex.h kernel/locking/mutex-debug.c +1 more+52 -450 replies
35/50locking/ww_mutex: Switch to _mutex_tinclude/linux/ww_mutex.h+4 -40 replies
36/50locking/mutex: Replace struct mutex in core codekernel/locking/mutex-debug.c kernel/locking/mutex.c kernel/locking/mutex.h+39 -390 replies
37/50locking/mutex: Rearrange items in mutex.hinclude/linux/mutex.h+10 -100 replies
38/50locking/mutex: Exclude non-ww_mutex API for RTkernel/locking/mutex.c+23 -100 replies
39/50locking/rtmutex: Add mutex variant for RTinclude/linux/mutex.h kernel/locking/rtmutex_api.c+173 -20 replies
40/50lib/test_lockup: Adapt to changed variables.lib/test_lockup.c+4 -40 replies
41/50futex: Validate waiter correctly in futex_proxy_trylock_atomic()kernel/futex.c+4 -00 replies
42/50futex: Cleanup stale commentskernel/futex.c+3 -61 reply
43/50futex: Correct the number of requeued waiters for PIkernel/futex.c+4 -00 replies
44/50futex: Restructure futex_requeue()kernel/futex.c+41 -490 replies
45/50futex: Clarify comment in futex_requeue()kernel/futex.c+20 -80 replies
46/50futex: Prevent requeue_pi() lock nesting issue on RTkernel/futex.c+278 -710 replies
47/50rtmutex: Prevent lockdep false positive with PI futexeskernel/locking/rtmutex_api.c+12 -00 replies
48/50preempt: Adjust PREEMPT_LOCK_OFFSET for RTinclude/linux/preempt.h+4 -00 replies
49/50locking/rtmutex: Implement equal priority lock stealingkernel/locking/rtmutex.c+35 -170 replies
50/50locking/rtmutex: Add adaptive spinwait mechanismkernel/locking/rtmutex.c+49 -10 replies

Reviews trailers per patch

None of the 50 patches has a Reviewed-by or Acked-by.

PatchGregory HaskinsValentin Schneider
SeriesT-b
1/50
2/50
3/50
4/50
5/50
6/50
7/50
8/50
9/50
10/50
11/50
12/50
13/50
14/50
15/50
16/50
17/50
18/50
19/50
20/50
21/50
22/50
23/50
24/50
25/50
26/50
27/50
28/50
29/50
30/50
31/50
32/50
33/50
34/50
35/50
36/50
37/50
38/50
39/50
40/50
41/50
42/50
43/50
44/50
45/50
46/50
47/50
48/50
49/50
50/50Originally-by

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [patch 00/50] locking, sched: The PREEMPT-RT locking infrastructure

Folks,

the following series contains the bulk of the PREEMPT-RT locking infrastructure. In PREEMPT-RT enabled kernels the following locking primitives are substituted by RT-Mutex based variants:

mutex, rw_semaphore, spinlock, rwlock

semaphores are not substituted because they do not provide strict owner semantics.

ww_mutexes are also not substituted because the usage sites are not really RT relevant and it would require a full reimplementation to make them work correctly based on rtmutex. That might change in the future, but for now utilizing the existing variant is considered a safe and sane choice.

Of course raw_spinlocks are not touched either as they protect low level operations in the scheduler, timers and hardware access.

The most interesting parts of the series which need a lot of eyeballs are:

- the scheduler bits which provide the infrastructure for spinlock and rwlock substitution to ensure that the task state is preserved when blocking on such a lock and a regular wakeup is handled correctly and not lost

- the rtmutex core implementation to handle lock contention on spinlocks and rwlocks correctly vs. the task state

- the rw_semaphore/rwlock substitutions which utilize the same implementation vs. the reader/writer handling

- the isolation of the ww_mutex code which allows to build it stand alone. The typedef based solution might look a bit odd on the first glance, but that turned out to be the least intrusive variant.

- the PI futex related bits to handle the interaction between blocking on the underlying rtmutex and contention on the hash bucket lock which is converted to a 'sleeping spinlock'.

The rest surely needs a thorough review as well, but those parts are pretty straight forward. Quite some code restructuring and the actual wrapper functions to replace the existing !RT implementations.

The series survived quite some internal testing in RT kernels and is part of the recent 5.13-rt1 release.

For !RT kernels there is no functional change.

The series is also available from git:

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git rtmutex

Thanks,

tglx

---
 a/kernel/locking/mutex-debug.h       |   29 -
 b/include/linux/mutex.h              |  165 ++++--
 b/include/linux/rbtree_types.h       |   34 +
 b/include/linux/rwbase_rt.h          |   37 +
 b/include/linux/rwlock_rt.h          |  140 +++++
 b/include/linux/spinlock_rt.h        |  151 +++++
 b/include/linux/spinlock_types_raw.h |   65 ++
 b/kernel/locking/rtmutex_api.c       |  647 ++++++++++++++++++++++++
 b/kernel/locking/rwbase_rt.c         |  263 +++++++++
 b/kernel/locking/spinlock_rt.c       |  257 +++++++++
 include/linux/debug_locks.h          |    3 
 include/linux/preempt.h              |    4 
 include/linux/rbtree.h               |   30 -
 include/linux/rtmutex.h              |    4 
 include/linux/rwlock_types.h         |   39 +
 include/linux/rwsem.h                |   58 ++
 include/linux/sched.h                |   77 ++
 include/linux/sched/wake_q.h         |    7 
 include/linux/spinlock.h             |   15 
 include/linux/spinlock_api_smp.h     |    3 
 include/linux/spinlock_types.h       |   45 +
 include/linux/ww_mutex.h             |   16 
 kernel/Kconfig.locks                 |    2 
 kernel/futex.c                       |  466 ++++++++++++-----
 kernel/locking/Makefile              |    3 
 kernel/locking/mutex-debug.c         |   25 
 kernel/locking/mutex.c               |  139 ++---
 kernel/locking/mutex.h               |   35 +
 kernel/locking/rtmutex.c             |  930 ++++++++++++-----------------------
 kernel/locking/rtmutex_common.h      |  110 ++--
 kernel/locking/rwsem.c               |  108 ++++
 kernel/locking/spinlock.c            |    7 
 kernel/locking/spinlock_debug.c      |    5 
 kernel/sched/core.c                  |  111 +++-
 lib/test_lockup.c                    |    8 
 35 files changed, 3031 insertions(+), 1007 deletions(-)