Linux v4.19.255-rt114-rc1
Subsystems (5)
cover Linux v4.19.255-rt114-rc1
Dear RT Folks,
This is the RT stable review cycle of patch 4.19.255-rt114-rc1.
Please scream at me if I messed something up. Please test the patches too.
The -rc release will be uploaded to kernel.org and will be deleted when the final release is out. This is just a review release (or release candidate).
The pre-releases will not be pushed to the git repository, only the final release is.
If all goes well, this patch will be converted to the next main release on 2022-10-31.
To build 4.19.255-rt114-rc1 directly, the following patches should be applied:
https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.19.tar.xz
https://www.kernel.org/pub/linux/kernel/v4.x/patch-4.19.255.xz
https://www.kernel.org/pub/linux/kernel/projects/rt/4.19/older/patch-4.19.255-rt114-rc1.patch.xz
Signing key fingerprint:
5BF6 7BC5 0826 72CA BB45 ACAE 587C 5ECA 5D0A 306C
All keys used for the above files and repositories can be found on the following git repository:
git://git.kernel.org/pub/scm/docs/kernel/pgpkeys.git
Enjoy!
Daniel
Changes from v4.19.255-rt113:
Daniel Wagner (3):
Revert "random: Use local locks for crng context access"
rcu: Update rcuwait
Linux 4.19.255-rt114-rc1
Sebastian Andrzej Siewior (6):
random: Bring back the local_locks
local_lock: Provide INIT_LOCAL_LOCK().
Revert "workqueue: Use local irq lock instead of irq disable regions"
timers: Keep interrupts disabled for TIMER_IRQSAFE timer.
timers: Don't block on ->expiry_lock for TIMER_IRQSAFE timers
workqueue: Use rcuwait for wq_manager_wait
drivers/char/random.c | 16 +++++++------ include/linux/locallock.h | 5 +++++ include/linux/rcuwait.h | 42 +++++++++++++++++++++++++++-------- kernel/exit.c | 7 ++++-- kernel/locking/percpu-rwsem.c | 2 +- kernel/rcu/update.c | 8 +++++++ kernel/time/timer.c | 12 ++++++++-- kernel/workqueue.c | 28 +++++++++++++++++------ localversion-rt | 2 +- 9 files changed, 93 insertions(+), 29 deletions(-)
Signature · 1 line
2.38.0
Quoted 11 lines · Daniel Wagner
Dear RT Folks,
This is the RT stable review cycle of patch 4.19.255-rt114-rc1.
Please scream at me if I messed something up. Please test the patches too.
The -rc release will be uploaded to kernel.org and will be deleted when the final release is out. This is just a review release (or release candidate).
Quoted 5 lines
The pre-releases will not be pushed to the git repository, only the final release is.
If all goes well, this patch will be converted to the next main release on 2022-10-31.
Timer changes seem not to be correct though:
[ 24.674424] BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:974 [ 24.674426] in_atomic(): 0, irqs_disabled(): 1, pid: 23, name: ktimersoftd/1 [ 25.730421] BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:974 [ 25.730424] in_atomic(): 0, irqs_disabled(): 1, pid: 11, name: ktimersoftd/0
I get those for when running any of the rttests. I suppose I am missing an additional fix:
- if (!IS_ENABLED(CONFIG_PREEMPT_RT_FULL) &&
- timer->flags & TIMER_IRQSAFE) {
+ if (timer->flags & TIMER_IRQSAFE) {
raw_spin_unlock(&base->lock);
call_timer_fn(timer, fn);
base->running_timer = NULL;is now queuing up fn callbacks with TIMER_IRQSAFE which then triggers:
+ if (IS_ENABLED(CONFIG_PREEMPT_RT) && !(timer->flags & TIMER_IRQSAFE))
+ might_sleep();
in del_timer_sync(). But this is just a guess.
Daniel
cover timers: backport timer bits
I've been looking at sleeping-while-atomic splats in v4.19.255-rt114-rc1.
It received a backport of an intermediate state of the timers patches which were part of v5.0 during the development. It remained unchanged in the v5.2 cycle and got merged slightly different upstream in v5.4.
This is a small series for v4.19.255-rt114-rc1 which contains the patch, that was merged in v5.4 and the two fixes that followed on top.
Sebastian
PATCH 1/3 timers: Prepare support for PREEMPT_RT
From: Anna-Maria Gleixner <[email protected]>
Upstream commit 030dcdd197d77374879bb5603d091eee7d8aba80
When PREEMPT_RT is enabled, the soft interrupt thread can be preempted. If the soft interrupt thread is preempted in the middle of a timer callback, then calling del_timer_sync() can lead to two issues:
- If the caller is on a remote CPU then it has to spin wait for the timer handler to complete. This can result in unbound priority inversion.
- If the caller originates from the task which preempted the timer handler on the same CPU, then spin waiting for the timer handler to complete is never going to end.
To avoid these issues, add a new lock to the timer base which is held around the execution of the timer callbacks. If del_timer_sync() detects that the timer callback is currently running, it blocks on the expiry lock. When the callback is finished, the expiry lock is dropped by the softirq thread which wakes up the waiter and the system makes progress.
This addresses both the priority inversion and the life lock issues.
This mechanism is not used for timers which are marked IRQSAFE as for those preemption is disabled accross the callback and therefore this situation cannot happen. The callbacks for such timers need to be individually audited for RT compliance.
The same issue can happen in virtual machines when the vCPU which runs a timer callback is scheduled out. If a second vCPU of the same guest calls del_timer_sync() it will spin wait for the other vCPU to be scheduled back in. The expiry lock mechanism would avoid that. It'd be trivial to enable this when paravirt spinlocks are enabled in a guest, but it's not clear whether this is an actual problem in the wild, so for now it's an RT only mechanism.
As the softirq thread can be preempted with PREEMPT_RT=y, the SMP variant of del_timer_sync() needs to be used on UP as well.
[ tglx: Refactored it for mainline ]
--- kernel/time/timer.c | 144 +++++++++++++++++++++++++++++--------------- 1 file changed, 95 insertions(+), 49 deletions(-)
kernel/time/timer.c +95 -49
@@ -198,7 +198,10 @@ EXPORT_SYMBOL(jiffies_64);struct timer_base {raw_spinlock_t lock;struct timer_list *running_timer; +#ifdef CONFIG_PREEMPT_RT spinlock_t expiry_lock; +atomic_t timer_waiters; +#endif unsigned long clk;unsigned long next_expiry;unsigned int cpu;
@@ -1227,25 +1230,6 @@ int del_timer(struct timer_list *timer) }EXPORT_SYMBOL (del_timer); -static int __try_to_del_timer_sync (struct timer_list *timer, -struct timer_base **basep) -{ -struct timer_base *base; -unsigned long flags; -int ret =-1 ; - -debug_assert_init (timer); - - *basep = base =lock_timer_base (timer, &flags); - -if (base->running_timer != timer) - ret =detach_if_pending (timer, base, true); - -raw_spin_unlock_irqrestore (&base->lock, flags); - -return ret; -} -/** * try_to_del_timer_sync - Try to deactivate a timer * @timer: timer to delete
@@ -1256,34 +1240,94 @@ static int __try_to_del_timer_sync(struct timer_list *timer,int try_to_del_timer_sync (struct timer_list *timer) {struct timer_base *base; +unsigned long flags; +int ret =-1 ; -return __try_to_del_timer_sync (timer, &base); +debug_assert_init (timer); + + base =lock_timer_base (timer, &flags); + +if (base->running_timer != timer) + ret =detach_if_pending (timer, base, true); + +raw_spin_unlock_irqrestore (&base->lock, flags); + +return ret; }EXPORT_SYMBOL (try_to_del_timer_sync); -#if defined(CONFIG_SMP ) || defined(CONFIG_PREEMPT_RT_FULL ) -static int __del_timer_sync (struct timer_list *timer) +#ifdef CONFIG_PREEMPT_RT +static __init void timer_base_init_expiry_lock (struct timer_base *base) { -struct timer_base *base; -int ret; +spin_lock_init (&base->expiry_lock); +} -for (;;) { - ret =__try_to_del_timer_sync (timer, &base); -if (ret >=0 ) -return ret; +static inline void timer_base_lock_expiry (struct timer_base *base) +{ +spin_lock (&base->expiry_lock); +} -if (READ_ONCE (timer->flags) &TIMER_IRQSAFE ) -continue ; +static inline void timer_base_unlock_expiry (struct timer_base *base) +{ +spin_unlock (&base->expiry_lock); +} -/* -* When accessing the lock, timers of base are no longer expired -* and so timer is no longer running. -*/ -spin_lock (&base->expiry_lock); +/* +* The counterpart to del_timer_wait_running(). +* +* If there is a waiter for base->expiry_lock, then it was waiting for the +* timer callback to finish. Drop expiry_lock and reaquire it. That allows +* the waiter to acquire the lock and make progress. +*/ +static void timer_sync_wait_running (struct timer_base *base) +{ +if (atomic_read (&base->timer_waiters)) {spin_unlock (&base->expiry_lock); +spin_lock (&base->expiry_lock); } } +/* +* This function is called on PREEMPT_RT kernels when the fast path +* deletion of a timer failed because the timer callback function was +* running. +* +* This prevents priority inversion, if the softirq thread on a remote CPU +* got preempted, and it prevents a life lock when the task which tries to +* delete a timer preempted the softirq thread running the timer callback +* function. +*/ +static void del_timer_wait_running (struct timer_list *timer) +{ +u32 tf; + + tf =READ_ONCE (timer->flags); +if (!(tf &TIMER_MIGRATING )) { +struct timer_base *base =get_timer_base (tf); + +/* +* Mark the base as contended and grab the expiry lock, +* which is held by the softirq across the timer +* callback. Drop the lock immediately so the softirq can +* expire the next timer. In theory the timer could already +* be running again, but that's more than unlikely and just +* causes another wait loop. +*/ +atomic_inc (&base->timer_waiters); +spin_lock_bh (&base->expiry_lock); +atomic_dec (&base->timer_waiters); +spin_unlock_bh (&base->expiry_lock); + } +} +#else +static inline void timer_base_init_expiry_lock (struct timer_base *base) { } +static inline void timer_base_lock_expiry (struct timer_base *base) { } +static inline void timer_base_unlock_expiry (struct timer_base *base) { } +static inline void timer_sync_wait_running (struct timer_base *base) { } +static inline void del_timer_wait_running (struct timer_list *timer) { } +#endif + +#if defined(CONFIG_SMP ) || defined(CONFIG_PREEMPT_RT_FULL )/** * del_timer_sync - deactivate a timer and wait for the handler to finish. * @timer: the timer to be deactivated
@@ -1322,6 +1366,8 @@ static int __del_timer_sync(struct timer_list *timer)*/ int del_timer_sync (struct timer_list *timer) { +int ret; +#ifdef CONFIG_LOCKDEP unsigned long flags;
@@ -1339,14 +1385,17 @@ int del_timer_sync(struct timer_list *timer)* could lead to deadlock. */ WARN_ON (in_irq () && !(timer->flags &TIMER_IRQSAFE )); -/* -* Must be able to sleep on PREEMPT_RT because of the slowpath in -* __del_timer_sync(). -*/ -if (IS_ENABLED (CONFIG_PREEMPT_RT ) && !(timer->flags &TIMER_IRQSAFE )) -might_sleep (); - return __del_timer_sync(timer); +do { + ret =try_to_del_timer_sync (timer); + +if (unlikely (ret <0 )) { +del_timer_wait_running (timer); +cpu_relax (); + } + }while (ret <0 ); + + return ret; }EXPORT_SYMBOL (del_timer_sync);#endif
@@ -1410,15 +1459,12 @@ static void expire_timers(struct timer_base *base, struct hlist_head *head)raw_spin_unlock (&base->lock);call_timer_fn (timer, fn); base->running_timer =NULL ; -spin_unlock (&base->expiry_lock); -spin_lock (&base->expiry_lock);raw_spin_lock (&base->lock); }else {raw_spin_unlock_irq (&base->lock);call_timer_fn (timer, fn); base->running_timer =NULL ; -spin_unlock (&base->expiry_lock); -spin_lock (&base->expiry_lock); +timer_sync_wait_running (base);raw_spin_lock_irq (&base->lock); } }
@@ -1715,7 +1761,7 @@ static inline void __run_timers(struct timer_base *base)if (!time_after_eq (jiffies, base->clk))return ; -spin_lock (&base->expiry_lock); +timer_base_lock_expiry (base);raw_spin_lock_irq (&base->lock);/*
@@ -1743,7 +1789,7 @@ static inline void __run_timers(struct timer_base *base)expire_timers (base, heads + levels); }raw_spin_unlock_irq (&base->lock); -spin_unlock (&base->expiry_lock); +timer_base_unlock_expiry (base); }/*
@@ -1990,7 +2036,7 @@ static void __init init_timer_cpu(int cpu) base->cpu = cpu;raw_spin_lock_init (&base->lock); base->clk = jiffies; -spin_lock_init (&base->expiry_lock); +timer_base_init_expiry_lock (base); } }
Signature · 1 line
2.37.2
PATCH 2/3 timers: Move clearing of base::timer_running under base:: Lock
In reply to Sebastian Andrzej SiewiorFrom: Thomas Gleixner <[email protected]>
Upstream commit bb7262b295472eb6858b5c49893954794027cd84
syzbot reported KCSAN data races vs. timer_base::timer_running being set to NULL without holding base::lock in expire_timers().
This looks innocent and most reads are clearly not problematic, but Frederic identified an issue which is:
int data = 0;
void timer_func(struct timer_list *t)
{
data = 1;}
CPU 0 CPU 1 ------------------------------ -------------------------- base = lock_timer_base(timer, &flags); raw_spin_unlock(&base->lock); if (base->running_timer != timer) call_timer_fn(timer, fn, baseclk); ret = detach_if_pending(timer, base, true); base->running_timer = NULL; raw_spin_unlock_irqrestore(&base->lock, flags); raw_spin_lock(&base->lock);
x = data;
If the timer has previously executed on CPU 1 and then CPU 0 can observe base->running_timer == NULL and returns, assuming the timer has completed, but it's not guaranteed on all architectures. The comment for del_timer_sync() makes that guarantee. Moving the assignment under base->lock prevents this.
For non-RT kernel it's performance wise completely irrelevant whether the store happens before or after taking the lock. For an RT kernel moving the store under the lock requires an extra unlock/lock pair in the case that there is a waiter for the timer, but that's not the end of the world.
--- kernel/time/timer.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/time/timer.c +4 -2
@@ -1282,8 +1282,10 @@ static inline void timer_base_unlock_expiry(struct timer_base *base)static void timer_sync_wait_running (struct timer_base *base) {if (atomic_read (&base->timer_waiters)) { +raw_spin_unlock_irq (&base->lock);spin_unlock (&base->expiry_lock);spin_lock (&base->expiry_lock); +raw_spin_lock_irq (&base->lock); } }
@@ -1458,14 +1460,14 @@ static void expire_timers(struct timer_base *base, struct hlist_head *head)if (timer->flags &TIMER_IRQSAFE ) {raw_spin_unlock (&base->lock);call_timer_fn (timer, fn); - base->running_timer =NULL ;raw_spin_lock (&base->lock); + base->running_timer =NULL ; }else {raw_spin_unlock_irq (&base->lock);call_timer_fn (timer, fn); +raw_spin_lock_irq (&base->lock); base->running_timer =NULL ;timer_sync_wait_running (base); -raw_spin_lock_irq (&base->lock); } } }
Signature · 1 line
2.37.2
PATCH 3/3 timers: Don't block on ->expiry_lock for TIMER_IRQSAFE timers
In reply to Sebastian Andrzej SiewiorUpstream commit c725dafc95f1b37027840aaeaa8b7e4e9cd20516
PREEMPT_RT does not spin and wait until a running timer completes its callback but instead it blocks on a sleeping lock to prevent a livelock in the case that the task waiting for the callback completion preempted the callback.
This cannot be done for timers flagged with TIMER_IRQSAFE. These timers can be canceled from an interrupt disabled context even on RT kernels.
The expiry callback of such timers is invoked with interrupts disabled so there is no need to use the expiry lock mechanism because obviously the callback cannot be preempted even on RT kernels.
Do not use the timer_base::expiry_lock mechanism when waiting for a running callback to complete if the timer is flagged with TIMER_IRQSAFE.
Also add a lockdep assertion for RT kernels to validate that the expiry lock mechanism is always invoked in preemptible context.
[ bigeasy: Dropping that lockdep_assert_preemption_enabled() check in
backport ]
--- kernel/time/timer.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
kernel/time/timer.c +10 -1
@@ -1304,7 +1304,7 @@ static void del_timer_wait_running(struct timer_list *timer)u32 tf; tf =READ_ONCE (timer->flags); -if (!(tf &TIMER_MIGRATING )) { +if (!(tf & (TIMER_MIGRATING |TIMER_IRQSAFE ))) {struct timer_base *base =get_timer_base (tf);/*
@@ -1388,6 +1388,15 @@ int del_timer_sync(struct timer_list *timer)*/ WARN_ON (in_irq () && !(timer->flags &TIMER_IRQSAFE )); +/* +* Must be able to sleep on PREEMPT_RT because of the slowpath in +* del_timer_wait_running(). +*/ +#if 0 +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && !(timer->flags &TIMER_IRQSAFE )) +lockdep_assert_preemption_enabled (); +#endif + do { ret = try_to_del_timer_sync(timer);
Signature · 1 line
2.37.2
Re: [PATCH RT 0/3] timers: backport timer bits
In reply to Sebastian Andrzej SiewiorHi Sebastian,
Quoted 8 lines · Sebastian Andrzej Siewior
I've been looking at sleeping-while-atomic splats in v4.19.255-rt114-rc1.
It received a backport of an intermediate state of the timers patches which were part of v5.0 during the development. It remained unchanged in the v5.2 cycle and got merged slightly different upstream in v5.4.This is a small series for v4.19.255-rt114-rc1 which contains the patch, that was merged in v5.4 and the two fixes that followed on top.
Thanks a lot for looking into it. With these patches the sleep in atomic context is gone and the quick test results are also looking good. I let it run a bit longer than usual just to make sure.
Daniel
PATCH 1/9 Revert "random: Use local locks for crng context access"
In reply to Daniel Wagnerv4.19.255-rt114-rc1 stable review patch.
If anyone has any objections, please let me know.
-----------
This reverts commit af5469c6f4f85f60f3ecc9bd541adfb6bdbeaff2.
--- drivers/char/random.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
drivers/char/random.c +4 -7
@@ -53,7 +53,6 @@#include <linux/uaccess.h> #include <linux/siphash.h> #include <linux/uio.h> -#include <linux/locallock.h> #include <crypto/chacha20.h> #include <crypto/blake2s.h> #include <asm/processor.h>
@@ -235,7 +234,6 @@ struct crng { static DEFINE_PER_CPU(struct crng, crngs) = { .generation =ULONG_MAX }; -DEFINE_LOCAL_IRQ_LOCK (crngs_lock);/* Used by crng_reseed() and crng_make_state() to extract a new seed from the input pool. */ static void extract_entropy (void *buf,size_t len);
@@ -364,7 +362,7 @@ static void crng_make_state(u32 chacha_state[CHACHA20_BLOCK_SIZE / sizeof(u32)],if (unlikely (crng_has_old_seed ()))crng_reseed (); -local_lock_irqsave (crngs_lock, flags); +local_irq_save (flags); crng =raw_cpu_ptr (&crngs);/*
@@ -389,7 +387,7 @@ static void crng_make_state(u32 chacha_state[CHACHA20_BLOCK_SIZE / sizeof(u32)],* should wind up here immediately. */ crng_fast_key_erasure (crng->key, chacha_state, random_data, random_data_len); -local_unlock_irqrestore (crngs_lock, flags); +local_irq_restore (flags); }static void _get_random_bytes (void *buf,size_t len)
@@ -514,7 +512,6 @@ struct batch_ ##type { \ static DEFINE_PER_CPU(struct batch_ ##type, batched_entropy_ ##type) = { \ .position = UINT_MAX \ }; \ -static DEFINE_LOCAL_IRQ_LOCK(batched_entropy_lock_ ##type); \ \ type get_random_ ##type(void) \ { \
@@ -530,7 +527,7 @@ type get_random_ ##type(void) \return ret; \ } \ \ - local_lock_irqsave(batched_entropy_lock_ ##type, flags); \ +local_irq_save (flags); \ batch = raw_cpu_ptr(&batched_entropy_##type); \ \ next_gen =READ_ONCE (base_crng.generation); \
@@ -544,7 +541,7 @@ type get_random_ ##type(void) \ ret = batch->entropy[batch->position]; \ batch->entropy[batch->position] =0 ; \ ++batch->position; \ - local_unlock_irqrestore(batched_entropy_lock_ ##type, flags); \ +local_irq_restore (flags); \return ret; \ } \ EXPORT_SYMBOL(get_random_ ##type);
Signature · 1 line
2.38.0
PATCH 2/9 random: Bring back the local_locks
In reply to Daniel WagnerFrom: Sebastian Andrzej Siewior <[email protected]>
v4.19.255-rt114-rc1 stable review patch.
If anyone has any objections, please let me know.
-----------
As part of the backports the random code lost its local_lock_t type and the whole operation became a local_irq_{disable|enable}() simply because the older kernel did not provide those primitives.
RT as of v4.9 has a slightly different variant of local_locks. Replace the local_irq_*() operations with matching local_lock_irq*() operations which were there as part of commit
77760fd7f7ae3 ("random: remove batched entropy locking")
--- drivers/char/random.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
drivers/char/random.c +10 -5
@@ -53,6 +53,7 @@#include <linux/uaccess.h> #include <linux/siphash.h> #include <linux/uio.h> +#include <linux/locallock.h> #include <crypto/chacha20.h> #include <crypto/blake2s.h> #include <asm/processor.h>
@@ -229,10 +230,12 @@ static struct {struct crng {u8 key[CHACHA20_KEY_SIZE ];unsigned long generation; +struct local_irq_lock lock; }; static DEFINE_PER_CPU(struct crng, crngs) = { - .generation =ULONG_MAX + .generation =ULONG_MAX , + .lock.lock =__SPIN_LOCK_UNLOCKED (crngs.lock.lock), };/* Used by crng_reseed() and crng_make_state() to extract a new seed from the input pool. */
@@ -362,7 +365,7 @@ static void crng_make_state(u32 chacha_state[CHACHA20_BLOCK_SIZE / sizeof(u32)],if (unlikely (crng_has_old_seed ()))crng_reseed (); -local_irq_save (flags); +local_lock_irqsave (crngs.lock, flags); crng =raw_cpu_ptr (&crngs);/*
@@ -387,7 +390,7 @@ static void crng_make_state(u32 chacha_state[CHACHA20_BLOCK_SIZE / sizeof(u32)],* should wind up here immediately. */ crng_fast_key_erasure (crng->key, chacha_state, random_data, random_data_len); -local_irq_restore (flags); +local_unlock_irqrestore (crngs.lock, flags); }static void _get_random_bytes (void *buf,size_t len)
@@ -505,11 +508,13 @@ struct batch_ ##type { \* formula of (integer_blocks + 0.5) * CHACHA20_BLOCK_SIZE. \ */ \type entropy[CHACHA20_BLOCK_SIZE *3 / (2 *sizeof (type))]; \ +struct local_irq_lock lock; \unsigned long generation; \unsigned int position; \ }; \ \ static DEFINE_PER_CPU(struct batch_ ##type, batched_entropy_ ##type) = { \ + .lock.lock = __SPIN_LOCK_UNLOCKED(batched_entropy_ ##type.lock.lock), \ .position = UINT_MAX \ }; \ \
@@ -527,7 +532,7 @@ type get_random_ ##type(void) \return ret; \ } \ \ -local_irq_save (flags); \ + local_lock_irqsave(batched_entropy_ ##type.lock, flags); \ batch = raw_cpu_ptr(&batched_entropy_##type); \ \ next_gen = READ_ONCE(base_crng.generation); \
@@ -541,7 +546,7 @@ type get_random_ ##type(void) \ ret = batch->entropy[batch->position]; \ batch->entropy[batch->position] =0 ; \ ++batch->position; \ -local_irq_restore (flags); \ + local_unlock_irqrestore(batched_entropy_ ##type.lock, flags); \ return ret; \ } \ EXPORT_SYMBOL(get_random_ ##type);
Signature · 1 line
2.38.0
PATCH 3/9 local_lock: Provide INIT_LOCAL_LOCK().
In reply to Daniel WagnerFrom: Sebastian Andrzej Siewior <[email protected]>
v4.19.255-rt114-rc1 stable review patch.
If anyone has any objections, please let me know.
-----------
The original code was using INIT_LOCAL_LOCK() and I tried to sneak around it and forgot that this code also needs to compile on !RT platforms.
Provide INIT_LOCAL_LOCK() to initialize properly on RT and do nothing on !RT. Let random.c use which is the only user so far and oes not compile on !RT otherwise.
--- drivers/char/random.c | 4 ++-- include/linux/locallock.h | 5 +++++ 2 files changed, 7 insertions(+), 2 deletions(-)
drivers/char/random.c +2 -2
@@ -235,7 +235,7 @@ struct crng { static DEFINE_PER_CPU(struct crng, crngs) = { .generation =ULONG_MAX , - .lock.lock =__SPIN_LOCK_UNLOCKED (crngs.lock.lock), + .lock =INIT_LOCAL_LOCK (crngs.lock), };/* Used by crng_reseed() and crng_make_state() to extract a new seed from the input pool. */
@@ -514,7 +514,7 @@ struct batch_ ##type { \ }; \ \ static DEFINE_PER_CPU(struct batch_ ##type, batched_entropy_ ##type) = { \ - .lock.lock = __SPIN_LOCK_UNLOCKED(batched_entropy_ ##type.lock.lock), \ + .lock = INIT_LOCAL_LOCK(batched_entropy_ ##type.lock), \ .position = UINT_MAX \ }; \ \
include/linux/locallock.h +5 -0
@@ -23,6 +23,8 @@ struct local_irq_lock {unsigned long flags; }; +#define INIT_LOCAL_LOCK (lvar) { .lock = __SPIN_LOCK_UNLOCKED((lvar).lock.lock) } +#define DEFINE_LOCAL_IRQ_LOCK (lvar) \ DEFINE_PER_CPU(struct local_irq_lock, lvar) = { \ .lock = __SPIN_LOCK_UNLOCKED((lvar).lock) }
@@ -241,6 +243,9 @@ static inline int __local_unlock_irqrestore(struct local_irq_lock *lv,#else /* PREEMPT_RT_BASE */ +struct local_irq_lock { }; +#define INIT_LOCAL_LOCK (lvar) { } +#define DEFINE_LOCAL_IRQ_LOCK (lvar) __typeof__(const int) lvar#define DECLARE_LOCAL_IRQ_LOCK (lvar) extern __typeof__(const int) lvar
Signature · 1 line
2.38.0
PATCH 4/9 Revert "workqueue: Use local irq lock instead of irq disable regions"
In reply to Daniel WagnerFrom: Sebastian Andrzej Siewior <[email protected]>
v4.19.255-rt114-rc1 stable review patch.
If anyone has any objections, please let me know.
-----------
This reverts the PREEMPT_RT related changes to workqueue. It reverts the usage of local_locks() and cpu_chill().
This is a preparation to pull in the PREEMPT_RT related changes which were merged upstream.
[wagi: 827b6f6962da ("workqueue: rework") already reverted
most of the changes, except the missing update in put_pwq_unlocked.]
--- kernel/workqueue.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
kernel/workqueue.c +1 -3
@@ -1496,11 +1496,9 @@ EXPORT_SYMBOL(queue_work_on);void delayed_work_timer_fn (struct timer_list *t) {struct delayed_work *dwork =from_timer (dwork, t, timer); -unsigned long flags; -local_irq_save (flags); +/* should have been called from irqsafe timer with irq already off */ __queue_work (dwork->cpu, dwork->wq, &dwork->work); -local_irq_restore (flags); }EXPORT_SYMBOL (delayed_work_timer_fn);
Signature · 1 line
2.38.0
PATCH 5/9 timers: Keep interrupts disabled for TIMER_IRQSAFE timer.
In reply to Daniel WagnerFrom: Sebastian Andrzej Siewior <[email protected]>
v4.19.255-rt114-rc1 stable review patch.
If anyone has any objections, please let me know.
-----------
Keep interrupts disabled across callback invocation for the TIMER_IRQSAFE as expected.
This is required for the timer used by workqueue after the upcomming rework.
--- kernel/time/timer.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/time/timer.c +1 -2
@@ -1397,8 +1397,7 @@ static void expire_timers(struct timer_base *base, struct hlist_head *head) fn = timer->function; -if (!IS_ENABLED (CONFIG_PREEMPT_RT_FULL ) && - timer->flags &TIMER_IRQSAFE ) { +if (timer->flags &TIMER_IRQSAFE ) {raw_spin_unlock (&base->lock);call_timer_fn (timer, fn); base->running_timer =NULL ;
Signature · 1 line
2.38.0
PATCH 6/9 timers: Don't block on ->expiry_lock for TIMER_IRQSAFE timers
In reply to Daniel WagnerFrom: Sebastian Andrzej Siewior <[email protected]>
v4.19.255-rt114-rc1 stable review patch.
If anyone has any objections, please let me know.
-----------
Upstream commit c725dafc95f1b37027840aaeaa8b7e4e9cd20516
PREEMPT_RT does not spin and wait until a running timer completes its callback but instead it blocks on a sleeping lock to prevent a livelock in the case that the task waiting for the callback completion preempted the callback.
This cannot be done for timers flagged with TIMER_IRQSAFE. These timers can be canceled from an interrupt disabled context even on RT kernels.
The expiry callback of such timers is invoked with interrupts disabled so there is no need to use the expiry lock mechanism because obviously the callback cannot be preempted even on RT kernels.
Do not use the timer_base::expiry_lock mechanism when waiting for a running callback to complete if the timer is flagged with TIMER_IRQSAFE.
Also add a lockdep assertion for RT kernels to validate that the expiry lock mechanism is always invoked in preemptible context.
[bigeasy: The logic in v4.19 is slightly different but the outcome is the
same as we must not sleep while waiting for the irqsafe timer to complete. The IRQSAFE timer can not be preempted.
The "lockdep annotation" is not available and has been replaced with might_sleep()]
--- kernel/time/timer.c | 9 +++++++++ 1 file changed, 9 insertions(+)
kernel/time/timer.c +9 -0
@@ -1272,6 +1272,9 @@ static int __del_timer_sync(struct timer_list *timer)if (ret >=0 )return ret; +if (READ_ONCE (timer->flags) &TIMER_IRQSAFE ) +continue ; +/* * When accessing the lock, timers of base are no longer expired * and so timer is no longer running.
@@ -1336,6 +1339,12 @@ int del_timer_sync(struct timer_list *timer)* could lead to deadlock. */ WARN_ON (in_irq () && !(timer->flags &TIMER_IRQSAFE )); +/* +* Must be able to sleep on PREEMPT_RT because of the slowpath in +* __del_timer_sync(). +*/ +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && !(timer->flags &TIMER_IRQSAFE )) +might_sleep ();return __del_timer_sync (timer); }
Signature · 1 line
2.38.0
PATCH 7/9 rcu: Update rcuwait
In reply to Daniel Wagnerv4.19.255-rt114-rc1 stable review patch.
If anyone has any objections, please let me know.
-----------
This is an all in one commit backporting updates for rcuwait:
- 03f4b48edae7 ("rcuwait: Annotate task_struct with __rcu")
- 191a43be61d6 ("rcuwait: Introduce rcuwait_active()")
- 5c21f7b322cb ("rcuwait: Introduce prepare_to and finish_rcuwait")
- 80fbaf1c3f29 ("rcuwait: Add @state argument to rcuwait_wait_event()")
- 9d9a6ebfea32 ("rcuwait: Let rcuwait_wake_up() return whether or not a task was awoken")
- 58d4292bd037 ("rcu: Uninline multi-use function: finish_rcuwait()")
--- include/linux/rcuwait.h | 42 +++++++++++++++++++++++++++-------- kernel/exit.c | 7 ++++-- kernel/locking/percpu-rwsem.c | 2 +- kernel/rcu/update.c | 8 +++++++ 4 files changed, 47 insertions(+), 12 deletions(-)
include/linux/rcuwait.h +33 -9
@@ -3,6 +3,7 @@#define _LINUX_RCUWAIT_H_#include <linux/rcupdate.h> +#include <linux/sched/signal.h> /* * rcuwait provides a way of blocking and waking up a single
@@ -18,7 +19,7 @@* awoken. */ struct rcuwait { -struct task_struct *task; +struct task_struct __rcu *task; };#define __RCUWAIT_INITIALIZER (name) \
@@ -29,14 +30,33 @@ static inline void rcuwait_init(struct rcuwait *w) w->task =NULL ; } -extern void rcuwait_wake_up (struct rcuwait *w); +extern int rcuwait_wake_up (struct rcuwait *w); + +/* +* Note: this provides no serialization and, just as with waitqueues, +* requires care to estimate as to whether or not the wait is active. +*/ +static inline int rcuwait_active (struct rcuwait *w) +{ +return !!rcu_access_pointer (w->task); +}/* * The caller is responsible for locking around rcuwait_wait_event(), -* such that writes to @task are properly serialized. +* and [prepare_to/finish]_rcuwait() such that writes to @task are +* properly serialized. */ -#define rcuwait_wait_event (w, condition) \ + +static inline void prepare_to_rcuwait (struct rcuwait *w) +{ +rcu_assign_pointer (w->task, current); +} + +extern void finish_rcuwait (struct rcuwait *w); + +#define rcuwait_wait_event (w, condition, state) \ ({ \ + int __ret = 0; \/* \ * Complain if we are called after do_exit()/exit_notify(), \ * as we cannot rely on the rcu critical region for the \
@@ -44,21 +64,25 @@ extern void rcuwait_wake_up(struct rcuwait *w);*/ \WARN_ON (current->exit_state); \ \ -rcu_assign_pointer ((w)->task, current); \ +prepare_to_rcuwait (w); \for (;;) { \/* \ * Implicit barrier (A) pairs with (B) in \ * rcuwait_wake_up(). \ */ \ -set_current_state (TASK_UNINTERRUPTIBLE ); \ +set_current_state (state); \if (condition) \break ; \ \ +if (signal_pending_state (state, current)) { \ + __ret = -EINTR ; \ +break ; \ + } \ + \schedule (); \ } \ - \ -WRITE_ONCE ((w)->task,NULL ); \ -__set_current_state (TASK_RUNNING ); \ +finish_rcuwait (w); \ + __ret; \ })#endif /* _LINUX_RCUWAIT_H_ */
kernel/exit.c +5 -2
@@ -291,8 +291,9 @@ struct task_struct *task_rcu_dereference(struct task_struct **ptask)return task; } -void rcuwait_wake_up (struct rcuwait *w) +int rcuwait_wake_up (struct rcuwait *w) { +int ret =0 ;struct task_struct *task;rcu_read_lock ();
@@ -316,8 +317,10 @@ void rcuwait_wake_up(struct rcuwait *w)*/ task =rcu_dereference (w->task);if (task) -wake_up_process (task); + ret =wake_up_process (task);rcu_read_unlock (); + +return ret; }/*
kernel/locking/percpu-rwsem.c +1 -1
@@ -159,7 +159,7 @@ void percpu_down_write(struct percpu_rw_semaphore *sem)*/ /* Wait for all now active readers to complete. */ -rcuwait_wait_event (&sem->writer,readers_active_check (sem)); +rcuwait_wait_event (&sem->writer,readers_active_check (sem),TASK_UNINTERRUPTIBLE ); }EXPORT_SYMBOL_GPL (percpu_down_write);
kernel/rcu/update.c +8 -0
@@ -53,6 +53,7 @@#include <linux/rcupdate_wait.h> #include <linux/sched/isolation.h> #include <linux/kprobes.h> +#include <linux/rcuwait.h> #define CREATE_TRACE_POINTS
@@ -375,6 +376,13 @@ void __wait_rcu_gp(bool checktiny, int n, call_rcu_func_t *crcu_array, }EXPORT_SYMBOL_GPL (__wait_rcu_gp); +void finish_rcuwait (struct rcuwait *w) +{ +rcu_assign_pointer (w->task,NULL ); +__set_current_state (TASK_RUNNING ); +} +EXPORT_SYMBOL_GPL (finish_rcuwait); +#ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD void init_rcu_head (struct rcu_head *head) {
Signature · 1 line
2.38.0
PATCH 8/9 workqueue: Use rcuwait for wq_manager_wait
In reply to Daniel WagnerFrom: Sebastian Andrzej Siewior <[email protected]>
v4.19.255-rt114-rc1 stable review patch.
If anyone has any objections, please let me know.
-----------
Upstream commit d8bb65ab70f702531aaaa11d9710f9450078e295
The workqueue code has it's internal spinlock (pool::lock) and also implicit spinlock usage in the wq_manager waitqueue. These spinlocks are converted to 'sleeping' spinlocks on a RT-kernel.
Workqueue functions can be invoked from contexts which are truly atomic even on a PREEMPT_RT enabled kernel. Taking sleeping locks from such contexts is forbidden.
pool::lock can be converted to a raw spinlock as the lock held times are short. But the workqueue manager waitqueue is handled inside of pool::lock held regions which again violates the lock nesting rules of raw and regular spinlocks.
The manager waitqueue has no special requirements like custom wakeup callbacks or mass wakeups. While it does not use exclusive wait mode explicitly there is no strict requirement to queue the waiters in a particular order as there is only one waiter at a time.
This allows to replace the waitqueue with rcuwait which solves the locking problem because rcuwait relies on existing locking.
--- kernel/workqueue.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-)
kernel/workqueue.c +20 -4
@@ -50,6 +50,7 @@#include <linux/sched/isolation.h> #include <linux/nmi.h> #include <linux/kvm_para.h> +#include <linux/rcuwait.h> #include "workqueue_internal.h"
@@ -299,7 +300,8 @@ static struct workqueue_attrs *wq_update_unbound_numa_attrs_buf;static DEFINE_MUTEX (wq_pool_mutex);/* protects pools and workqueues list */ static DEFINE_MUTEX (wq_pool_attach_mutex);/* protects worker attach/detach */ static DEFINE_RAW_SPINLOCK (wq_mayday_lock);/* protects wq->maydays list */ -static DECLARE_SWAIT_QUEUE_HEAD (wq_manager_wait);/* wait for manager to go away */ +/* wait for manager to go away */ +static struct rcuwait manager_wait =__RCUWAIT_INITIALIZER (manager_wait);static LIST_HEAD (workqueues);/* PR: list of all workqueues */ static bool workqueue_freezing;/* PL: have wqs started freezing? */
@@ -2023,7 +2025,7 @@ static bool manage_workers(struct worker *worker) pool->manager =NULL ; pool->flags &= ~POOL_MANAGER_ACTIVE ; -swake_up_one (&wq_manager_wait); +rcuwait_wake_up (&manager_wait);return true; }
@@ -3344,6 +3346,18 @@ static void rcu_free_pool(struct rcu_head *rcu)kfree (pool); } +/* This returns with the lock held on success (pool manager is inactive). */ +static bool wq_manager_inactive (struct worker_pool *pool) +{ +raw_spin_lock_irq (&pool->lock); + +if (pool->flags &POOL_MANAGER_ACTIVE ) { +raw_spin_unlock_irq (&pool->lock); +return false; + } +return true; +} +/** * put_unbound_pool - put a worker_pool * @pool: worker_pool to put
@@ -3379,10 +3393,12 @@ static void put_unbound_pool(struct worker_pool *pool)* Become the manager and destroy all workers. This prevents * @pool's workers from blocking on attach_mutex. We're the last * manager and @pool gets freed with the flag set. +* Because of how wq_manager_inactive() works, we will hold the +* spinlock after a successful wait. */ raw_spin_lock_irq (&pool->lock); -swait_event_lock_irq (wq_manager_wait, - !(pool->flags &POOL_MANAGER_ACTIVE ), pool->lock); +rcuwait_wait_event (&manager_wait,wq_manager_inactive (pool), +TASK_UNINTERRUPTIBLE ); pool->flags |=POOL_MANAGER_ACTIVE ;while ((worker =first_idle_worker (pool)))
Signature · 1 line
2.38.0
PATCH 9/9 Linux 4.19.255-rt114-rc1
In reply to Daniel Wagnerv4.19.255-rt114-rc1 stable review patch.
If anyone has any objections, please let me know.
-----------
--- localversion-rt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
localversion-rt +1 -1
@@ -1 +1 @@ --rt113 +-rt114-rc1
Signature · 1 line
2.38.0