posix-timers: Prepare for PREEMPT_RT - part 1
cover posix-timers: Prepare for PREEMPT_RT - part 1
The following series prepares posix-timers for RT. The main change here is to utilize the hrtimer synchronization mechanism to prevent priority inversion and live locks on timer deletion.
This does not cover the posix CPU timers as they need more special treatment for RT which is covered in a separate series.
Applies on top of:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/core
Thanks,
tglx
8<---------------
fs/timerfd.c | 6 +++- include/linux/posix-timers.h | 5 ++- kernel/time/alarmtimer.c | 2 - kernel/time/itimer.c | 1 kernel/time/posix-timers.c | 61 +++++++++++++++++++++++++++++++++++-------- 5 files changed, 60 insertions(+), 15 deletions(-)
PATCH 1/7 alarmtimer: Prepare for PREEMPT_RT
Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.
[ tglx: Split out of combo patch ]
--- kernel/time/alarmtimer.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/alarmtimer.c +1 -1
@@ -432,7 +432,7 @@ int alarm_cancel(struct alarm *alarm)int ret =alarm_try_to_cancel (alarm);if (ret >=0 )return ret; -cpu_relax (); +hrtimer_cancel_wait_running (&alarm->timer); } }EXPORT_SYMBOL_GPL (alarm_cancel);
tip-bot for Anna-Maria Gleixner
merge bot[tip:timers/core] alarmtimer: Prepare for PREEMPT_RT
alarmtimer: Prepare for PREEMPT_RT
Commit-ID: 1f8e8bd8b74c8089a43bc5f1f24e4bf0f855d760 Gitweb: https://git.kernel.org/tip/1f8e8bd8b74c8089a43bc5f1f24e4bf0f855d760 Author: Anna-Maria Gleixner <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:49 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 17:46:41 +0200
alarmtimer: Prepare for PREEMPT_RT
Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.
[ tglx: Split out of combo patch ]
--- kernel/time/alarmtimer.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/alarmtimer.c +1 -1
@@ -432,7 +432,7 @@ int alarm_cancel(struct alarm *alarm)int ret =alarm_try_to_cancel (alarm);if (ret >=0 )return ret; -cpu_relax (); +hrtimer_cancel_wait_running (&alarm->timer); } }EXPORT_SYMBOL_GPL (alarm_cancel);
tip-bot for Anna-Maria Gleixner
merge bot[tip:timers/core] alarmtimer: Prepare for PREEMPT_RT
In reply to Thomas Gleixneralarmtimer: Prepare for PREEMPT_RT
Commit-ID: 51ae33092bb8320497ec75ddc5ab383d8fafd55c Gitweb: https://git.kernel.org/tip/51ae33092bb8320497ec75ddc5ab383d8fafd55c Author: Anna-Maria Gleixner <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:49 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 20:51:23 +0200
alarmtimer: Prepare for PREEMPT_RT
Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.
[ tglx: Split out of combo patch ]
--- kernel/time/alarmtimer.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/alarmtimer.c +1 -1
@@ -432,7 +432,7 @@ int alarm_cancel(struct alarm *alarm)int ret =alarm_try_to_cancel (alarm);if (ret >=0 )return ret; -cpu_relax (); +hrtimer_cancel_wait_running (&alarm->timer); } }EXPORT_SYMBOL_GPL (alarm_cancel);
PATCH 2/7 timerfd: Prepare for PREEMPT_RT
In reply to Thomas GleixnerUse the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.
[ tglx: Split out of combo patch ]
--- fs/timerfd.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
fs/timerfd.c +5 -1
@@ -471,7 +471,11 @@ static int do_timerfd_settime(int ufd, ibreak ; }spin_unlock_irq (&ctx->wqh.lock); -cpu_relax (); + +if (isalarm (ctx)) +hrtimer_cancel_wait_running (&ctx->t.alarm.timer); +else +hrtimer_cancel_wait_running (&ctx->t.tmr); }/*
tip-bot for Anna-Maria Gleixner
merge bot[tip:timers/core] timerfd: Prepare for PREEMPT_RT
timerfd: Prepare for PREEMPT_RT
Commit-ID: 4da1306fb920a267b5ea21ee15cd771c7bc09cc6 Gitweb: https://git.kernel.org/tip/4da1306fb920a267b5ea21ee15cd771c7bc09cc6 Author: Anna-Maria Gleixner <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:50 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 17:46:41 +0200
timerfd: Prepare for PREEMPT_RT
Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.
[ tglx: Split out of combo patch ]
--- fs/timerfd.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
fs/timerfd.c +5 -1
@@ -471,7 +471,11 @@ static int do_timerfd_settime(int ufd, int flags,break ; }spin_unlock_irq (&ctx->wqh.lock); -cpu_relax (); + +if (isalarm (ctx)) +hrtimer_cancel_wait_running (&ctx->t.alarm.timer); +else +hrtimer_cancel_wait_running (&ctx->t.tmr); }/*
tip-bot for Anna-Maria Gleixner
merge bot[tip:timers/core] timerfd: Prepare for PREEMPT_RT
In reply to Thomas Gleixnertimerfd: Prepare for PREEMPT_RT
Commit-ID: a125ecc16453a4fe0ba865c7df87b9c722991fdf Gitweb: https://git.kernel.org/tip/a125ecc16453a4fe0ba865c7df87b9c722991fdf Author: Anna-Maria Gleixner <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:50 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 20:51:23 +0200
timerfd: Prepare for PREEMPT_RT
Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.
[ tglx: Split out of combo patch ]
--- fs/timerfd.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
fs/timerfd.c +5 -1
@@ -471,7 +471,11 @@ static int do_timerfd_settime(int ufd, int flags,break ; }spin_unlock_irq (&ctx->wqh.lock); -cpu_relax (); + +if (isalarm (ctx)) +hrtimer_cancel_wait_running (&ctx->t.alarm.timer); +else +hrtimer_cancel_wait_running (&ctx->t.tmr); }/*
PATCH 3/7 itimers: Prepare for PREEMPT_RT
In reply to Thomas GleixnerUse the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.
As a benefit the retry loop gains the missing cpu_relax() on !RT.
[ tglx: Split out of combo patch ]
--- kernel/time/itimer.c | 1 + 1 file changed, 1 insertion(+)
kernel/time/itimer.c +1 -0
@@ -213,6 +213,7 @@ int do_setitimer(int which, struct itime/* We are sharing ->siglock with it_real_fn() */ if (hrtimer_try_to_cancel (timer) <0 ) {spin_unlock_irq (&tsk->sighand->siglock); +hrtimer_cancel_wait_running (timer); goto again; } expires =timeval_to_ktime (value->it_value);
tip-bot for Anna-Maria Gleixner
merge bot[tip:timers/core] itimers: Prepare for PREEMPT_RT
itimers: Prepare for PREEMPT_RT
Commit-ID: cab46ec655eec1b5dbb0c17a25e19f67c539f00b Gitweb: https://git.kernel.org/tip/cab46ec655eec1b5dbb0c17a25e19f67c539f00b Author: Anna-Maria Gleixner <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:51 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 17:46:41 +0200
itimers: Prepare for PREEMPT_RT
Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.
As a benefit the retry loop gains the missing cpu_relax() on !RT.
[ tglx: Split out of combo patch ]
--- kernel/time/itimer.c | 1 + 1 file changed, 1 insertion(+)
kernel/time/itimer.c +1 -0
@@ -213,6 +213,7 @@ again:/* We are sharing ->siglock with it_real_fn() */ if (hrtimer_try_to_cancel (timer) <0 ) {spin_unlock_irq (&tsk->sighand->siglock); +hrtimer_cancel_wait_running (timer); goto again; } expires =timeval_to_ktime (value->it_value);
tip-bot for Anna-Maria Gleixner
merge bot[tip:timers/core] itimers: Prepare for PREEMPT_RT
In reply to Thomas Gleixneritimers: Prepare for PREEMPT_RT
Commit-ID: c7e6d704a0097e59667495cf52dcc4e1085e620b Gitweb: https://git.kernel.org/tip/c7e6d704a0097e59667495cf52dcc4e1085e620b Author: Anna-Maria Gleixner <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:51 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 20:51:24 +0200
itimers: Prepare for PREEMPT_RT
Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.
As a benefit the retry loop gains the missing cpu_relax() on !RT.
[ tglx: Split out of combo patch ]
--- kernel/time/itimer.c | 1 + 1 file changed, 1 insertion(+)
kernel/time/itimer.c +1 -0
@@ -213,6 +213,7 @@ again:/* We are sharing ->siglock with it_real_fn() */ if (hrtimer_try_to_cancel (timer) <0 ) {spin_unlock_irq (&tsk->sighand->siglock); +hrtimer_cancel_wait_running (timer); goto again; } expires =timeval_to_ktime (value->it_value);
PATCH 4/7 posix-timers: Cleanup the flag/flags confusion
In reply to Thomas Gleixnerdo_timer_settime() has a 'flags' argument and uses 'flag' for the interrupt flags, which is confusing at best.
Rename the argument so 'flags' can be used for interrupt flags as usual.
--- kernel/time/posix-timers.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/time/posix-timers.c +5 -5
@@ -844,13 +844,13 @@ int common_timer_set(struct k_itimer *tireturn 0 ; } -static int do_timer_settime (timer_t timer_id,int flags, +static int do_timer_settime (timer_t timer_id,int tmr_flags,struct itimerspec64 *new_spec64,struct itimerspec64 *old_spec64) {const struct k_clock *kc;struct k_itimer *timr; -unsigned long flag; +unsigned long flags;int error =0 ; if (!timespec64_valid(&new_spec64->it_interval) ||
@@ -860,7 +860,7 @@ static int do_timer_settime(timer_t timeif (old_spec64)memset (old_spec64,0 ,sizeof (*old_spec64)); retry: - timr =lock_timer (timer_id, &flag); + timr =lock_timer (timer_id, &flags);if (!timr)return -EINVAL ;
@@ -868,9 +868,9 @@ static int do_timer_settime(timer_t timeif (WARN_ON_ONCE (!kc || !kc->timer_set)) error = -EINVAL ;else - error = kc->timer_set (timr, flags, new_spec64, old_spec64); + error = kc->timer_set (timr, tmr_flags, new_spec64, old_spec64); -unlock_timer (timr, flag); +unlock_timer (timr, flags);if (error ==TIMER_RETRY ) { old_spec64 =NULL ;// We already got the old time... goto retry;
tip-bot for Anna-Maria Gleixner
merge bot[tip:timers/core] posix-timers: Cleanup the flag/flags confusion
posix-timers: Cleanup the flag/flags confusion
Commit-ID: b0ccc6eb0d7e0b7d346b118ccc8b38bf18e39b7f Gitweb: https://git.kernel.org/tip/b0ccc6eb0d7e0b7d346b118ccc8b38bf18e39b7f Author: Anna-Maria Gleixner <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:52 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 17:46:42 +0200
posix-timers: Cleanup the flag/flags confusion
do_timer_settime() has a 'flags' argument and uses 'flag' for the interrupt flags, which is confusing at best.
Rename the argument so 'flags' can be used for interrupt flags as usual.
--- kernel/time/posix-timers.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/time/posix-timers.c +5 -5
@@ -844,13 +844,13 @@ int common_timer_set(struct k_itimer *timr, int flags,return 0 ; } -static int do_timer_settime (timer_t timer_id,int flags, +static int do_timer_settime (timer_t timer_id,int tmr_flags,struct itimerspec64 *new_spec64,struct itimerspec64 *old_spec64) {const struct k_clock *kc;struct k_itimer *timr; -unsigned long flag; +unsigned long flags;int error =0 ; if (!timespec64_valid(&new_spec64->it_interval) ||
@@ -860,7 +860,7 @@ static int do_timer_settime(timer_t timer_id, int flags,if (old_spec64)memset (old_spec64,0 ,sizeof (*old_spec64)); retry: - timr =lock_timer (timer_id, &flag); + timr =lock_timer (timer_id, &flags);if (!timr)return -EINVAL ;
@@ -868,9 +868,9 @@ retry:if (WARN_ON_ONCE (!kc || !kc->timer_set)) error = -EINVAL ;else - error = kc->timer_set (timr, flags, new_spec64, old_spec64); + error = kc->timer_set (timr, tmr_flags, new_spec64, old_spec64); -unlock_timer (timr, flag); +unlock_timer (timr, flags);if (error ==TIMER_RETRY ) { old_spec64 =NULL ;// We already got the old time... goto retry;
tip-bot for Thomas Gleixner
merge bot[tip:timers/core] posix-timers: Cleanup the flag/flags confusion
In reply to Thomas Gleixnerposix-timers: Cleanup the flag/flags confusion
Commit-ID: 21670ee44f1e3565030bcabc62178b8e5eb2fce7 Gitweb: https://git.kernel.org/tip/21670ee44f1e3565030bcabc62178b8e5eb2fce7 Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:52 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 20:51:24 +0200
posix-timers: Cleanup the flag/flags confusion
do_timer_settime() has a 'flags' argument and uses 'flag' for the interrupt flags, which is confusing at best.
Rename the argument so 'flags' can be used for interrupt flags as usual.
--- kernel/time/posix-timers.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/time/posix-timers.c +5 -5
@@ -844,13 +844,13 @@ int common_timer_set(struct k_itimer *timr, int flags,return 0 ; } -static int do_timer_settime (timer_t timer_id,int flags, +static int do_timer_settime (timer_t timer_id,int tmr_flags,struct itimerspec64 *new_spec64,struct itimerspec64 *old_spec64) {const struct k_clock *kc;struct k_itimer *timr; -unsigned long flag; +unsigned long flags;int error =0 ; if (!timespec64_valid(&new_spec64->it_interval) ||
@@ -860,7 +860,7 @@ static int do_timer_settime(timer_t timer_id, int flags,if (old_spec64)memset (old_spec64,0 ,sizeof (*old_spec64)); retry: - timr =lock_timer (timer_id, &flag); + timr =lock_timer (timer_id, &flags);if (!timr)return -EINVAL ;
@@ -868,9 +868,9 @@ retry:if (WARN_ON_ONCE (!kc || !kc->timer_set)) error = -EINVAL ;else - error = kc->timer_set (timr, flags, new_spec64, old_spec64); + error = kc->timer_set (timr, tmr_flags, new_spec64, old_spec64); -unlock_timer (timr, flag); +unlock_timer (timr, flags);if (error ==TIMER_RETRY ) { old_spec64 =NULL ;// We already got the old time... goto retry;
PATCH 5/7 posix-timers: Rework cancel retry loops
In reply to Thomas GleixnerAs a preparatory step for adding the PREEMPT RT specific synchronization mechanism to wait for a running timer callback, rework the timer cancel retry loops so they call a common function. This allows trivial substitution in one place.
--- kernel/time/posix-timers.c | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-)
kernel/time/posix-timers.c +23 -6
@@ -805,6 +805,17 @@ static int common_hrtimer_try_to_cancel(return hrtimer_try_to_cancel (&timr->it.real.timer); } +static struct k_itimer *timer_wait_running (struct k_itimer *timer, +unsigned long *flags) +{ +timer_t timer_id =READ_ONCE (timer->it_id); + +unlock_timer (timer, *flags); +cpu_relax (); +/* Relock the timer. It might be not longer hashed. */ +return lock_timer (timer_id, flags); +} +/* Set a POSIX.1b interval timer. */ int common_timer_set(struct k_itimer *timr, int flags, struct itimerspec64 *new_setting,
@@ -859,8 +870,9 @@ static int do_timer_settime(timer_t timeif (old_spec64)memset (old_spec64,0 ,sizeof (*old_spec64)); -retry: + timr =lock_timer (timer_id, &flags); +retry:if (!timr)return -EINVAL ;
@@ -870,11 +882,14 @@ static int do_timer_settime(timer_t timeelse error = kc->timer_set (timr, tmr_flags, new_spec64, old_spec64); -unlock_timer (timr, flags);if (error ==TIMER_RETRY ) { - old_spec64 =NULL ;// We already got the old time... +// We already got the old time... + old_spec64 =NULL ; +/* Unlocks and relocks the timer if it still exists */ + timr =timer_wait_running (timr, &flags); goto retry; } +unlock_timer (timr, flags);return error; }
@@ -951,13 +966,15 @@ SYSCALL_DEFINE1(timer_delete, timer_t, tstruct k_itimer *timer;unsigned long flags; -retry_delete: timer =lock_timer (timer_id, &flags); + +retry_delete:if (!timer)return -EINVAL ; -if (timer_delete_hook (timer) ==TIMER_RETRY ) { -unlock_timer (timer, flags); +if (unlikely (timer_delete_hook (timer) ==TIMER_RETRY )) { +/* Unlocks and relocks the timer if it still exists */ + timer =timer_wait_running (timer, &flags); goto retry_delete; }
tip-bot for Thomas Gleixner
merge bot[tip:timers/core] posix-timers: Rework cancel retry loops
posix-timers: Rework cancel retry loops
Commit-ID: f8d1b0549263354b8d8854fefc521ac536be70ff Gitweb: https://git.kernel.org/tip/f8d1b0549263354b8d8854fefc521ac536be70ff Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:53 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 17:46:42 +0200
posix-timers: Rework cancel retry loops
As a preparatory step for adding the PREEMPT RT specific synchronization mechanism to wait for a running timer callback, rework the timer cancel retry loops so they call a common function. This allows trivial substitution in one place.
--- kernel/time/posix-timers.c | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-)
kernel/time/posix-timers.c +23 -6
@@ -805,6 +805,17 @@ static int common_hrtimer_try_to_cancel(struct k_itimer *timr)return hrtimer_try_to_cancel (&timr->it.real.timer); } +static struct k_itimer *timer_wait_running (struct k_itimer *timer, +unsigned long *flags) +{ +timer_t timer_id =READ_ONCE (timer->it_id); + +unlock_timer (timer, *flags); +cpu_relax (); +/* Relock the timer. It might be not longer hashed. */ +return lock_timer (timer_id, flags); +} +/* Set a POSIX.1b interval timer. */ int common_timer_set(struct k_itimer *timr, int flags, struct itimerspec64 *new_setting,
@@ -859,8 +870,9 @@ static int do_timer_settime(timer_t timer_id, int tmr_flags,if (old_spec64)memset (old_spec64,0 ,sizeof (*old_spec64)); -retry: + timr =lock_timer (timer_id, &flags); +retry:if (!timr)return -EINVAL ;
@@ -870,11 +882,14 @@ retry:else error = kc->timer_set (timr, tmr_flags, new_spec64, old_spec64); -unlock_timer (timr, flags);if (error ==TIMER_RETRY ) { - old_spec64 =NULL ;// We already got the old time... +// We already got the old time... + old_spec64 =NULL ; +/* Unlocks and relocks the timer if it still exists */ + timr =timer_wait_running (timr, &flags); goto retry; } +unlock_timer (timr, flags);return error; }
@@ -951,13 +966,15 @@ SYSCALL_DEFINE1(timer_delete, timer_t, timer_id)struct k_itimer *timer;unsigned long flags; -retry_delete: timer =lock_timer (timer_id, &flags); + +retry_delete:if (!timer)return -EINVAL ; -if (timer_delete_hook (timer) ==TIMER_RETRY ) { -unlock_timer (timer, flags); +if (unlikely (timer_delete_hook (timer) ==TIMER_RETRY )) { +/* Unlocks and relocks the timer if it still exists */ + timer =timer_wait_running (timer, &flags); goto retry_delete; }
tip-bot for Thomas Gleixner
merge bot[tip:timers/core] posix-timers: Rework cancel retry loops
In reply to Thomas Gleixnerposix-timers: Rework cancel retry loops
Commit-ID: 6945e5c2abe008302b20266248d6de95575311a8 Gitweb: https://git.kernel.org/tip/6945e5c2abe008302b20266248d6de95575311a8 Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:53 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 20:51:24 +0200
posix-timers: Rework cancel retry loops
As a preparatory step for adding the PREEMPT RT specific synchronization mechanism to wait for a running timer callback, rework the timer cancel retry loops so they call a common function. This allows trivial substitution in one place.
--- kernel/time/posix-timers.c | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-)
kernel/time/posix-timers.c +23 -6
@@ -805,6 +805,17 @@ static int common_hrtimer_try_to_cancel(struct k_itimer *timr)return hrtimer_try_to_cancel (&timr->it.real.timer); } +static struct k_itimer *timer_wait_running (struct k_itimer *timer, +unsigned long *flags) +{ +timer_t timer_id =READ_ONCE (timer->it_id); + +unlock_timer (timer, *flags); +cpu_relax (); +/* Relock the timer. It might be not longer hashed. */ +return lock_timer (timer_id, flags); +} +/* Set a POSIX.1b interval timer. */ int common_timer_set(struct k_itimer *timr, int flags, struct itimerspec64 *new_setting,
@@ -859,8 +870,9 @@ static int do_timer_settime(timer_t timer_id, int tmr_flags,if (old_spec64)memset (old_spec64,0 ,sizeof (*old_spec64)); -retry: + timr =lock_timer (timer_id, &flags); +retry:if (!timr)return -EINVAL ;
@@ -870,11 +882,14 @@ retry:else error = kc->timer_set (timr, tmr_flags, new_spec64, old_spec64); -unlock_timer (timr, flags);if (error ==TIMER_RETRY ) { - old_spec64 =NULL ;// We already got the old time... +// We already got the old time... + old_spec64 =NULL ; +/* Unlocks and relocks the timer if it still exists */ + timr =timer_wait_running (timr, &flags); goto retry; } +unlock_timer (timr, flags);return error; }
@@ -951,13 +966,15 @@ SYSCALL_DEFINE1(timer_delete, timer_t, timer_id)struct k_itimer *timer;unsigned long flags; -retry_delete: timer =lock_timer (timer_id, &flags); + +retry_delete:if (!timer)return -EINVAL ; -if (timer_delete_hook (timer) ==TIMER_RETRY ) { -unlock_timer (timer, flags); +if (unlikely (timer_delete_hook (timer) ==TIMER_RETRY )) { +/* Unlocks and relocks the timer if it still exists */ + timer =timer_wait_running (timer, &flags); goto retry_delete; }
PATCH 6/7 posix-timers: Move rcu_head out of it union
In reply to Thomas GleixnerTimer deletion on PREEMPT_RT is prone to priority inversion and live locks. The hrtimer code has a synchronization mechanism for this. Posix CPU timers will grow one.
But that mechanism cannot be invoked while holding the k_itimer lock because that can deadlock against the running timer callback. So the lock must be dropped which allows the timer to be freed.
The timer free can be prevented by taking RCU readlock before dropping the lock, but because the rcu_head is part of the 'it' union a concurrent free will overwrite the hrtimer on which the task is trying to synchronize.
Move the rcu_head out of the union to prevent this.
[ tglx: Fixed up kernel-doc. Rewrote changelog ]
--- include/linux/posix-timers.h | 5 +++-- kernel/time/posix-timers.c | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-)
include/linux/posix-timers.h +3 -2
@@ -85,7 +85,8 @@ static inline int clockid_to_fd(const cl* @it_process: The task to wakeup on clock_nanosleep (CPU timers) * @sigq: Pointer to preallocated sigqueue * @it: Union representing the various posix timer type -* internals. Also used for rcu freeing the timer. +* internals. +* @rcu: RCU head for freeing the timer. */ struct k_itimer {struct list_head list;
@@ -114,8 +115,8 @@ struct k_itimer {struct {struct alarm alarmtimer; } alarm; -struct rcu_head rcu; } it; +struct rcu_head rcu; };void run_posix_cpu_timers (struct task_struct *task);
kernel/time/posix-timers.c +2 -2
@@ -442,7 +442,7 @@ static struct k_itimer * alloc_posix_timstatic void k_itimer_rcu_free (struct rcu_head *head) { - struct k_itimer *tmr = container_of(head, struct k_itimer, it.rcu); + struct k_itimer *tmr = container_of(head, struct k_itimer, rcu);kmem_cache_free (posix_timers_cache, tmr); }
@@ -459,7 +459,7 @@ static void release_posix_timer(struct k }put_pid (tmr->it_pid);sigqueue_free (tmr->sigq); -call_rcu (&tmr->it.rcu, k_itimer_rcu_free); +call_rcu (&tmr->rcu, k_itimer_rcu_free); }static int common_timer_create (struct k_itimer *new_timer)
tip-bot for Sebastian Andrzej Siewior
merge bot[tip:timers/core] posix-timers: Move rcu_head out of it union
posix-timers: Move rcu_head out of it union
Commit-ID: eb5d344194342c08406e0f04c224007ea7338c11 Gitweb: https://git.kernel.org/tip/eb5d344194342c08406e0f04c224007ea7338c11 Author: Sebastian Andrzej Siewior <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:54 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 17:46:42 +0200
posix-timers: Move rcu_head out of it union
Timer deletion on PREEMPT_RT is prone to priority inversion and live locks. The hrtimer code has a synchronization mechanism for this. Posix CPU timers will grow one.
But that mechanism cannot be invoked while holding the k_itimer lock because that can deadlock against the running timer callback. So the lock must be dropped which allows the timer to be freed.
The timer free can be prevented by taking RCU readlock before dropping the lock, but because the rcu_head is part of the 'it' union a concurrent free will overwrite the hrtimer on which the task is trying to synchronize.
Move the rcu_head out of the union to prevent this.
[ tglx: Fixed up kernel-doc. Rewrote changelog ]
--- include/linux/posix-timers.h | 5 +++-- kernel/time/posix-timers.c | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-)
include/linux/posix-timers.h +3 -2
@@ -85,7 +85,8 @@ static inline int clockid_to_fd(const clockid_t clk)* @it_process: The task to wakeup on clock_nanosleep (CPU timers) * @sigq: Pointer to preallocated sigqueue * @it: Union representing the various posix timer type -* internals. Also used for rcu freeing the timer. +* internals. +* @rcu: RCU head for freeing the timer. */ struct k_itimer {struct list_head list;
@@ -114,8 +115,8 @@ struct k_itimer {struct {struct alarm alarmtimer; } alarm; -struct rcu_head rcu; } it; +struct rcu_head rcu; };void run_posix_cpu_timers (struct task_struct *task);
kernel/time/posix-timers.c +2 -2
@@ -442,7 +442,7 @@ static struct k_itimer * alloc_posix_timer(void)static void k_itimer_rcu_free (struct rcu_head *head) { - struct k_itimer *tmr = container_of(head, struct k_itimer, it.rcu); + struct k_itimer *tmr = container_of(head, struct k_itimer, rcu);kmem_cache_free (posix_timers_cache, tmr); }
@@ -459,7 +459,7 @@ static void release_posix_timer(struct k_itimer *tmr, int it_id_set) }put_pid (tmr->it_pid);sigqueue_free (tmr->sigq); -call_rcu (&tmr->it.rcu, k_itimer_rcu_free); +call_rcu (&tmr->rcu, k_itimer_rcu_free); }static int common_timer_create (struct k_itimer *new_timer)
tip-bot for Sebastian Andrzej Siewior
merge bot[tip:timers/core] posix-timers: Move rcu_head out of it union
In reply to Thomas Gleixnerposix-timers: Move rcu_head out of it union
Commit-ID: 5d99b32a009e900a561f6a42ea7afe5b21288b8a Gitweb: https://git.kernel.org/tip/5d99b32a009e900a561f6a42ea7afe5b21288b8a Author: Sebastian Andrzej Siewior <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:54 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 20:51:25 +0200
posix-timers: Move rcu_head out of it union
Timer deletion on PREEMPT_RT is prone to priority inversion and live locks. The hrtimer code has a synchronization mechanism for this. Posix CPU timers will grow one.
But that mechanism cannot be invoked while holding the k_itimer lock because that can deadlock against the running timer callback. So the lock must be dropped which allows the timer to be freed.
The timer free can be prevented by taking RCU readlock before dropping the lock, but because the rcu_head is part of the 'it' union a concurrent free will overwrite the hrtimer on which the task is trying to synchronize.
Move the rcu_head out of the union to prevent this.
[ tglx: Fixed up kernel-doc. Rewrote changelog ]
--- include/linux/posix-timers.h | 5 +++-- kernel/time/posix-timers.c | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-)
include/linux/posix-timers.h +3 -2
@@ -85,7 +85,8 @@ static inline int clockid_to_fd(const clockid_t clk)* @it_process: The task to wakeup on clock_nanosleep (CPU timers) * @sigq: Pointer to preallocated sigqueue * @it: Union representing the various posix timer type -* internals. Also used for rcu freeing the timer. +* internals. +* @rcu: RCU head for freeing the timer. */ struct k_itimer {struct list_head list;
@@ -114,8 +115,8 @@ struct k_itimer {struct {struct alarm alarmtimer; } alarm; -struct rcu_head rcu; } it; +struct rcu_head rcu; };void run_posix_cpu_timers (struct task_struct *task);
kernel/time/posix-timers.c +2 -2
@@ -442,7 +442,7 @@ static struct k_itimer * alloc_posix_timer(void)static void k_itimer_rcu_free (struct rcu_head *head) { - struct k_itimer *tmr = container_of(head, struct k_itimer, it.rcu); + struct k_itimer *tmr = container_of(head, struct k_itimer, rcu);kmem_cache_free (posix_timers_cache, tmr); }
@@ -459,7 +459,7 @@ static void release_posix_timer(struct k_itimer *tmr, int it_id_set) }put_pid (tmr->it_pid);sigqueue_free (tmr->sigq); -call_rcu (&tmr->it.rcu, k_itimer_rcu_free); +call_rcu (&tmr->rcu, k_itimer_rcu_free); }static int common_timer_create (struct k_itimer *new_timer)
PATCH 7/7 posix-timers: Prepare for PREEMPT_RT
In reply to Thomas GleixnerPosix timer delete retry loops are affected by the same priority inversion and live lock issues as the other timers.
Provide a RT specific synchronization function which keeps a reference to the timer by holding rcu read lock to prevent the timer from being freed, dropping the timer lock and invoking the timer specific wait function.
This does not yet cover posix CPU timers because they need more special treatment on PREEMPT_RT.
--- kernel/time/posix-timers.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
kernel/time/posix-timers.c +22 -0
@@ -805,6 +805,27 @@ static int common_hrtimer_try_to_cancel(return hrtimer_try_to_cancel (&timr->it.real.timer); } +#ifdef CONFIG_PREEMPT_RT +static struct k_itimer *timer_wait_running (struct k_itimer *timer, +unsigned long *flags) +{ +const struct k_clock *kc =READ_ONCE (timer->kclock); +timer_t timer_id =READ_ONCE (timer->it_id); + +/* Prevent kfree(timer) after dropping the lock */ +rcu_read_lock (); +unlock_timer (timer, *flags); + +if (kc->timer_arm == common_hrtimer_arm) +hrtimer_cancel_wait_running (&timer->it.real.timer); +else if (kc == &alarm_clock) +hrtimer_cancel_wait_running (&timer->it.alarm.alarmtimer.timer); +rcu_read_unlock (); + +/* Relock the timer. It might be not longer hashed. */ +return lock_timer (timer_id, flags); +} +#else static struct k_itimer *timer_wait_running (struct k_itimer *timer,unsigned long *flags) {
@@ -815,6 +836,7 @@ static struct k_itimer *timer_wait_runni/* Relock the timer. It might be not longer hashed. */ return lock_timer (timer_id, flags); } +#endif /* Set a POSIX.1b interval timer. */ int common_timer_set(struct k_itimer *timr, int flags,
Re: [patch 7/7] posix-timers: Prepare for PREEMPT_RT
Quoted 4 lines · Thomas Gleixner
kernel/time/posix-timers.c +4
-0+static struct k_itimer *timer_wait_running (struct k_itimer *timer, +unsigned long *flags) +{ +const struct k_clock *kc =READ_ONCE (timer->kclock);
kernel/time/posix-timers.c static int common_hrtimer_try_to_cancel( · line 822
Quoted 10 lines
kernel/time/posix-timers.c +10
-0+timer_t timer_id =READ_ONCE (timer->it_id); + +/* Prevent kfree(timer) after dropping the lock */ +rcu_read_lock (); +unlock_timer (timer, *flags); + +if (kc->timer_arm == common_hrtimer_arm) +hrtimer_cancel_wait_running (&timer->it.real.timer); +else if (kc == &alarm_clock) +hrtimer_cancel_wait_running (&timer->it.alarm.alarmtimer.timer);
else WARN();
Quoted 5 lines
kernel/time/posix-timers.c +5
-0+rcu_read_unlock (); + +/* Relock the timer. It might be not longer hashed. */ +return lock_timer (timer_id, flags); +}
Re: [patch 7/7] posix-timers: Prepare for PREEMPT_RT
Quoted 17 lines · Peter Zijlstra
Quoted 4 lines · Thomas Gleixner
kernel/time/posix-timers.c +4
-0+static struct k_itimer *timer_wait_running (struct k_itimer *timer, +unsigned long *flags) +{ +const struct k_clock *kc =READ_ONCE (timer->kclock);kernel/time/posix-timers.c static int common_hrtimer_try_to_cancel( · line 822
Quoted 10 lines
kernel/time/posix-timers.c +10
-0+timer_t timer_id =READ_ONCE (timer->it_id); + +/* Prevent kfree(timer) after dropping the lock */ +rcu_read_lock (); +unlock_timer (timer, *flags); + +if (kc->timer_arm == common_hrtimer_arm) +hrtimer_cancel_wait_running (&timer->it.real.timer); +else if (kc == &alarm_clock) +hrtimer_cancel_wait_running (&timer->it.alarm.alarmtimer.timer);else WARN();
Yup. Working on it ...
tip-bot for Thomas Gleixner
merge bot[tip:timers/core] posix-timers: Prepare for PREEMPT_RT
In reply to Thomas Gleixnerposix-timers: Prepare for PREEMPT_RT
Commit-ID: 3a839db3eaeeef31520de45f3b078204d068e3d0 Gitweb: https://git.kernel.org/tip/3a839db3eaeeef31520de45f3b078204d068e3d0 Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:55 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 17:46:43 +0200
posix-timers: Prepare for PREEMPT_RT
Posix timer delete retry loops are affected by the same priority inversion and live lock issues as the other timers.
Provide a RT specific synchronization function which keeps a reference to the timer by holding rcu read lock to prevent the timer from being freed, dropping the timer lock and invoking the timer specific wait function.
This does not yet cover posix CPU timers because they need more special treatment on PREEMPT_RT.
--- kernel/time/posix-timers.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
kernel/time/posix-timers.c +24 -0
@@ -805,6 +805,29 @@ static int common_hrtimer_try_to_cancel(struct k_itimer *timr)return hrtimer_try_to_cancel (&timr->it.real.timer); } +#ifdef CONFIG_PREEMPT_RT +static struct k_itimer *timer_wait_running (struct k_itimer *timer, +unsigned long *flags) +{ +const struct k_clock *kc =READ_ONCE (timer->kclock); +timer_t timer_id =READ_ONCE (timer->it_id); + +/* Prevent kfree(timer) after dropping the lock */ +rcu_read_lock (); +unlock_timer (timer, *flags); + +if (kc->timer_arm == common_hrtimer_arm) +hrtimer_cancel_wait_running (&timer->it.real.timer); +else if (kc == &alarm_clock) +hrtimer_cancel_wait_running (&timer->it.alarm.alarmtimer.timer); +else +WARN_ON_ONCE (1 ); +rcu_read_unlock (); + +/* Relock the timer. It might be not longer hashed. */ +return lock_timer (timer_id, flags); +} +#else static struct k_itimer *timer_wait_running (struct k_itimer *timer,unsigned long *flags) {
@@ -815,6 +838,7 @@ static struct k_itimer *timer_wait_running(struct k_itimer *timer,/* Relock the timer. It might be not longer hashed. */ return lock_timer (timer_id, flags); } +#endif /* Set a POSIX.1b interval timer. */ int common_timer_set(struct k_itimer *timr, int flags,
tip-bot for Thomas Gleixner
merge bot[tip:timers/core] posix-timers: Prepare for PREEMPT_RT
In reply to Thomas Gleixnerposix-timers: Prepare for PREEMPT_RT
Commit-ID: 08a3c192c93f4359a94bf47971e55b0324b72b8b Gitweb: https://git.kernel.org/tip/08a3c192c93f4359a94bf47971e55b0324b72b8b Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 31 Jul 2019 00:33:55 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 1 Aug 2019 20:51:25 +0200
posix-timers: Prepare for PREEMPT_RT
Posix timer delete retry loops are affected by the same priority inversion and live lock issues as the other timers.
Provide a RT specific synchronization function which keeps a reference to the timer by holding rcu read lock to prevent the timer from being freed, dropping the timer lock and invoking the timer specific wait function.
This does not yet cover posix CPU timers because they need more special treatment on PREEMPT_RT.
--- kernel/time/posix-timers.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
kernel/time/posix-timers.c +24 -0
@@ -805,6 +805,29 @@ static int common_hrtimer_try_to_cancel(struct k_itimer *timr)return hrtimer_try_to_cancel (&timr->it.real.timer); } +#ifdef CONFIG_PREEMPT_RT +static struct k_itimer *timer_wait_running (struct k_itimer *timer, +unsigned long *flags) +{ +const struct k_clock *kc =READ_ONCE (timer->kclock); +timer_t timer_id =READ_ONCE (timer->it_id); + +/* Prevent kfree(timer) after dropping the lock */ +rcu_read_lock (); +unlock_timer (timer, *flags); + +if (kc->timer_arm == common_hrtimer_arm) +hrtimer_cancel_wait_running (&timer->it.real.timer); +else if (kc == &alarm_clock) +hrtimer_cancel_wait_running (&timer->it.alarm.alarmtimer.timer); +else +WARN_ON_ONCE (1 ); +rcu_read_unlock (); + +/* Relock the timer. It might be not longer hashed. */ +return lock_timer (timer_id, flags); +} +#else static struct k_itimer *timer_wait_running (struct k_itimer *timer,unsigned long *flags) {
@@ -815,6 +838,7 @@ static struct k_itimer *timer_wait_running(struct k_itimer *timer,/* Relock the timer. It might be not longer hashed. */ return lock_timer (timer_id, flags); } +#endif /* Set a POSIX.1b interval timer. */ int common_timer_set(struct k_itimer *timr, int flags,
Quoted 7 lines · Thomas Gleixner
The following series prepares posix-timers for RT. The main change here is to utilize the hrtimer synchronization mechanism to prevent priority inversion and live locks on timer deletion.
This does not cover the posix CPU timers as they need more special treatment for RT which is covered in a separate series.