Skip to contentSkip to conversation

posix-timers: Prepare for PREEMPT_RT - part 1

PATCH In review Thomas Gleixner 3 participants 25 messages Jul 30, 2019 – Aug 1, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover posix-timers: Prepare for PREEMPT_RT - part 1

The following series prepares posix-timers for RT. The main change here is to utilize the hrtimer synchronization mechanism to prevent priority inversion and live locks on timer deletion.

This does not cover the posix CPU timers as they need more special treatment for RT which is covered in a separate series.

Applies on top of:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/core

Thanks,

tglx

8<---------------

 fs/timerfd.c                 |    6 +++-
 include/linux/posix-timers.h |    5 ++-
 kernel/time/alarmtimer.c     |    2 -
 kernel/time/itimer.c         |    1 
 kernel/time/posix-timers.c   |   61 +++++++++++++++++++++++++++++++++++--------
 5 files changed, 60 insertions(+), 15 deletions(-)
#1 lore raw

PATCH 1/7 alarmtimer: Prepare for PREEMPT_RT

Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.

[ tglx: Split out of combo patch ]

---
 kernel/time/alarmtimer.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/alarmtimer.c +1 -1
@@ -432,7 +432,7 @@ int alarm_cancel(struct alarm *alarm)
 		int ret = alarm_try_to_cancel(alarm);
 		if (ret >= 0)
 			return ret;
-		cpu_relax();
+		hrtimer_cancel_wait_running(&alarm->timer);
 	}
 }
 EXPORT_SYMBOL_GPL(alarm_cancel);
#2 lore raw

[tip:timers/core] alarmtimer: Prepare for PREEMPT_RT

alarmtimer: Prepare for PREEMPT_RT
Commit-ID:  1f8e8bd8b74c8089a43bc5f1f24e4bf0f855d760
Gitweb:     https://git.kernel.org/tip/1f8e8bd8b74c8089a43bc5f1f24e4bf0f855d760
Author:     Anna-Maria Gleixner <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:49 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 17:46:41 +0200

alarmtimer: Prepare for PREEMPT_RT

Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.

[ tglx: Split out of combo patch ]

---
 kernel/time/alarmtimer.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/alarmtimer.c +1 -1
index 57518efc3810..36947449dba2 100644
@@ -432,7 +432,7 @@ int alarm_cancel(struct alarm *alarm)
 		int ret = alarm_try_to_cancel(alarm);
 		if (ret >= 0)
 			return ret;
-		cpu_relax();
+		hrtimer_cancel_wait_running(&alarm->timer);
 	}
 }
 EXPORT_SYMBOL_GPL(alarm_cancel);
#3 lore raw

[tip:timers/core] alarmtimer: Prepare for PREEMPT_RT

In reply to Thomas Gleixner
alarmtimer: Prepare for PREEMPT_RT
Commit-ID:  51ae33092bb8320497ec75ddc5ab383d8fafd55c
Gitweb:     https://git.kernel.org/tip/51ae33092bb8320497ec75ddc5ab383d8fafd55c
Author:     Anna-Maria Gleixner <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:49 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 20:51:23 +0200

alarmtimer: Prepare for PREEMPT_RT

Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.

[ tglx: Split out of combo patch ]

---
 kernel/time/alarmtimer.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/alarmtimer.c +1 -1
index 57518efc3810..36947449dba2 100644
@@ -432,7 +432,7 @@ int alarm_cancel(struct alarm *alarm)
 		int ret = alarm_try_to_cancel(alarm);
 		if (ret >= 0)
 			return ret;
-		cpu_relax();
+		hrtimer_cancel_wait_running(&alarm->timer);
 	}
 }
 EXPORT_SYMBOL_GPL(alarm_cancel);
#4 lore raw

PATCH 2/7 timerfd: Prepare for PREEMPT_RT

In reply to Thomas Gleixner

Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.

[ tglx: Split out of combo patch ]

---
 fs/timerfd.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
fs/timerfd.c +5 -1
@@ -471,7 +471,11 @@ static int do_timerfd_settime(int ufd, i
 				break;
 		}
 		spin_unlock_irq(&ctx->wqh.lock);
-		cpu_relax();
+
+		if (isalarm(ctx))
+			hrtimer_cancel_wait_running(&ctx->t.alarm.timer);
+		else
+			hrtimer_cancel_wait_running(&ctx->t.tmr);
 	}
 
 	/*
#5 lore raw

[tip:timers/core] timerfd: Prepare for PREEMPT_RT

timerfd: Prepare for PREEMPT_RT
Commit-ID:  4da1306fb920a267b5ea21ee15cd771c7bc09cc6
Gitweb:     https://git.kernel.org/tip/4da1306fb920a267b5ea21ee15cd771c7bc09cc6
Author:     Anna-Maria Gleixner <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:50 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 17:46:41 +0200

timerfd: Prepare for PREEMPT_RT

Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.

[ tglx: Split out of combo patch ]

---
 fs/timerfd.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
fs/timerfd.c +5 -1
index 6a6fc8aa1de7..48305ba41e3c 100644
@@ -471,7 +471,11 @@ static int do_timerfd_settime(int ufd, int flags,
 				break;
 		}
 		spin_unlock_irq(&ctx->wqh.lock);
-		cpu_relax();
+
+		if (isalarm(ctx))
+			hrtimer_cancel_wait_running(&ctx->t.alarm.timer);
+		else
+			hrtimer_cancel_wait_running(&ctx->t.tmr);
 	}
 
 	/*
#6 lore raw

[tip:timers/core] timerfd: Prepare for PREEMPT_RT

In reply to Thomas Gleixner
timerfd: Prepare for PREEMPT_RT
Commit-ID:  a125ecc16453a4fe0ba865c7df87b9c722991fdf
Gitweb:     https://git.kernel.org/tip/a125ecc16453a4fe0ba865c7df87b9c722991fdf
Author:     Anna-Maria Gleixner <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:50 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 20:51:23 +0200

timerfd: Prepare for PREEMPT_RT

Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.

[ tglx: Split out of combo patch ]

---
 fs/timerfd.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
fs/timerfd.c +5 -1
index 6a6fc8aa1de7..48305ba41e3c 100644
@@ -471,7 +471,11 @@ static int do_timerfd_settime(int ufd, int flags,
 				break;
 		}
 		spin_unlock_irq(&ctx->wqh.lock);
-		cpu_relax();
+
+		if (isalarm(ctx))
+			hrtimer_cancel_wait_running(&ctx->t.alarm.timer);
+		else
+			hrtimer_cancel_wait_running(&ctx->t.tmr);
 	}
 
 	/*
#7 lore raw

PATCH 3/7 itimers: Prepare for PREEMPT_RT

In reply to Thomas Gleixner

Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.

As a benefit the retry loop gains the missing cpu_relax() on !RT.

[ tglx: Split out of combo patch ]

---
 kernel/time/itimer.c |    1 +
 1 file changed, 1 insertion(+)
kernel/time/itimer.c +1 -0
@@ -213,6 +213,7 @@ int do_setitimer(int which, struct itime
 		/* We are sharing ->siglock with it_real_fn() */
 		if (hrtimer_try_to_cancel(timer) < 0) {
 			spin_unlock_irq(&tsk->sighand->siglock);
+			hrtimer_cancel_wait_running(timer);
 			goto again;
 		}
 		expires = timeval_to_ktime(value->it_value);
#8 lore raw

[tip:timers/core] itimers: Prepare for PREEMPT_RT

itimers: Prepare for PREEMPT_RT
Commit-ID:  cab46ec655eec1b5dbb0c17a25e19f67c539f00b
Gitweb:     https://git.kernel.org/tip/cab46ec655eec1b5dbb0c17a25e19f67c539f00b
Author:     Anna-Maria Gleixner <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:51 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 17:46:41 +0200

itimers: Prepare for PREEMPT_RT

Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.

As a benefit the retry loop gains the missing cpu_relax() on !RT.

[ tglx: Split out of combo patch ]

---
 kernel/time/itimer.c | 1 +
 1 file changed, 1 insertion(+)
kernel/time/itimer.c +1 -0
index 02068b2d5862..9d26fd4ba4c0 100644
@@ -213,6 +213,7 @@ again:
 		/* We are sharing ->siglock with it_real_fn() */
 		if (hrtimer_try_to_cancel(timer) < 0) {
 			spin_unlock_irq(&tsk->sighand->siglock);
+			hrtimer_cancel_wait_running(timer);
 			goto again;
 		}
 		expires = timeval_to_ktime(value->it_value);
#9 lore raw

[tip:timers/core] itimers: Prepare for PREEMPT_RT

In reply to Thomas Gleixner
itimers: Prepare for PREEMPT_RT
Commit-ID:  c7e6d704a0097e59667495cf52dcc4e1085e620b
Gitweb:     https://git.kernel.org/tip/c7e6d704a0097e59667495cf52dcc4e1085e620b
Author:     Anna-Maria Gleixner <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:51 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 20:51:24 +0200

itimers: Prepare for PREEMPT_RT

Use the hrtimer_cancel_wait_running() synchronization mechanism to prevent priority inversion and live locks on PREEMPT_RT.

As a benefit the retry loop gains the missing cpu_relax() on !RT.

[ tglx: Split out of combo patch ]

---
 kernel/time/itimer.c | 1 +
 1 file changed, 1 insertion(+)
kernel/time/itimer.c +1 -0
index 02068b2d5862..9d26fd4ba4c0 100644
@@ -213,6 +213,7 @@ again:
 		/* We are sharing ->siglock with it_real_fn() */
 		if (hrtimer_try_to_cancel(timer) < 0) {
 			spin_unlock_irq(&tsk->sighand->siglock);
+			hrtimer_cancel_wait_running(timer);
 			goto again;
 		}
 		expires = timeval_to_ktime(value->it_value);
#10 lore raw

PATCH 4/7 posix-timers: Cleanup the flag/flags confusion

In reply to Thomas Gleixner

do_timer_settime() has a 'flags' argument and uses 'flag' for the interrupt flags, which is confusing at best.

Rename the argument so 'flags' can be used for interrupt flags as usual.

---
 kernel/time/posix-timers.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/time/posix-timers.c +5 -5
@@ -844,13 +844,13 @@ int common_timer_set(struct k_itimer *ti
 	return 0;
 }
 
-static int do_timer_settime(timer_t timer_id, int flags,
+static int do_timer_settime(timer_t timer_id, int tmr_flags,
 			    struct itimerspec64 *new_spec64,
 			    struct itimerspec64 *old_spec64)
 {
 	const struct k_clock *kc;
 	struct k_itimer *timr;
-	unsigned long flag;
+	unsigned long flags;
 	int error = 0;
 
 	if (!timespec64_valid(&new_spec64->it_interval) ||
@@ -860,7 +860,7 @@ static int do_timer_settime(timer_t time
 	if (old_spec64)
 		memset(old_spec64, 0, sizeof(*old_spec64));
 retry:
-	timr = lock_timer(timer_id, &flag);
+	timr = lock_timer(timer_id, &flags);
 	if (!timr)
 		return -EINVAL;
 
@@ -868,9 +868,9 @@ static int do_timer_settime(timer_t time
 	if (WARN_ON_ONCE(!kc || !kc->timer_set))
 		error = -EINVAL;
 	else
-		error = kc->timer_set(timr, flags, new_spec64, old_spec64);
+		error = kc->timer_set(timr, tmr_flags, new_spec64, old_spec64);
 
-	unlock_timer(timr, flag);
+	unlock_timer(timr, flags);
 	if (error == TIMER_RETRY) {
 		old_spec64 = NULL;	// We already got the old time...
 		goto retry;
#11 lore raw

[tip:timers/core] posix-timers: Cleanup the flag/flags confusion

posix-timers: Cleanup the flag/flags confusion
Commit-ID:  b0ccc6eb0d7e0b7d346b118ccc8b38bf18e39b7f
Gitweb:     https://git.kernel.org/tip/b0ccc6eb0d7e0b7d346b118ccc8b38bf18e39b7f
Author:     Anna-Maria Gleixner <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:52 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 17:46:42 +0200

posix-timers: Cleanup the flag/flags confusion

do_timer_settime() has a 'flags' argument and uses 'flag' for the interrupt flags, which is confusing at best.

Rename the argument so 'flags' can be used for interrupt flags as usual.

---
 kernel/time/posix-timers.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/time/posix-timers.c +5 -5
index d7f2d91acdac..f5aedd2f60df 100644
@@ -844,13 +844,13 @@ int common_timer_set(struct k_itimer *timr, int flags,
 	return 0;
 }
 
-static int do_timer_settime(timer_t timer_id, int flags,
+static int do_timer_settime(timer_t timer_id, int tmr_flags,
 			    struct itimerspec64 *new_spec64,
 			    struct itimerspec64 *old_spec64)
 {
 	const struct k_clock *kc;
 	struct k_itimer *timr;
-	unsigned long flag;
+	unsigned long flags;
 	int error = 0;
 
 	if (!timespec64_valid(&new_spec64->it_interval) ||
@@ -860,7 +860,7 @@ static int do_timer_settime(timer_t timer_id, int flags,
 	if (old_spec64)
 		memset(old_spec64, 0, sizeof(*old_spec64));
 retry:
-	timr = lock_timer(timer_id, &flag);
+	timr = lock_timer(timer_id, &flags);
 	if (!timr)
 		return -EINVAL;
 
@@ -868,9 +868,9 @@ retry:
 	if (WARN_ON_ONCE(!kc || !kc->timer_set))
 		error = -EINVAL;
 	else
-		error = kc->timer_set(timr, flags, new_spec64, old_spec64);
+		error = kc->timer_set(timr, tmr_flags, new_spec64, old_spec64);
 
-	unlock_timer(timr, flag);
+	unlock_timer(timr, flags);
 	if (error == TIMER_RETRY) {
 		old_spec64 = NULL;	// We already got the old time...
 		goto retry;
#12 lore raw

[tip:timers/core] posix-timers: Cleanup the flag/flags confusion

In reply to Thomas Gleixner
posix-timers: Cleanup the flag/flags confusion
Commit-ID:  21670ee44f1e3565030bcabc62178b8e5eb2fce7
Gitweb:     https://git.kernel.org/tip/21670ee44f1e3565030bcabc62178b8e5eb2fce7
Author:     Thomas Gleixner <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:52 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 20:51:24 +0200

posix-timers: Cleanup the flag/flags confusion

do_timer_settime() has a 'flags' argument and uses 'flag' for the interrupt flags, which is confusing at best.

Rename the argument so 'flags' can be used for interrupt flags as usual.

---
 kernel/time/posix-timers.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/time/posix-timers.c +5 -5
index d7f2d91acdac..f5aedd2f60df 100644
@@ -844,13 +844,13 @@ int common_timer_set(struct k_itimer *timr, int flags,
 	return 0;
 }
 
-static int do_timer_settime(timer_t timer_id, int flags,
+static int do_timer_settime(timer_t timer_id, int tmr_flags,
 			    struct itimerspec64 *new_spec64,
 			    struct itimerspec64 *old_spec64)
 {
 	const struct k_clock *kc;
 	struct k_itimer *timr;
-	unsigned long flag;
+	unsigned long flags;
 	int error = 0;
 
 	if (!timespec64_valid(&new_spec64->it_interval) ||
@@ -860,7 +860,7 @@ static int do_timer_settime(timer_t timer_id, int flags,
 	if (old_spec64)
 		memset(old_spec64, 0, sizeof(*old_spec64));
 retry:
-	timr = lock_timer(timer_id, &flag);
+	timr = lock_timer(timer_id, &flags);
 	if (!timr)
 		return -EINVAL;
 
@@ -868,9 +868,9 @@ retry:
 	if (WARN_ON_ONCE(!kc || !kc->timer_set))
 		error = -EINVAL;
 	else
-		error = kc->timer_set(timr, flags, new_spec64, old_spec64);
+		error = kc->timer_set(timr, tmr_flags, new_spec64, old_spec64);
 
-	unlock_timer(timr, flag);
+	unlock_timer(timr, flags);
 	if (error == TIMER_RETRY) {
 		old_spec64 = NULL;	// We already got the old time...
 		goto retry;
#13 lore raw

PATCH 5/7 posix-timers: Rework cancel retry loops

In reply to Thomas Gleixner

As a preparatory step for adding the PREEMPT RT specific synchronization mechanism to wait for a running timer callback, rework the timer cancel retry loops so they call a common function. This allows trivial substitution in one place.

---
 kernel/time/posix-timers.c |   29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)
kernel/time/posix-timers.c +23 -6
@@ -805,6 +805,17 @@ static int common_hrtimer_try_to_cancel(
 	return hrtimer_try_to_cancel(&timr->it.real.timer);
 }
 
+static struct k_itimer *timer_wait_running(struct k_itimer *timer,
+					   unsigned long *flags)
+{
+	timer_t timer_id = READ_ONCE(timer->it_id);
+
+	unlock_timer(timer, *flags);
+	cpu_relax();
+	/* Relock the timer. It might be not longer hashed. */
+	return lock_timer(timer_id, flags);
+}
+
 /* Set a POSIX.1b interval timer. */
 int common_timer_set(struct k_itimer *timr, int flags,
 		     struct itimerspec64 *new_setting,
@@ -859,8 +870,9 @@ static int do_timer_settime(timer_t time
 
 	if (old_spec64)
 		memset(old_spec64, 0, sizeof(*old_spec64));
-retry:
+
 	timr = lock_timer(timer_id, &flags);
+retry:
 	if (!timr)
 		return -EINVAL;
 
@@ -870,11 +882,14 @@ static int do_timer_settime(timer_t time
 	else
 		error = kc->timer_set(timr, tmr_flags, new_spec64, old_spec64);
 
-	unlock_timer(timr, flags);
 	if (error == TIMER_RETRY) {
-		old_spec64 = NULL;	// We already got the old time...
+		// We already got the old time...
+		old_spec64 = NULL;
+		/* Unlocks and relocks the timer if it still exists */
+		timr = timer_wait_running(timr, &flags);
 		goto retry;
 	}
+	unlock_timer(timr, flags);
 
 	return error;
 }
@@ -951,13 +966,15 @@ SYSCALL_DEFINE1(timer_delete, timer_t, t
 	struct k_itimer *timer;
 	unsigned long flags;
 
-retry_delete:
 	timer = lock_timer(timer_id, &flags);
+
+retry_delete:
 	if (!timer)
 		return -EINVAL;
 
-	if (timer_delete_hook(timer) == TIMER_RETRY) {
-		unlock_timer(timer, flags);
+	if (unlikely(timer_delete_hook(timer) == TIMER_RETRY)) {
+		/* Unlocks and relocks the timer if it still exists */
+		timer = timer_wait_running(timer, &flags);
 		goto retry_delete;
 	}
 
#14 lore raw

[tip:timers/core] posix-timers: Rework cancel retry loops

posix-timers: Rework cancel retry loops
Commit-ID:  f8d1b0549263354b8d8854fefc521ac536be70ff
Gitweb:     https://git.kernel.org/tip/f8d1b0549263354b8d8854fefc521ac536be70ff
Author:     Thomas Gleixner <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:53 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 17:46:42 +0200

posix-timers: Rework cancel retry loops

As a preparatory step for adding the PREEMPT RT specific synchronization mechanism to wait for a running timer callback, rework the timer cancel retry loops so they call a common function. This allows trivial substitution in one place.

---
 kernel/time/posix-timers.c | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)
kernel/time/posix-timers.c +23 -6
index f5aedd2f60df..bbe8f9686a70 100644
@@ -805,6 +805,17 @@ static int common_hrtimer_try_to_cancel(struct k_itimer *timr)
 	return hrtimer_try_to_cancel(&timr->it.real.timer);
 }
 
+static struct k_itimer *timer_wait_running(struct k_itimer *timer,
+					   unsigned long *flags)
+{
+	timer_t timer_id = READ_ONCE(timer->it_id);
+
+	unlock_timer(timer, *flags);
+	cpu_relax();
+	/* Relock the timer. It might be not longer hashed. */
+	return lock_timer(timer_id, flags);
+}
+
 /* Set a POSIX.1b interval timer. */
 int common_timer_set(struct k_itimer *timr, int flags,
 		     struct itimerspec64 *new_setting,
@@ -859,8 +870,9 @@ static int do_timer_settime(timer_t timer_id, int tmr_flags,
 
 	if (old_spec64)
 		memset(old_spec64, 0, sizeof(*old_spec64));
-retry:
+
 	timr = lock_timer(timer_id, &flags);
+retry:
 	if (!timr)
 		return -EINVAL;
 
@@ -870,11 +882,14 @@ retry:
 	else
 		error = kc->timer_set(timr, tmr_flags, new_spec64, old_spec64);
 
-	unlock_timer(timr, flags);
 	if (error == TIMER_RETRY) {
-		old_spec64 = NULL;	// We already got the old time...
+		// We already got the old time...
+		old_spec64 = NULL;
+		/* Unlocks and relocks the timer if it still exists */
+		timr = timer_wait_running(timr, &flags);
 		goto retry;
 	}
+	unlock_timer(timr, flags);
 
 	return error;
 }
@@ -951,13 +966,15 @@ SYSCALL_DEFINE1(timer_delete, timer_t, timer_id)
 	struct k_itimer *timer;
 	unsigned long flags;
 
-retry_delete:
 	timer = lock_timer(timer_id, &flags);
+
+retry_delete:
 	if (!timer)
 		return -EINVAL;
 
-	if (timer_delete_hook(timer) == TIMER_RETRY) {
-		unlock_timer(timer, flags);
+	if (unlikely(timer_delete_hook(timer) == TIMER_RETRY)) {
+		/* Unlocks and relocks the timer if it still exists */
+		timer = timer_wait_running(timer, &flags);
 		goto retry_delete;
 	}
 
#15 lore raw

[tip:timers/core] posix-timers: Rework cancel retry loops

In reply to Thomas Gleixner
posix-timers: Rework cancel retry loops
Commit-ID:  6945e5c2abe008302b20266248d6de95575311a8
Gitweb:     https://git.kernel.org/tip/6945e5c2abe008302b20266248d6de95575311a8
Author:     Thomas Gleixner <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:53 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 20:51:24 +0200

posix-timers: Rework cancel retry loops

As a preparatory step for adding the PREEMPT RT specific synchronization mechanism to wait for a running timer callback, rework the timer cancel retry loops so they call a common function. This allows trivial substitution in one place.

---
 kernel/time/posix-timers.c | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)
kernel/time/posix-timers.c +23 -6
index f5aedd2f60df..bbe8f9686a70 100644
@@ -805,6 +805,17 @@ static int common_hrtimer_try_to_cancel(struct k_itimer *timr)
 	return hrtimer_try_to_cancel(&timr->it.real.timer);
 }
 
+static struct k_itimer *timer_wait_running(struct k_itimer *timer,
+					   unsigned long *flags)
+{
+	timer_t timer_id = READ_ONCE(timer->it_id);
+
+	unlock_timer(timer, *flags);
+	cpu_relax();
+	/* Relock the timer. It might be not longer hashed. */
+	return lock_timer(timer_id, flags);
+}
+
 /* Set a POSIX.1b interval timer. */
 int common_timer_set(struct k_itimer *timr, int flags,
 		     struct itimerspec64 *new_setting,
@@ -859,8 +870,9 @@ static int do_timer_settime(timer_t timer_id, int tmr_flags,
 
 	if (old_spec64)
 		memset(old_spec64, 0, sizeof(*old_spec64));
-retry:
+
 	timr = lock_timer(timer_id, &flags);
+retry:
 	if (!timr)
 		return -EINVAL;
 
@@ -870,11 +882,14 @@ retry:
 	else
 		error = kc->timer_set(timr, tmr_flags, new_spec64, old_spec64);
 
-	unlock_timer(timr, flags);
 	if (error == TIMER_RETRY) {
-		old_spec64 = NULL;	// We already got the old time...
+		// We already got the old time...
+		old_spec64 = NULL;
+		/* Unlocks and relocks the timer if it still exists */
+		timr = timer_wait_running(timr, &flags);
 		goto retry;
 	}
+	unlock_timer(timr, flags);
 
 	return error;
 }
@@ -951,13 +966,15 @@ SYSCALL_DEFINE1(timer_delete, timer_t, timer_id)
 	struct k_itimer *timer;
 	unsigned long flags;
 
-retry_delete:
 	timer = lock_timer(timer_id, &flags);
+
+retry_delete:
 	if (!timer)
 		return -EINVAL;
 
-	if (timer_delete_hook(timer) == TIMER_RETRY) {
-		unlock_timer(timer, flags);
+	if (unlikely(timer_delete_hook(timer) == TIMER_RETRY)) {
+		/* Unlocks and relocks the timer if it still exists */
+		timer = timer_wait_running(timer, &flags);
 		goto retry_delete;
 	}
 
#16 lore raw

PATCH 6/7 posix-timers: Move rcu_head out of it union

In reply to Thomas Gleixner

Timer deletion on PREEMPT_RT is prone to priority inversion and live locks. The hrtimer code has a synchronization mechanism for this. Posix CPU timers will grow one.

But that mechanism cannot be invoked while holding the k_itimer lock because that can deadlock against the running timer callback. So the lock must be dropped which allows the timer to be freed.

The timer free can be prevented by taking RCU readlock before dropping the lock, but because the rcu_head is part of the 'it' union a concurrent free will overwrite the hrtimer on which the task is trying to synchronize.

Move the rcu_head out of the union to prevent this.

[ tglx: Fixed up kernel-doc. Rewrote changelog ]

---
 include/linux/posix-timers.h |    5 +++--
 kernel/time/posix-timers.c   |    4 ++--
 2 files changed, 5 insertions(+), 4 deletions(-)
include/linux/posix-timers.h +3 -2
@@ -85,7 +85,8 @@ static inline int clockid_to_fd(const cl
  * @it_process:		The task to wakeup on clock_nanosleep (CPU timers)
  * @sigq:		Pointer to preallocated sigqueue
  * @it:			Union representing the various posix timer type
- *			internals. Also used for rcu freeing the timer.
+ *			internals.
+ * @rcu:		RCU head for freeing the timer.
  */
 struct k_itimer {
 	struct list_head	list;
@@ -114,8 +115,8 @@ struct k_itimer {
 		struct {
 			struct alarm	alarmtimer;
 		} alarm;
-		struct rcu_head		rcu;
 	} it;
+	struct rcu_head		rcu;
 };
 
 void run_posix_cpu_timers(struct task_struct *task);
kernel/time/posix-timers.c +2 -2
@@ -442,7 +442,7 @@ static struct k_itimer * alloc_posix_tim
 
 static void k_itimer_rcu_free(struct rcu_head *head)
 {
-	struct k_itimer *tmr = container_of(head, struct k_itimer, it.rcu);
+	struct k_itimer *tmr = container_of(head, struct k_itimer, rcu);
 
 	kmem_cache_free(posix_timers_cache, tmr);
 }
@@ -459,7 +459,7 @@ static void release_posix_timer(struct k
 	}
 	put_pid(tmr->it_pid);
 	sigqueue_free(tmr->sigq);
-	call_rcu(&tmr->it.rcu, k_itimer_rcu_free);
+	call_rcu(&tmr->rcu, k_itimer_rcu_free);
 }
 
 static int common_timer_create(struct k_itimer *new_timer)
#17 lore raw

[tip:timers/core] posix-timers: Move rcu_head out of it union

posix-timers: Move rcu_head out of it union
Commit-ID:  eb5d344194342c08406e0f04c224007ea7338c11
Gitweb:     https://git.kernel.org/tip/eb5d344194342c08406e0f04c224007ea7338c11
Author:     Sebastian Andrzej Siewior <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:54 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 17:46:42 +0200

posix-timers: Move rcu_head out of it union

Timer deletion on PREEMPT_RT is prone to priority inversion and live locks. The hrtimer code has a synchronization mechanism for this. Posix CPU timers will grow one.

But that mechanism cannot be invoked while holding the k_itimer lock because that can deadlock against the running timer callback. So the lock must be dropped which allows the timer to be freed.

The timer free can be prevented by taking RCU readlock before dropping the lock, but because the rcu_head is part of the 'it' union a concurrent free will overwrite the hrtimer on which the task is trying to synchronize.

Move the rcu_head out of the union to prevent this.

[ tglx: Fixed up kernel-doc. Rewrote changelog ]

---
 include/linux/posix-timers.h | 5 +++--
 kernel/time/posix-timers.c   | 4 ++--
 2 files changed, 5 insertions(+), 4 deletions(-)
include/linux/posix-timers.h +3 -2
index b20798fc5191..604cec0e41ba 100644
@@ -85,7 +85,8 @@ static inline int clockid_to_fd(const clockid_t clk)
  * @it_process:		The task to wakeup on clock_nanosleep (CPU timers)
  * @sigq:		Pointer to preallocated sigqueue
  * @it:			Union representing the various posix timer type
- *			internals. Also used for rcu freeing the timer.
+ *			internals.
+ * @rcu:		RCU head for freeing the timer.
  */
 struct k_itimer {
 	struct list_head	list;
@@ -114,8 +115,8 @@ struct k_itimer {
 		struct {
 			struct alarm	alarmtimer;
 		} alarm;
-		struct rcu_head		rcu;
 	} it;
+	struct rcu_head		rcu;
 };
 
 void run_posix_cpu_timers(struct task_struct *task);
kernel/time/posix-timers.c +2 -2
index bbe8f9686a70..3e663f982c82 100644
@@ -442,7 +442,7 @@ static struct k_itimer * alloc_posix_timer(void)
 
 static void k_itimer_rcu_free(struct rcu_head *head)
 {
-	struct k_itimer *tmr = container_of(head, struct k_itimer, it.rcu);
+	struct k_itimer *tmr = container_of(head, struct k_itimer, rcu);
 
 	kmem_cache_free(posix_timers_cache, tmr);
 }
@@ -459,7 +459,7 @@ static void release_posix_timer(struct k_itimer *tmr, int it_id_set)
 	}
 	put_pid(tmr->it_pid);
 	sigqueue_free(tmr->sigq);
-	call_rcu(&tmr->it.rcu, k_itimer_rcu_free);
+	call_rcu(&tmr->rcu, k_itimer_rcu_free);
 }
 
 static int common_timer_create(struct k_itimer *new_timer)
#18 lore raw

[tip:timers/core] posix-timers: Move rcu_head out of it union

In reply to Thomas Gleixner
posix-timers: Move rcu_head out of it union
Commit-ID:  5d99b32a009e900a561f6a42ea7afe5b21288b8a
Gitweb:     https://git.kernel.org/tip/5d99b32a009e900a561f6a42ea7afe5b21288b8a
Author:     Sebastian Andrzej Siewior <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:54 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 20:51:25 +0200

posix-timers: Move rcu_head out of it union

Timer deletion on PREEMPT_RT is prone to priority inversion and live locks. The hrtimer code has a synchronization mechanism for this. Posix CPU timers will grow one.

But that mechanism cannot be invoked while holding the k_itimer lock because that can deadlock against the running timer callback. So the lock must be dropped which allows the timer to be freed.

The timer free can be prevented by taking RCU readlock before dropping the lock, but because the rcu_head is part of the 'it' union a concurrent free will overwrite the hrtimer on which the task is trying to synchronize.

Move the rcu_head out of the union to prevent this.

[ tglx: Fixed up kernel-doc. Rewrote changelog ]

---
 include/linux/posix-timers.h | 5 +++--
 kernel/time/posix-timers.c   | 4 ++--
 2 files changed, 5 insertions(+), 4 deletions(-)
include/linux/posix-timers.h +3 -2
index b20798fc5191..604cec0e41ba 100644
@@ -85,7 +85,8 @@ static inline int clockid_to_fd(const clockid_t clk)
  * @it_process:		The task to wakeup on clock_nanosleep (CPU timers)
  * @sigq:		Pointer to preallocated sigqueue
  * @it:			Union representing the various posix timer type
- *			internals. Also used for rcu freeing the timer.
+ *			internals.
+ * @rcu:		RCU head for freeing the timer.
  */
 struct k_itimer {
 	struct list_head	list;
@@ -114,8 +115,8 @@ struct k_itimer {
 		struct {
 			struct alarm	alarmtimer;
 		} alarm;
-		struct rcu_head		rcu;
 	} it;
+	struct rcu_head		rcu;
 };
 
 void run_posix_cpu_timers(struct task_struct *task);
kernel/time/posix-timers.c +2 -2
index bbe8f9686a70..3e663f982c82 100644
@@ -442,7 +442,7 @@ static struct k_itimer * alloc_posix_timer(void)
 
 static void k_itimer_rcu_free(struct rcu_head *head)
 {
-	struct k_itimer *tmr = container_of(head, struct k_itimer, it.rcu);
+	struct k_itimer *tmr = container_of(head, struct k_itimer, rcu);
 
 	kmem_cache_free(posix_timers_cache, tmr);
 }
@@ -459,7 +459,7 @@ static void release_posix_timer(struct k_itimer *tmr, int it_id_set)
 	}
 	put_pid(tmr->it_pid);
 	sigqueue_free(tmr->sigq);
-	call_rcu(&tmr->it.rcu, k_itimer_rcu_free);
+	call_rcu(&tmr->rcu, k_itimer_rcu_free);
 }
 
 static int common_timer_create(struct k_itimer *new_timer)
#19 lore raw

PATCH 7/7 posix-timers: Prepare for PREEMPT_RT

In reply to Thomas Gleixner

Posix timer delete retry loops are affected by the same priority inversion and live lock issues as the other timers.

Provide a RT specific synchronization function which keeps a reference to the timer by holding rcu read lock to prevent the timer from being freed, dropping the timer lock and invoking the timer specific wait function.

This does not yet cover posix CPU timers because they need more special treatment on PREEMPT_RT.

---
 kernel/time/posix-timers.c |   22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
kernel/time/posix-timers.c +22 -0
@@ -805,6 +805,27 @@ static int common_hrtimer_try_to_cancel(
 	return hrtimer_try_to_cancel(&timr->it.real.timer);
 }
 
+#ifdef CONFIG_PREEMPT_RT
+static struct k_itimer *timer_wait_running(struct k_itimer *timer,
+					   unsigned long *flags)
+{
+	const struct k_clock *kc = READ_ONCE(timer->kclock);
+	timer_t timer_id = READ_ONCE(timer->it_id);
+
+	/* Prevent kfree(timer) after dropping the lock */
+	rcu_read_lock();
+	unlock_timer(timer, *flags);
+
+	if (kc->timer_arm == common_hrtimer_arm)
+		hrtimer_cancel_wait_running(&timer->it.real.timer);
+	else if (kc == &alarm_clock)
+		hrtimer_cancel_wait_running(&timer->it.alarm.alarmtimer.timer);
+	rcu_read_unlock();
+
+	/* Relock the timer. It might be not longer hashed. */
+	return lock_timer(timer_id, flags);
+}
+#else
 static struct k_itimer *timer_wait_running(struct k_itimer *timer,
 					   unsigned long *flags)
 {
@@ -815,6 +836,7 @@ static struct k_itimer *timer_wait_runni
 	/* Relock the timer. It might be not longer hashed. */
 	return lock_timer(timer_id, flags);
 }
+#endif
 
 /* Set a POSIX.1b interval timer. */
 int common_timer_set(struct k_itimer *timr, int flags,
#20 lore raw

Re: [patch 7/7] posix-timers: Prepare for PREEMPT_RT

Quoted 4 lines · Thomas Gleixner
kernel/time/posix-timers.c +4 -0
+static struct k_itimer *timer_wait_running(struct k_itimer *timer,
+					   unsigned long *flags)
+{
+	const struct k_clock *kc = READ_ONCE(timer->kclock);

kernel/time/posix-timers.c static int common_hrtimer_try_to_cancel( · line 822

Quoted 10 lines
kernel/time/posix-timers.c +10 -0
+	timer_t timer_id = READ_ONCE(timer->it_id);
+
+	/* Prevent kfree(timer) after dropping the lock */
+	rcu_read_lock();
+	unlock_timer(timer, *flags);
+
+	if (kc->timer_arm == common_hrtimer_arm)
+		hrtimer_cancel_wait_running(&timer->it.real.timer);
+	else if (kc == &alarm_clock)
+		hrtimer_cancel_wait_running(&timer->it.alarm.alarmtimer.timer);
	else WARN();
Quoted 5 lines
kernel/time/posix-timers.c +5 -0
+	rcu_read_unlock();
+
+	/* Relock the timer. It might be not longer hashed. */
+	return lock_timer(timer_id, flags);
+}
#21 lore raw

Re: [patch 7/7] posix-timers: Prepare for PREEMPT_RT

Quoted 17 lines · Peter Zijlstra
Quoted 4 lines · Thomas Gleixner
kernel/time/posix-timers.c +4 -0
+static struct k_itimer *timer_wait_running(struct k_itimer *timer,
+					   unsigned long *flags)
+{
+	const struct k_clock *kc = READ_ONCE(timer->kclock);

kernel/time/posix-timers.c static int common_hrtimer_try_to_cancel( · line 822

Quoted 10 lines
kernel/time/posix-timers.c +10 -0
+	timer_t timer_id = READ_ONCE(timer->it_id);
+
+	/* Prevent kfree(timer) after dropping the lock */
+	rcu_read_lock();
+	unlock_timer(timer, *flags);
+
+	if (kc->timer_arm == common_hrtimer_arm)
+		hrtimer_cancel_wait_running(&timer->it.real.timer);
+	else if (kc == &alarm_clock)
+		hrtimer_cancel_wait_running(&timer->it.alarm.alarmtimer.timer);
	else WARN();

Yup. Working on it ...

#22 lore raw

[tip:timers/core] posix-timers: Prepare for PREEMPT_RT

In reply to Thomas Gleixner
posix-timers: Prepare for PREEMPT_RT
Commit-ID:  3a839db3eaeeef31520de45f3b078204d068e3d0
Gitweb:     https://git.kernel.org/tip/3a839db3eaeeef31520de45f3b078204d068e3d0
Author:     Thomas Gleixner <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:55 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 17:46:43 +0200

posix-timers: Prepare for PREEMPT_RT

Posix timer delete retry loops are affected by the same priority inversion and live lock issues as the other timers.

Provide a RT specific synchronization function which keeps a reference to the timer by holding rcu read lock to prevent the timer from being freed, dropping the timer lock and invoking the timer specific wait function.

This does not yet cover posix CPU timers because they need more special treatment on PREEMPT_RT.

---
 kernel/time/posix-timers.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
kernel/time/posix-timers.c +24 -0
index 3e663f982c82..a71c1aab071c 100644
@@ -805,6 +805,29 @@ static int common_hrtimer_try_to_cancel(struct k_itimer *timr)
 	return hrtimer_try_to_cancel(&timr->it.real.timer);
 }
 
+#ifdef CONFIG_PREEMPT_RT
+static struct k_itimer *timer_wait_running(struct k_itimer *timer,
+					   unsigned long *flags)
+{
+	const struct k_clock *kc = READ_ONCE(timer->kclock);
+	timer_t timer_id = READ_ONCE(timer->it_id);
+
+	/* Prevent kfree(timer) after dropping the lock */
+	rcu_read_lock();
+	unlock_timer(timer, *flags);
+
+	if (kc->timer_arm == common_hrtimer_arm)
+		hrtimer_cancel_wait_running(&timer->it.real.timer);
+	else if (kc == &alarm_clock)
+		hrtimer_cancel_wait_running(&timer->it.alarm.alarmtimer.timer);
+	else
+		WARN_ON_ONCE(1);
+	rcu_read_unlock();
+
+	/* Relock the timer. It might be not longer hashed. */
+	return lock_timer(timer_id, flags);
+}
+#else
 static struct k_itimer *timer_wait_running(struct k_itimer *timer,
 					   unsigned long *flags)
 {
@@ -815,6 +838,7 @@ static struct k_itimer *timer_wait_running(struct k_itimer *timer,
 	/* Relock the timer. It might be not longer hashed. */
 	return lock_timer(timer_id, flags);
 }
+#endif
 
 /* Set a POSIX.1b interval timer. */
 int common_timer_set(struct k_itimer *timr, int flags,
#23 lore raw

[tip:timers/core] posix-timers: Prepare for PREEMPT_RT

In reply to Thomas Gleixner
posix-timers: Prepare for PREEMPT_RT
Commit-ID:  08a3c192c93f4359a94bf47971e55b0324b72b8b
Gitweb:     https://git.kernel.org/tip/08a3c192c93f4359a94bf47971e55b0324b72b8b
Author:     Thomas Gleixner <[email protected]>
AuthorDate: Wed, 31 Jul 2019 00:33:55 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 1 Aug 2019 20:51:25 +0200

posix-timers: Prepare for PREEMPT_RT

Posix timer delete retry loops are affected by the same priority inversion and live lock issues as the other timers.

Provide a RT specific synchronization function which keeps a reference to the timer by holding rcu read lock to prevent the timer from being freed, dropping the timer lock and invoking the timer specific wait function.

This does not yet cover posix CPU timers because they need more special treatment on PREEMPT_RT.

---
 kernel/time/posix-timers.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
kernel/time/posix-timers.c +24 -0
index 3e663f982c82..a71c1aab071c 100644
@@ -805,6 +805,29 @@ static int common_hrtimer_try_to_cancel(struct k_itimer *timr)
 	return hrtimer_try_to_cancel(&timr->it.real.timer);
 }
 
+#ifdef CONFIG_PREEMPT_RT
+static struct k_itimer *timer_wait_running(struct k_itimer *timer,
+					   unsigned long *flags)
+{
+	const struct k_clock *kc = READ_ONCE(timer->kclock);
+	timer_t timer_id = READ_ONCE(timer->it_id);
+
+	/* Prevent kfree(timer) after dropping the lock */
+	rcu_read_lock();
+	unlock_timer(timer, *flags);
+
+	if (kc->timer_arm == common_hrtimer_arm)
+		hrtimer_cancel_wait_running(&timer->it.real.timer);
+	else if (kc == &alarm_clock)
+		hrtimer_cancel_wait_running(&timer->it.alarm.alarmtimer.timer);
+	else
+		WARN_ON_ONCE(1);
+	rcu_read_unlock();
+
+	/* Relock the timer. It might be not longer hashed. */
+	return lock_timer(timer_id, flags);
+}
+#else
 static struct k_itimer *timer_wait_running(struct k_itimer *timer,
 					   unsigned long *flags)
 {
@@ -815,6 +838,7 @@ static struct k_itimer *timer_wait_running(struct k_itimer *timer,
 	/* Relock the timer. It might be not longer hashed. */
 	return lock_timer(timer_id, flags);
 }
+#endif
 
 /* Set a POSIX.1b interval timer. */
 int common_timer_set(struct k_itimer *timr, int flags,
Acked-by on the v1 series
#24 lore raw
In reply to Thomas Gleixner
Quoted 7 lines · Thomas Gleixner

The following series prepares posix-timers for RT. The main change here is to utilize the hrtimer synchronization mechanism to prevent priority inversion and live locks on timer deletion.

This does not cover the posix CPU timers as they need more special treatment for RT which is covered in a separate series.