Skip to contentSkip to conversation

signal: Last two bits for PREEMPT_RT

PATCH Superseded Sebastian Andrzej Siewior 2 participants 8 messages Jul 20, 2022 – Jul 26, 2022 Scheduler

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover signal: Last two bits for PREEMPT_RT

This are the last two signal/ptrace related bits to get PREEMPT_RT working.

Sebastian

#1 lore raw

PATCH 1/2 signal: Don't disable preemption in ptrace_stop() on PREEMPT_RT.

Commit
53da1d9456fe7 ("fix ptrace slowness")

is just band aid around the problem.
The invocation of do_notify_parent_cldstop() wakes the parent and makes it runnable. The scheduler then wants to replace this still running task with the parent. With the read_lock() acquired this is not possible because preemption is disabled and so this is deferred until read_unlock(). This scheduling point is undesired and is avoided by disabling preemption around the unlock operation enabled again before the schedule() invocation without a preemption point.
This is only undesired because the parent sleeps a cycle in wait_task_inactive() until the traced task leaves the run-queue in schedule(). It is not a correctness issue, it is just band aid to avoid the visbile delay which sums up over multiple invocations.
The task can still be preempted if an interrupt occurs between preempt_enable_no_resched() and freezable_schedule() because on the IRQ-exit path of the interrupt scheduling _will_ happen. This is ignored since it does not happen very often.

On PREEMPT_RT keeping preemption disabled during the invocation of cgroup_enter_frozen() becomes a problem because the function acquires css_set_lock which is a sleeping lock on PREEMPT_RT and must not be acquired with disabled preemption.

Don't disable preemption on PREEMPT_RT. Remove the TODO regarding adding read_unlock_no_resched() as there is no need for it and will cause harm.

---
 kernel/signal.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/signal.c +4 -4
@@ -2297,13 +2297,13 @@ static int ptrace_stop(int exit_code, in
 	/*
 	 * Don't want to allow preemption here, because
 	 * sys_ptrace() needs this task to be inactive.
-	 *
-	 * XXX: implement read_unlock_no_resched().
 	 */
-	preempt_disable();
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+		preempt_disable();
 	read_unlock(&tasklist_lock);
 	cgroup_enter_frozen();
-	preempt_enable_no_resched();
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+		preempt_enable_no_resched();
 	freezable_schedule();
 	cgroup_leave_frozen(true);
 
#2 lore raw

PATCH 2/2 sched: Consider task_struct::saved_state in wait_task_inactive().

In reply to Sebastian Andrzej Siewior

Ptrace is using wait_task_inactive() to wait for the tracee to reach a certain task state. On PREEMPT_RT that state may be stored in task_struct::saved_state while the tracee blocks on a sleeping lock and task_struct::__state is set to TASK_RTLOCK_WAIT.
It is not possible to check only for TASK_RTLOCK_WAIT to be sure that the task is blocked on a sleeping lock because during wake up (after the sleeping lock has been acquired) the task state is set TASK_RUNNING. After the task in on CPU and acquired the pi_lock it will reset the state accordingly but until then TASK_RUNNING will be observed (with the desired state saved in saved_state).

Check also for task_struct::saved_state if the desired match was not found in task_struct::__state on PREEMPT_RT. If the state was found in saved_state, wait until the task is idle and state is visible in task_struct::__state.

---
 kernel/sched/core.c |   46 +++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 41 insertions(+), 5 deletions(-)
kernel/sched/core.c +41 -5
@@ -3257,6 +3257,40 @@ int migrate_swap(struct task_struct *cur
 }
 #endif /* CONFIG_NUMA_BALANCING */
 
+#ifdef CONFIG_PREEMPT_RT
+static __always_inline bool state_mismatch(struct task_struct *p, unsigned int match_state)
+{
+	unsigned long flags;
+	bool mismatch;
+
+	raw_spin_lock_irqsave(&p->pi_lock, flags);
+	mismatch = READ_ONCE(p->__state) != match_state &&
+		READ_ONCE(p->saved_state) != match_state;
+	raw_spin_unlock_irqrestore(&p->pi_lock, flags);
+	return mismatch;
+}
+static __always_inline bool state_match(struct task_struct *p, unsigned int match_state,
+					bool *wait)
+{
+	if (READ_ONCE(p->__state) == match_state)
+		return true;
+	if (READ_ONCE(p->saved_state) != match_state)
+		return false;
+	*wait = true;
+	return true;
+}
+#else
+static __always_inline bool state_mismatch(struct task_struct *p, unsigned int match_state)
+{
+	return READ_ONCE(p->__state) != match_state;
+}
+static __always_inline bool state_match(struct task_struct *p, unsigned int match_state,
+					bool *wait)
+{
+	return READ_ONCE(p->__state) == match_state;
+}
+#endif
+
 /*
  * wait_task_inactive - wait for a thread to unschedule.
  *
@@ -3275,7 +3309,7 @@ int migrate_swap(struct task_struct *cur
  */
 unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state)
 {
-	int running, queued;
+	bool running, wait;
 	struct rq_flags rf;
 	unsigned long ncsw;
 	struct rq *rq;
@@ -3301,7 +3335,7 @@ unsigned long wait_task_inactive(struct
 		 * is actually now running somewhere else!
 		 */
 		while (task_running(rq, p)) {
-			if (match_state && unlikely(READ_ONCE(p->__state) != match_state))
+			if (match_state && state_mismatch(p, match_state))
 				return 0;
 			cpu_relax();
 		}
@@ -3314,10 +3348,12 @@ unsigned long wait_task_inactive(struct
 		rq = task_rq_lock(p, &rf);
 		trace_sched_wait_task(p);
 		running = task_running(rq, p);
-		queued = task_on_rq_queued(p);
+		wait = task_on_rq_queued(p);
 		ncsw = 0;
-		if (!match_state || READ_ONCE(p->__state) == match_state)
+
+		if (!match_state || state_match(p, match_state, &wait))
 			ncsw = p->nvcsw | LONG_MIN; /* sets MSB */
+
 		task_rq_unlock(rq, p, &rf);
 
 		/*
@@ -3346,7 +3382,7 @@ unsigned long wait_task_inactive(struct
 		 * running right now), it's preempted, and we should
 		 * yield - it could be a while.
 		 */
-		if (unlikely(queued)) {
+		if (unlikely(wait)) {
 			ktime_t to = NSEC_PER_SEC / HZ;
 
 			set_current_state(TASK_UNINTERRUPTIBLE);
reviewer Reviewed-by on v1 2/2
#3 lore raw

Re: [PATCH 2/2] sched: Consider task_struct::saved_state in wait_task_inactive().

Quoted 10 lines · Sebastian Andrzej Siewior

Ptrace is using wait_task_inactive() to wait for the tracee to reach a certain task state. On PREEMPT_RT that state may be stored in task_struct::saved_state while the tracee blocks on a sleeping lock and task_struct::__state is set to TASK_RTLOCK_WAIT.
It is not possible to check only for TASK_RTLOCK_WAIT to be sure that the task is blocked on a sleeping lock because during wake up (after the sleeping lock has been acquired) the task state is set TASK_RUNNING. After the task in on CPU and acquired the pi_lock it will reset the state accordingly but until then TASK_RUNNING will be observed (with the desired state saved in saved_state).

Quoted 5 lines

Check also for task_struct::saved_state if the desired match was not found in task_struct::__state on PREEMPT_RT. If the state was found in saved_state, wait until the task is idle and state is visible in task_struct::__state.

I have a suggestion and a comment below, but other than that this looks OK.

kernel/sched/core.c int migrate_swap(struct task_struct *cur · line 3260

Quoted 11 lines
---
 kernel/sched/core.c |   46 +++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 41 insertions(+), 5 deletions(-)
kernel/sched/core.c +1 -0
@@ -3257,6 +3257,40 @@ int migrate_swap(struct task_struct *cur
 }
 #endif /* CONFIG_NUMA_BALANCING */
 
+#ifdef CONFIG_PREEMPT_RT

Would something like the below be useful?

/*
* If p->saved_state is anything else than TASK_RUNNING, then p blocked on an
* rtlock *before* voluntarily calling into schedule() after setting its state
* to X. For things like ptrace (X=TASK_TRACED), the task could have more work
* to do upon acquiring the lock before whoever called wait_task_inactive()
* should return. IOW, we have to wait for:
*
* p.saved_state = TASK_RUNNING
* p.__state = X
*
* which implies the task isn't blocked on an RT lock and got to schedule() by
* itself.
*
* Also see comments in ttwu_state_match().
*/

Quoted 37 lines
kernel/sched/core.c +33 -0
+static __always_inline bool state_mismatch(struct task_struct *p, unsigned int match_state)
+{
+	unsigned long flags;
+	bool mismatch;
+
+	raw_spin_lock_irqsave(&p->pi_lock, flags);
+	mismatch = READ_ONCE(p->__state) != match_state &&
+		READ_ONCE(p->saved_state) != match_state;
+	raw_spin_unlock_irqrestore(&p->pi_lock, flags);
+	return mismatch;
+}
+static __always_inline bool state_match(struct task_struct *p, unsigned int match_state,
+					bool *wait)
+{
+	if (READ_ONCE(p->__state) == match_state)
+		return true;
+	if (READ_ONCE(p->saved_state) != match_state)
+		return false;
+	*wait = true;
+	return true;
+}
+#else
+static __always_inline bool state_mismatch(struct task_struct *p, unsigned int match_state)
+{
+	return READ_ONCE(p->__state) != match_state;
+}
+static __always_inline bool state_match(struct task_struct *p, unsigned int match_state,
+					bool *wait)
+{
+	return READ_ONCE(p->__state) == match_state;
+}
+#endif
+
 /*
  * wait_task_inactive - wait for a thread to unschedule.
  *

kernel/sched/core.c unsigned long wait_task_inactive(struct · line 3385

Quoted 5 lines
kernel/sched/core.c +1 -1
@@ -3346,7 +3382,7 @@ unsigned long wait_task_inactive(struct
               * running right now), it's preempted, and we should
               * yield - it could be a while.
               */
-		if (unlikely(queued)) {
+		if (unlikely(wait)) {

We could be repeatedly doing this for as long as the task is blocked on the rtlock, but IIUC that's the same story on !PREEMPT_RT if it's just a queued task preempted by a higher prio task, it may take a while for it to schedule() and dequeue...

Quoted 3 lines
kernel/sched/core.c +0 -0
                      ktime_t to = NSEC_PER_SEC / HZ;
 
                      set_current_state(TASK_UNINTERRUPTIBLE);
#4 lore raw

Re: [PATCH 2/2] sched: Consider task_struct::saved_state in wait_task_inactive().

Quoted 10 lines · Valentin Schneider

kernel/sched/core.c int migrate_swap(struct task_struct *cur · line 3260

Quoted 7 lines
kernel/sched/core.c +1 -0
@@ -3257,6 +3257,40 @@ int migrate_swap(struct task_struct *cur
 }
 #endif /* CONFIG_NUMA_BALANCING */
 
+#ifdef CONFIG_PREEMPT_RT

Would something like the below be useful?

Quoted 15 lines

/*
* If p->saved_state is anything else than TASK_RUNNING, then p blocked on an
* rtlock *before* voluntarily calling into schedule() after setting its state
* to X. For things like ptrace (X=TASK_TRACED), the task could have more work
* to do upon acquiring the lock before whoever called wait_task_inactive()
* should return. IOW, we have to wait for:
*
* p.saved_state = TASK_RUNNING
* p.__state = X
*
* which implies the task isn't blocked on an RT lock and got to schedule() by
* itself.
*
* Also see comments in ttwu_state_match().
*/

This sums up the code. I would s/schedule/schedule_rtlock/ since there are two entrypoints.

…

Quoted 11 lines

kernel/sched/core.c unsigned long wait_task_inactive(struct · line 3385

Quoted 6 lines
kernel/sched/core.c +1 -1
@@ -3346,7 +3382,7 @@ unsigned long wait_task_inactive(struct
               * running right now), it's preempted, and we should
               * yield - it could be a while.
               */
-		if (unlikely(queued)) {
+		if (unlikely(wait)) {

We could be repeatedly doing this for as long as the task is blocked on the rtlock, but IIUC that's the same story on !PREEMPT_RT if it's just a queued task preempted by a higher prio task, it may take a while for it to schedule() and dequeue...

Yes.

Quoted 3 lines

kernel/sched/core.c unsigned long wait_task_inactive(struct · line 3388

Quoted 3 lines
kernel/sched/core.c +0 -0
                      ktime_t to = NSEC_PER_SEC / HZ;
 
                      set_current_state(TASK_UNINTERRUPTIBLE);

Sebastian

#5 lore raw

Re: [PATCH 2/2] sched: Consider task_struct::saved_state in wait_task_inactive().

Quoted 27 lines · Sebastian Andrzej Siewior
Quoted 10 lines · Valentin Schneider

kernel/sched/core.c int migrate_swap(struct task_struct *cur · line 3260

Quoted 7 lines
kernel/sched/core.c +1 -0
@@ -3257,6 +3257,40 @@ int migrate_swap(struct task_struct *cur
 }
 #endif /* CONFIG_NUMA_BALANCING */
 
+#ifdef CONFIG_PREEMPT_RT

Would something like the below be useful?

Quoted 15 lines

/*
* If p->saved_state is anything else than TASK_RUNNING, then p blocked on an
* rtlock *before* voluntarily calling into schedule() after setting its state
* to X. For things like ptrace (X=TASK_TRACED), the task could have more work
* to do upon acquiring the lock before whoever called wait_task_inactive()
* should return. IOW, we have to wait for:
*
* p.saved_state = TASK_RUNNING
* p.__state = X
*
* which implies the task isn't blocked on an RT lock and got to schedule() by
* itself.
*
* Also see comments in ttwu_state_match().
*/

Quoted 2 lines

This sums up the code. I would s/schedule/schedule_rtlock/ since there are two entrypoints.

Right, this any better?

/*
* Consider:
*
* set_special_state(X);
*
* do_things()

 *    // Somewhere in there is an rtlock that can be contended:
 *    current_save_and_set_rtlock_wait_state();
 *    [...]
 *    schedule_rtlock(); (A)
 *    [...]
 *    current_restore_rtlock_saved_state();

*
* schedule(); (B)
*
* If p->saved_state is anything else than TASK_RUNNING, then p blocked on an
* rtlock (A) *before* voluntarily calling into schedule() (B) after setting its
* state to X. For things like ptrace (X=TASK_TRACED), the task could have more
* work to do upon acquiring the lock in do_things() before whoever called
* wait_task_inactive() should return. IOW, we have to wait for:
*
* p.saved_state = TASK_RUNNING
* p.__state = X
*
* which implies the task isn't blocked on an RT lock and got to schedule() (B). *
* Also see comments in ttwu_state_match().
*/

#6 lore raw

PATCH v2 2/2 sched: Consider task_struct::saved_state in wait_task_inactive().

Ptrace is using wait_task_inactive() to wait for the tracee to reach a certain task state. On PREEMPT_RT that state may be stored in task_struct::saved_state while the tracee blocks on a sleeping lock and task_struct::__state is set to TASK_RTLOCK_WAIT.
It is not possible to check only for TASK_RTLOCK_WAIT to be sure that the task is blocked on a sleeping lock because during wake up (after the sleeping lock has been acquired) the task state is set TASK_RUNNING. After the task in on CPU and acquired the pi_lock it will reset the state accordingly but until then TASK_RUNNING will be observed (with the desired state saved in saved_state).

Check also for task_struct::saved_state if the desired match was not found in task_struct::__state on PREEMPT_RT. If the state was found in saved_state, wait until the task is idle and state is visible in task_struct::__state.

---
v1…v2:
- Add a comment suggested by Valentin Schneider.

 kernel/sched/core.c |   76 ++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 71 insertions(+), 5 deletions(-)
kernel/sched/core.c +71 -5
@@ -3257,6 +3257,70 @@ int migrate_swap(struct task_struct *cur
 }
 #endif /* CONFIG_NUMA_BALANCING */
 
+#ifdef CONFIG_PREEMPT_RT
+
+/*
+ * Consider:
+ *
+ *  set_special_state(X);
+ *
+ *  do_things()
+ *    // Somewhere in there is an rtlock that can be contended:
+ *    current_save_and_set_rtlock_wait_state();
+ *    [...]
+ *    schedule_rtlock(); (A)
+ *    [...]
+ *    current_restore_rtlock_saved_state();
+ *
+ *  schedule(); (B)
+ *
+ * If p->saved_state is anything else than TASK_RUNNING, then p blocked on an
+ * rtlock (A) *before* voluntarily calling into schedule() (B) after setting its
+ * state to X. For things like ptrace (X=TASK_TRACED), the task could have more
+ * work to do upon acquiring the lock in do_things() before whoever called
+ * wait_task_inactive() should return. IOW, we have to wait for:
+ *
+ *   p.saved_state = TASK_RUNNING
+ *   p.__state     = X
+ *
+ * which implies the task isn't blocked on an RT lock and got to schedule() (B).
+ *
+ * Also see comments in ttwu_state_match().
+ */
+
+static __always_inline bool state_mismatch(struct task_struct *p, unsigned int match_state)
+{
+	unsigned long flags;
+	bool mismatch;
+
+	raw_spin_lock_irqsave(&p->pi_lock, flags);
+	mismatch = READ_ONCE(p->__state) != match_state &&
+		READ_ONCE(p->saved_state) != match_state;
+	raw_spin_unlock_irqrestore(&p->pi_lock, flags);
+	return mismatch;
+}
+static __always_inline bool state_match(struct task_struct *p, unsigned int match_state,
+					bool *wait)
+{
+	if (READ_ONCE(p->__state) == match_state)
+		return true;
+	if (READ_ONCE(p->saved_state) != match_state)
+		return false;
+	*wait = true;
+	return true;
+}
+#else
+static __always_inline bool state_mismatch(struct task_struct *p, unsigned int match_state)
+{
+	return READ_ONCE(p->__state) != match_state;
+}
+static __always_inline bool state_match(struct task_struct *p, unsigned int match_state,
+					bool *wait)
+{
+	return READ_ONCE(p->__state) == match_state;
+}
+#endif
+
 /*
  * wait_task_inactive - wait for a thread to unschedule.
  *
@@ -3275,7 +3339,7 @@ int migrate_swap(struct task_struct *cur
  */
 unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state)
 {
-	int running, queued;
+	bool running, wait;
 	struct rq_flags rf;
 	unsigned long ncsw;
 	struct rq *rq;
@@ -3301,7 +3365,7 @@ unsigned long wait_task_inactive(struct
 		 * is actually now running somewhere else!
 		 */
 		while (task_running(rq, p)) {
-			if (match_state && unlikely(READ_ONCE(p->__state) != match_state))
+			if (match_state && state_mismatch(p, match_state))
 				return 0;
 			cpu_relax();
 		}
@@ -3314,10 +3378,12 @@ unsigned long wait_task_inactive(struct
 		rq = task_rq_lock(p, &rf);
 		trace_sched_wait_task(p);
 		running = task_running(rq, p);
-		queued = task_on_rq_queued(p);
+		wait = task_on_rq_queued(p);
 		ncsw = 0;
-		if (!match_state || READ_ONCE(p->__state) == match_state)
+
+		if (!match_state || state_match(p, match_state, &wait))
 			ncsw = p->nvcsw | LONG_MIN; /* sets MSB */
+
 		task_rq_unlock(rq, p, &rf);
 
 		/*
@@ -3346,7 +3412,7 @@ unsigned long wait_task_inactive(struct
 		 * running right now), it's preempted, and we should
 		 * yield - it could be a while.
 		 */
-		if (unlikely(queued)) {
+		if (unlikely(wait)) {
 			ktime_t to = NSEC_PER_SEC / HZ;
 
 			set_current_state(TASK_UNINTERRUPTIBLE);
#7 lore raw
In reply to Sebastian Andrzej Siewior
Quoted 2 lines · On 2022-07-20 17:44:33 [+0200], To [email protected] wrote:

This are the last two signal/ptrace related bits to get PREEMPT_RT working.

a polite ping for the series. This is one of two road blocks to get RT enabled in v5.20. I don't want to add any pressure just point out that I can't sit still for days since the end is near ;)

Sebastian