Skip to contentSkip to conversation

psi: stop relying on timer_pending for poll_work rescheduling

PATCH Superseded Suren Baghdasaryan 2 participants 8 messages Jul 7, 2021 – Jul 8, 2021

Subsystems (3)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Suren Baghdasaryan

author maintainer
#0 lore raw

PATCH v3 1/1 psi: stop relying on timer_pending for poll_work rescheduling

Psi polling mechanism is trying to minimize the number of wakeups to run psi_poll_work and is currently relying on timer_pending() to detect when this work is already scheduled. This provides a window of opportunity for psi_group_change to schedule an immediate psi_poll_work after poll_timer_fn got called but before psi_poll_work could reschedule itself. Below is the depiction of this entire window:

poll_timer_fn
wake_up_interruptible(&group->poll_wait);

psi_poll_worker
wait_event_interruptible(group->poll_wait, ...)
psi_poll_work

    psi_schedule_poll_work
      if (timer_pending(&group->poll_timer)) return;
      ...
      mod_timer(&group->poll_timer, jiffies + delay);

Prior to 461daba06bdc we used to rely on poll_scheduled atomic which was reset and set back inside psi_poll_work and therefore this race window was much smaller.
The larger window causes increased number of wakeups and our partners report visible power regression of ~10mA after applying 461daba06bdc. Bring back the poll_scheduled atomic and make this race window even narrower by resetting poll_scheduled only when we reach polling expiration time. This does not completely eliminate the possibility of extra wakeups caused by a race with psi_group_change however it will limit it to the worst case scenario of one extra wakeup per every tracking window (0.5s in the worst case).
This patch also ensures correct ordering between clearing poll_scheduled flag and obtaining changed_states using memory barrier. Correct ordering between updating changed_states and setting poll_scheduled is ensured by atomic_xchg operation.
By tracing the number of immediate rescheduling attempts performed by psi_group_change and the number of these attempts being blocked due to psi monitor being already active, we can assess the effects of this change:

Before the patch:

                                           Run#1    Run#2      Run#3
Immediate reschedules attempted:           684365   1385156    1261240
Immediate reschedules blocked:             682846   1381654    1258682
Immediate reschedules (delta):             1519     3502       2558
Immediate reschedules (% of attempted):    0.22%    0.25%      0.20%

After the patch:

                                           Run#1    Run#2      Run#3
Immediate reschedules attempted:           882244   770298    426218
Immediate reschedules blocked:             881996   769796    426074
Immediate reschedules (delta):             248      502       144
Immediate reschedules (% of attempted):    0.03%    0.07%     0.03%

The number of non-blocked immediate reschedules dropped from 0.22-0.25% to 0.03-0.07%. The drop is attributed to the decrease in the race window size and the fact that we allow this race only when psi monitors reach polling window expiration time.

Fixes 461daba06bdc ("psi: eliminate kthread_worker from psi trigger scheduling mechanism")

---
- Replaced atomic_cmpxchg() with atomic_xchg() to ensure correct ordering per PeterZ
- Added memory barrier between resetting poll_scheduled and obtaining changed_states per PeterZ and Johannes
- Added a paragraph in the patch description about the ordering guarantees added in this patch

 include/linux/psi_types.h |  1 +
 kernel/sched/psi.c        | 46 +++++++++++++++++++++++++++++----------
 2 files changed, 36 insertions(+), 11 deletions(-)
include/linux/psi_types.h +1 -0
index 0a23300d49af..ef8bd89d065e 100644
@@ -158,6 +158,7 @@ struct psi_group {
 	struct timer_list poll_timer;
 	wait_queue_head_t poll_wait;
 	atomic_t poll_wakeup;
+	atomic_t poll_scheduled;
 
 	/* Protects data used by the monitor */
 	struct mutex trigger_lock;
kernel/sched/psi.c +35 -11

97 lines not on this page · raw message

Signature · 1 line
2.32.0.93.g670b81a890-goog
maintainer
#1 lore raw

This looks good to me now code wise. Just a comment on the comments:

Quoted 12 lines · Suren Baghdasaryan
kernel/sched/psi.c +3 -2
@@ -559,18 +560,14 @@ static u64 update_triggers(struct psi_group *group, u64 now)
 	return now + group->poll_min_period;
 }
 
-/* Schedule polling if it's not already scheduled. */
-static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay)
+/* Schedule polling if it's not already scheduled or forced. */
+static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay,
+				   bool force)
 {
 	struct task_struct *task;
 

kernel/sched/psi.c static u64 update_triggers(struct psi_group *group, u64 now) · line 571

Quoted 10 lines
kernel/sched/psi.c +2 -7
-	/*
-	 * Do not reschedule if already scheduled.
-	 * Possible race with a timer scheduled after this check but before
-	 * mod_timer below can be tolerated because group->polling_next_update
-	 * will keep updates on schedule.
-	 */
-	if (timer_pending(&group->poll_timer))
+	/* xchg should be called even when !force to set poll_scheduled */
+	if (atomic_xchg(&group->poll_scheduled, 1) && !force)
 		return;

This explains what the code does, but not why. It would be good to explain the ordering with poll_work, here or there. But both sides should mention each other.

Quoted 10 lines
kernel/sched/psi.c +6 -0
@@ -595,6 +595,28 @@ static void psi_poll_work(struct psi_group *group)
 
 	now = sched_clock();
 
+	if (now > group->polling_until) {
+		/*
+		 * We are either about to start or might stop polling if no
+		 * state change was recorded. Resetting poll_scheduled leaves
+		 * a small window for psi_group_change to sneak in and schedule
+		 * an immegiate poll_work before we get to rescheduling. One

kernel/sched/psi.c static void psi_poll_work(struct psi_group *group) · line 613

Quoted 10 lines
kernel/sched/psi.c +10 -0
+		 * potential extra wakeup at the end of the polling window
+		 * should be negligible and polling_next_update still keeps
+		 * updates correctly on schedule.
+		 */
+		atomic_set(&group->poll_scheduled, 0);
+		/*
+		 * Ensure that operations of clearing group->poll_scheduled and
+		 * obtaining changed_states are not reordered.
+		 */
+		smp_mb();

Same here, it would be good to explain that this is ordering the scheduler with the timer such that no events are missed. Feel free to reuse my race diagram from the other thread - those are better at conveying the situation than freeform text.

Thanks

author maintainer
#2 lore raw
Quoted 16 lines · Johannes Weiner

This looks good to me now code wise. Just a comment on the comments:

Quoted 12 lines · Suren Baghdasaryan
kernel/sched/psi.c +3 -2
@@ -559,18 +560,14 @@ static u64 update_triggers(struct psi_group *group, u64 now)
      return now + group->poll_min_period;
 }
 
-/* Schedule polling if it's not already scheduled. */
-static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay)
+/* Schedule polling if it's not already scheduled or forced. */
+static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay,
+                                bool force)
 {
      struct task_struct *task;
 
Quoted 14 lines

kernel/sched/psi.c static u64 update_triggers(struct psi_group *group, u64 now) · line 571

Quoted 10 lines
kernel/sched/psi.c +2 -7
-     /*
-      * Do not reschedule if already scheduled.
-      * Possible race with a timer scheduled after this check but before
-      * mod_timer below can be tolerated because group->polling_next_update
-      * will keep updates on schedule.
-      */
-     if (timer_pending(&group->poll_timer))
+     /* xchg should be called even when !force to set poll_scheduled */
+     if (atomic_xchg(&group->poll_scheduled, 1) && !force)
              return;

This explains what the code does, but not why. It would be good to explain the ordering with poll_work, here or there. But both sides should mention each other.

How about this:

/*
* atomic_xchg should be called even when !force to always set poll_scheduled
* and to provide a memory barrier (see the comment inside psi_poll_work). */

Quoted 18 lines
Quoted 18 lines
kernel/sched/psi.c +13 -0
@@ -595,6 +595,28 @@ static void psi_poll_work(struct psi_group *group)
 
      now = sched_clock();
 
+     if (now > group->polling_until) {
+             /*
+              * We are either about to start or might stop polling if no
+              * state change was recorded. Resetting poll_scheduled leaves
+              * a small window for psi_group_change to sneak in and schedule
+              * an immegiate poll_work before we get to rescheduling. One
+              * potential extra wakeup at the end of the polling window
+              * should be negligible and polling_next_update still keeps
+              * updates correctly on schedule.
+              */
+             atomic_set(&group->poll_scheduled, 0);
+             /*
+              * Ensure that operations of clearing group->poll_scheduled and
Quoted 8 lines

kernel/sched/psi.c static void psi_poll_work(struct psi_group *group) · line 613

Quoted 3 lines
kernel/sched/psi.c +3 -0
+              * obtaining changed_states are not reordered.
+              */
+             smp_mb();

Same here, it would be good to explain that this is ordering the scheduler with the timer such that no events are missed. Feel free to reuse my race diagram from the other thread - those are better at conveying the situation than freeform text.

I tried to make your diagram a bit less abstract by using the actual names. How about this?

/*
* We need to enforce ordering between poll_scheduled and psi_group_cpu.times
* reads and writes in psi_poll_work and psi_group_change functions.
Otherwise we
* might fail to reschedule the timer when monitored states change:
*
* psi_poll_work:

 *     poll_scheduled = 0
 *     smp_mb()
 *     changed_states = collect_percpu_times()
 *     if changed_states && xchg(poll_scheduled, 1) == 0
 *         mod_timer()
 *
 * psi_group_change:
 *     record_times()
 *     smp_mb()
 *     if xchg(poll_scheduled, 1) == 0

* mod_timer()
*
* atomic_xchg in psi_schedule_poll_work implements an implicit memory
barrier but
* we need an explicit one here.
*/

If we remove smp_mb barriers then there are the following possible reordering cases:

Case1: reordering in psi_poll_work
psi_poll_work psi_group_change
changed_states = collect_percpu_times()
record_times() if xchg(poll_scheduled,
1) == 0 <-- false

                                                  mod_timer()
  poll_scheduled = 0
  if changed_states && xchg(poll_scheduled, 1) == 0 <-- changed_states is false
      mod_timer()

Case2: reordering in psi_group_change
psi_poll_work psi_group_change
if xchg(poll_scheduled,
1) == 0 <-- false

                                                  mod_timer()
  poll_scheduled = 0
  changed_states = collect_percpu_times()
                                                  record_times()
  if changed_states && xchg(poll_scheduled, 1) == 0 <-- changed_states is false
      mod_timer()

In both cases mod_timer() is not called, poll update is missed. But describing this all in the comments would be an overkill IMHO. WDYT?

Quoted 2 lines

Thanks

maintainer
#3 lore raw
Quoted 32 lines · Suren Baghdasaryan
Quoted 16 lines · Johannes Weiner

This looks good to me now code wise. Just a comment on the comments:

Quoted 12 lines · Suren Baghdasaryan
kernel/sched/psi.c +3 -2
@@ -559,18 +560,14 @@ static u64 update_triggers(struct psi_group *group, u64 now)
      return now + group->poll_min_period;
 }
 
-/* Schedule polling if it's not already scheduled. */
-static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay)
+/* Schedule polling if it's not already scheduled or forced. */
+static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay,
+                                bool force)
 {
      struct task_struct *task;
 
Quoted 14 lines

kernel/sched/psi.c static u64 update_triggers(struct psi_group *group, u64 now) · line 571

Quoted 10 lines
kernel/sched/psi.c +2 -7
-     /*
-      * Do not reschedule if already scheduled.
-      * Possible race with a timer scheduled after this check but before
-      * mod_timer below can be tolerated because group->polling_next_update
-      * will keep updates on schedule.
-      */
-     if (timer_pending(&group->poll_timer))
+     /* xchg should be called even when !force to set poll_scheduled */
+     if (atomic_xchg(&group->poll_scheduled, 1) && !force)
              return;

This explains what the code does, but not why. It would be good to explain the ordering with poll_work, here or there. But both sides should mention each other.

Quoted 6 lines

How about this:

/*
* atomic_xchg should be called even when !force to always set poll_scheduled
* and to provide a memory barrier (see the comment inside psi_poll_work). */

The memory barrier part makes sense, but the first part says what the code does and the message is unclear to me. Are you worried somebody might turn this around in the future and only conditionalize on poll_scheduled when !force? Essentially, I don't see the downside of dropping that. But maybe I'm missing something.

	/*
	 * The xchg implies a full barrier that matches the one
	 * in psi_poll_work() (see corresponding comment there).
	 */
Quoted 36 lines
Quoted 17 lines
Quoted 17 lines
kernel/sched/psi.c +13 -0
@@ -595,6 +595,28 @@ static void psi_poll_work(struct psi_group *group)
 
      now = sched_clock();
 
+     if (now > group->polling_until) {
+             /*
+              * We are either about to start or might stop polling if no
+              * state change was recorded. Resetting poll_scheduled leaves
+              * a small window for psi_group_change to sneak in and schedule
+              * an immegiate poll_work before we get to rescheduling. One
+              * potential extra wakeup at the end of the polling window
+              * should be negligible and polling_next_update still keeps
+              * updates correctly on schedule.
+              */
+             atomic_set(&group->poll_scheduled, 0);
+             /*
+              * Ensure that operations of clearing group->poll_scheduled and
Quoted 8 lines

kernel/sched/psi.c static void psi_poll_work(struct psi_group *group) · line 613

Quoted 3 lines
kernel/sched/psi.c +3 -0
+              * obtaining changed_states are not reordered.
+              */
+             smp_mb();

Same here, it would be good to explain that this is ordering the scheduler with the timer such that no events are missed. Feel free to reuse my race diagram from the other thread - those are better at conveying the situation than freeform text.

I tried to make your diagram a bit less abstract by using the actual names. How about this?

/*
* We need to enforce ordering between poll_scheduled and psi_group_cpu.times
* reads and writes in psi_poll_work and psi_group_change functions.
Otherwise we
* might fail to reschedule the timer when monitored states change:
*
* psi_poll_work:

Quoted 5 lines
 *     poll_scheduled = 0
 *     smp_mb()
 *     changed_states = collect_percpu_times()
 *     if changed_states && xchg(poll_scheduled, 1) == 0

* mod_timer()

Those last two lines aren't relevant for the race, right? I'd leave those out to not distract from it.

Quoted 5 lines

* psi_group_change:

 *     record_times()
 *     smp_mb()
 *     if xchg(poll_scheduled, 1) == 0

* mod_timer()

The reason I tend to keep these more abstract is because 1) the names of the functions change (I had already sent out patches to rename half the variable and function names in this diagram), while the architecture (task change vs poll worker) likely won't, and 2) because it's easy to drown out what the reads, writes, and thus the race condition is with code details and function call indirections.

How about a compromise?

/*
* A task change can race with the poll worker that is supposed to
* report on it. To avoid missing events, ensure ordering between
* poll_scheduled and the task state accesses, such that if the poll
* worker misses the state update, the task change is guaranteed to
* reschedule the poll worker:
*
* poll worker:

 *   atomic_set(poll_scheduled, 0)
 *   smp_mb()
 *   LOAD states

*
* task change:
* STORE states
* if atomic_xchg(poll_scheduled, 1) == 0:
* schedule poll worker
*
* The atomic_xchg() implies a full barrier.
*/
smp_mb();

This gives a high-level view of what's happening but it can still be mapped to the code by following the poll_scheduled variable.

Quoted 25 lines

If we remove smp_mb barriers then there are the following possible reordering cases:

Case1: reordering in psi_poll_work
psi_poll_work psi_group_change
changed_states = collect_percpu_times()
record_times() if xchg(poll_scheduled,
1) == 0 <-- false

                                                  mod_timer()
  poll_scheduled = 0
  if changed_states && xchg(poll_scheduled, 1) == 0 <-- changed_states is false
      mod_timer()

Case2: reordering in psi_group_change
psi_poll_work psi_group_change
if xchg(poll_scheduled,
1) == 0 <-- false

                                                  mod_timer()
  poll_scheduled = 0
  changed_states = collect_percpu_times()
                                                  record_times()
  if changed_states && xchg(poll_scheduled, 1) == 0 <-- changed_states is false
      mod_timer()
Quoted 3 lines

In both cases mod_timer() is not called, poll update is missed. But describing this all in the comments would be an overkill IMHO. WDYT?

Yeah, I also think that's overkill. The failure cases can be derived from the concurrency diagram and explanation.

Thanks

author maintainer
#4 lore raw

t

Quoted 41 lines · Johannes Weiner
Quoted 32 lines · Suren Baghdasaryan
Quoted 16 lines · Johannes Weiner

This looks good to me now code wise. Just a comment on the comments:

Quoted 12 lines · Suren Baghdasaryan
kernel/sched/psi.c +3 -2
@@ -559,18 +560,14 @@ static u64 update_triggers(struct psi_group *group, u64 now)
      return now + group->poll_min_period;
 }
 
-/* Schedule polling if it's not already scheduled. */
-static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay)
+/* Schedule polling if it's not already scheduled or forced. */
+static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay,
+                                bool force)
 {
      struct task_struct *task;
 
Quoted 14 lines

kernel/sched/psi.c static u64 update_triggers(struct psi_group *group, u64 now) · line 571

Quoted 10 lines
kernel/sched/psi.c +2 -7
-     /*
-      * Do not reschedule if already scheduled.
-      * Possible race with a timer scheduled after this check but before
-      * mod_timer below can be tolerated because group->polling_next_update
-      * will keep updates on schedule.
-      */
-     if (timer_pending(&group->poll_timer))
+     /* xchg should be called even when !force to set poll_scheduled */
+     if (atomic_xchg(&group->poll_scheduled, 1) && !force)
              return;

This explains what the code does, but not why. It would be good to explain the ordering with poll_work, here or there. But both sides should mention each other.

Quoted 6 lines

How about this:

/*
* atomic_xchg should be called even when !force to always set poll_scheduled
* and to provide a memory barrier (see the comment inside psi_poll_work). */

Quoted 5 lines

The memory barrier part makes sense, but the first part says what the code does and the message is unclear to me. Are you worried somebody might turn this around in the future and only conditionalize on poll_scheduled when !force? Essentially, I don't see the downside of dropping that. But maybe I'm missing something.

Actually you are right. Originally I was worried that there might be a case when poll_scheduled==0 and force==true and if someone flips the conditions we will reschedule the timer but will not set poll_scheduled back to 1. However I don't think this condition is possible. We set force=true only when we skipped resetting poll_schedule to 0 and on initial wakeup we always reset poll_schedule. How about changing the comment to this:

/*
* atomic_xchg should be called even when !force to provide a
* full memory barrier (see the comment inside psi_poll_work). */

Quoted 47 lines

/*
* The xchg implies a full barrier that matches the one
* in psi_poll_work() (see corresponding comment there). */

Quoted 36 lines
Quoted 17 lines
Quoted 17 lines
kernel/sched/psi.c +13 -0
@@ -595,6 +595,28 @@ static void psi_poll_work(struct psi_group *group)
 
      now = sched_clock();
 
+     if (now > group->polling_until) {
+             /*
+              * We are either about to start or might stop polling if no
+              * state change was recorded. Resetting poll_scheduled leaves
+              * a small window for psi_group_change to sneak in and schedule
+              * an immegiate poll_work before we get to rescheduling. One
+              * potential extra wakeup at the end of the polling window
+              * should be negligible and polling_next_update still keeps
+              * updates correctly on schedule.
+              */
+             atomic_set(&group->poll_scheduled, 0);
+             /*
+              * Ensure that operations of clearing group->poll_scheduled and
Quoted 8 lines

kernel/sched/psi.c static void psi_poll_work(struct psi_group *group) · line 613

Quoted 3 lines
kernel/sched/psi.c +3 -0
+              * obtaining changed_states are not reordered.
+              */
+             smp_mb();

Same here, it would be good to explain that this is ordering the scheduler with the timer such that no events are missed. Feel free to reuse my race diagram from the other thread - those are better at conveying the situation than freeform text.

I tried to make your diagram a bit less abstract by using the actual names. How about this?

/*
* We need to enforce ordering between poll_scheduled and psi_group_cpu.times
* reads and writes in psi_poll_work and psi_group_change functions.
Otherwise we
* might fail to reschedule the timer when monitored states change:
*
* psi_poll_work:

Quoted 5 lines
 *     poll_scheduled = 0
 *     smp_mb()
 *     changed_states = collect_percpu_times()
 *     if changed_states && xchg(poll_scheduled, 1) == 0

* mod_timer()

Quoted 2 lines

Those last two lines aren't relevant for the race, right? I'd leave those out to not distract from it.

They did help me illustrate the two failure cases but yeah, someone who can read the code can derive the rest :)

Quoted 7 lines
Quoted 6 lines

* psi_group_change:

 *     record_times()
 *     smp_mb()
 *     if xchg(poll_scheduled, 1) == 0

* mod_timer()

Quoted 6 lines

The reason I tend to keep these more abstract is because 1) the names of the functions change (I had already sent out patches to rename half the variable and function names in this diagram), while the architecture (task change vs poll worker) likely won't, and 2) because it's easy to drown out what the reads, writes, and thus the race condition is with code details and function call indirections.

Got it.

Quoted 24 lines

How about a compromise?

/*
* A task change can race with the poll worker that is supposed to
* report on it. To avoid missing events, ensure ordering between
* poll_scheduled and the task state accesses, such that if the poll
* worker misses the state update, the task change is guaranteed to
* reschedule the poll worker:
*
* poll worker:

 *   atomic_set(poll_scheduled, 0)
 *   smp_mb()
 *   LOAD states

*
* task change:
* STORE states
* if atomic_xchg(poll_scheduled, 1) == 0:
* schedule poll worker
*
* The atomic_xchg() implies a full barrier.
*/
smp_mb();

Quoted 2 lines

This gives a high-level view of what's happening but it can still be mapped to the code by following the poll_scheduled variable.

This looks really good to me.
If you agree on the first comment modification, should I respin the next version?

Quoted 34 lines
Quoted 26 lines

If we remove smp_mb barriers then there are the following possible reordering cases:

Case1: reordering in psi_poll_work
psi_poll_work psi_group_change
changed_states = collect_percpu_times()
record_times() if xchg(poll_scheduled,
1) == 0 <-- false

                                                  mod_timer()
  poll_scheduled = 0
  if changed_states && xchg(poll_scheduled, 1) == 0 <-- changed_states is false
      mod_timer()

Case2: reordering in psi_group_change
psi_poll_work psi_group_change
if xchg(poll_scheduled,
1) == 0 <-- false

                                                  mod_timer()
  poll_scheduled = 0
  changed_states = collect_percpu_times()
                                                  record_times()
  if changed_states && xchg(poll_scheduled, 1) == 0 <-- changed_states is false
      mod_timer()
Quoted 3 lines

In both cases mod_timer() is not called, poll update is missed. But describing this all in the comments would be an overkill IMHO. WDYT?

Yeah, I also think that's overkill. The failure cases can be derived from the concurrency diagram and explanation.

Thanks

maintainer
#5 lore raw
Quoted 46 lines · Suren Baghdasaryan
Quoted 39 lines · Johannes Weiner
Quoted 31 lines · Suren Baghdasaryan
Quoted 15 lines · Johannes Weiner

This looks good to me now code wise. Just a comment on the comments:

Quoted 12 lines · Suren Baghdasaryan
kernel/sched/psi.c +3 -2
@@ -559,18 +560,14 @@ static u64 update_triggers(struct psi_group *group, u64 now)
      return now + group->poll_min_period;
 }
 
-/* Schedule polling if it's not already scheduled. */
-static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay)
+/* Schedule polling if it's not already scheduled or forced. */
+static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay,
+                                bool force)
 {
      struct task_struct *task;
 
Quoted 14 lines

kernel/sched/psi.c static u64 update_triggers(struct psi_group *group, u64 now) · line 571

Quoted 10 lines
kernel/sched/psi.c +2 -7
-     /*
-      * Do not reschedule if already scheduled.
-      * Possible race with a timer scheduled after this check but before
-      * mod_timer below can be tolerated because group->polling_next_update
-      * will keep updates on schedule.
-      */
-     if (timer_pending(&group->poll_timer))
+     /* xchg should be called even when !force to set poll_scheduled */
+     if (atomic_xchg(&group->poll_scheduled, 1) && !force)
              return;

This explains what the code does, but not why. It would be good to explain the ordering with poll_work, here or there. But both sides should mention each other.

Quoted 6 lines

How about this:

/*
* atomic_xchg should be called even when !force to always set poll_scheduled
* and to provide a memory barrier (see the comment inside psi_poll_work). */

Quoted 5 lines

The memory barrier part makes sense, but the first part says what the code does and the message is unclear to me. Are you worried somebody might turn this around in the future and only conditionalize on poll_scheduled when !force? Essentially, I don't see the downside of dropping that. But maybe I'm missing something.

Quoted 4 lines

Actually you are right. Originally I was worried that there might be a case when poll_scheduled==0 and force==true and if someone flips the conditions we will reschedule the timer but will not set poll_scheduled back to 1.

Oh I see.

Right, flipping the condition doesn't make sense because we need poll_scheduled to be set when we go ahead - whether we're forcing or not. I.e. if we were in a locked section, we'd write it like this:

	if (poll_scheduled)
		if (!force)
			return;
	else
		poll_scheduled = 1;
Quoted 9 lines

However I don't think this condition is possible. We set force=true only when we skipped resetting poll_schedule to 0 and on initial wakeup we always reset poll_schedule. How about changing the comment to this:

/*
* atomic_xchg should be called even when !force to provide a
* full memory barrier (see the comment inside psi_poll_work). */

Personally, I still find this more confusing than no comment on !force, because when you read it it sort of raises the question what the alternatives would be. And the alternatives appear to be nonsensical code rather than legitimate options.

But I won't insist if you prefer to leave it in. Your call.

Quoted 21 lines
Quoted 21 lines

/*
* A task change can race with the poll worker that is supposed to
* report on it. To avoid missing events, ensure ordering between
* poll_scheduled and the task state accesses, such that if the poll
* worker misses the state update, the task change is guaranteed to
* reschedule the poll worker:
*
* poll worker:

 *   atomic_set(poll_scheduled, 0)
 *   smp_mb()
 *   LOAD states

*
* task change:
* STORE states
* if atomic_xchg(poll_scheduled, 1) == 0:
* schedule poll worker
*
* The atomic_xchg() implies a full barrier.
*/
smp_mb();

Quoted 6 lines
Quoted 2 lines

This gives a high-level view of what's happening but it can still be mapped to the code by following the poll_scheduled variable.

This looks really good to me.
If you agree on the first comment modification, should I respin the next version?

Yeah, sounds good to me!

author maintainer
#6 lore raw
Quoted 75 lines · Johannes Weiner
Quoted 46 lines · Suren Baghdasaryan
Quoted 39 lines · Johannes Weiner
Quoted 31 lines · Suren Baghdasaryan
Quoted 15 lines · Johannes Weiner

This looks good to me now code wise. Just a comment on the comments:

Quoted 12 lines · Suren Baghdasaryan
kernel/sched/psi.c +3 -2
@@ -559,18 +560,14 @@ static u64 update_triggers(struct psi_group *group, u64 now)
      return now + group->poll_min_period;
 }
 
-/* Schedule polling if it's not already scheduled. */
-static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay)
+/* Schedule polling if it's not already scheduled or forced. */
+static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay,
+                                bool force)
 {
      struct task_struct *task;
 
Quoted 14 lines

kernel/sched/psi.c static u64 update_triggers(struct psi_group *group, u64 now) · line 571

Quoted 10 lines
kernel/sched/psi.c +2 -7
-     /*
-      * Do not reschedule if already scheduled.
-      * Possible race with a timer scheduled after this check but before
-      * mod_timer below can be tolerated because group->polling_next_update
-      * will keep updates on schedule.
-      */
-     if (timer_pending(&group->poll_timer))
+     /* xchg should be called even when !force to set poll_scheduled */
+     if (atomic_xchg(&group->poll_scheduled, 1) && !force)
              return;

This explains what the code does, but not why. It would be good to explain the ordering with poll_work, here or there. But both sides should mention each other.

Quoted 6 lines

How about this:

/*
* atomic_xchg should be called even when !force to always set poll_scheduled
* and to provide a memory barrier (see the comment inside psi_poll_work). */

Quoted 5 lines

The memory barrier part makes sense, but the first part says what the code does and the message is unclear to me. Are you worried somebody might turn this around in the future and only conditionalize on poll_scheduled when !force? Essentially, I don't see the downside of dropping that. But maybe I'm missing something.

Quoted 4 lines

Actually you are right. Originally I was worried that there might be a case when poll_scheduled==0 and force==true and if someone flips the conditions we will reschedule the timer but will not set poll_scheduled back to 1.

Oh I see.

Right, flipping the condition doesn't make sense because we need poll_scheduled to be set when we go ahead - whether we're forcing or not. I.e. if we were in a locked section, we'd write it like this:

        if (poll_scheduled)
                if (!force)
                        return;
        else
                poll_scheduled = 1;
Quoted 9 lines

However I don't think this condition is possible. We set force=true only when we skipped resetting poll_schedule to 0 and on initial wakeup we always reset poll_schedule. How about changing the comment to this:

/*
* atomic_xchg should be called even when !force to provide a
* full memory barrier (see the comment inside psi_poll_work). */

Quoted 6 lines

Personally, I still find this more confusing than no comment on !force, because when you read it it sort of raises the question what the alternatives would be. And the alternatives appear to be nonsensical code rather than legitimate options.

But I won't insist if you prefer to leave it in. Your call.

I would like to keep it as a precaution, if you don't mind. In case someone in the future thinks about "optimizing" this by flipping the condition, hopefully the comment will give them a pause to think about it :)

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

/*
* A task change can race with the poll worker that is supposed to
* report on it. To avoid missing events, ensure ordering between
* poll_scheduled and the task state accesses, such that if the poll
* worker misses the state update, the task change is guaranteed to
* reschedule the poll worker:
*
* poll worker:

 *   atomic_set(poll_scheduled, 0)
 *   smp_mb()
 *   LOAD states

*
* task change:
* STORE states
* if atomic_xchg(poll_scheduled, 1) == 0:
* schedule poll worker
*
* The atomic_xchg() implies a full barrier.
*/
smp_mb();

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

This gives a high-level view of what's happening but it can still be mapped to the code by following the poll_scheduled variable.

This looks really good to me.
If you agree on the first comment modification, should I respin the next version?

Yeah, sounds good to me!

Thanks! I'll post an update shortly.

author maintainer
#7 lore raw
Quoted 114 lines · Suren Baghdasaryan
Quoted 75 lines · Johannes Weiner
Quoted 46 lines · Suren Baghdasaryan
Quoted 39 lines · Johannes Weiner
Quoted 31 lines · Suren Baghdasaryan
Quoted 15 lines · Johannes Weiner

This looks good to me now code wise. Just a comment on the comments:

Quoted 12 lines · Suren Baghdasaryan
(quoted diff) +3 -2
@@ -559,18 +560,14 @@ static u64 update_triggers(struct psi_group *group, u64 now)
      return now + group->poll_min_period;
 }
 
-/* Schedule polling if it's not already scheduled. */
-static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay)
+/* Schedule polling if it's not already scheduled or forced. */
+static void psi_schedule_poll_work(struct psi_group *group, unsigned long delay,
+                                bool force)
 {
      struct task_struct *task;
 
Quoted 14 lines

Review static u64 update_triggers(struct psi_group *group, u64 now) · line 571

Quoted 10 lines
(quoted diff) +2 -7
-     /*
-      * Do not reschedule if already scheduled.
-      * Possible race with a timer scheduled after this check but before
-      * mod_timer below can be tolerated because group->polling_next_update
-      * will keep updates on schedule.
-      */
-     if (timer_pending(&group->poll_timer))
+     /* xchg should be called even when !force to set poll_scheduled */
+     if (atomic_xchg(&group->poll_scheduled, 1) && !force)
              return;

This explains what the code does, but not why. It would be good to explain the ordering with poll_work, here or there. But both sides should mention each other.

Quoted 6 lines

How about this:

/*
* atomic_xchg should be called even when !force to always set poll_scheduled
* and to provide a memory barrier (see the comment inside psi_poll_work). */

Quoted 5 lines

The memory barrier part makes sense, but the first part says what the code does and the message is unclear to me. Are you worried somebody might turn this around in the future and only conditionalize on poll_scheduled when !force? Essentially, I don't see the downside of dropping that. But maybe I'm missing something.

Quoted 4 lines

Actually you are right. Originally I was worried that there might be a case when poll_scheduled==0 and force==true and if someone flips the conditions we will reschedule the timer but will not set poll_scheduled back to 1.

Oh I see.

Right, flipping the condition doesn't make sense because we need poll_scheduled to be set when we go ahead - whether we're forcing or not. I.e. if we were in a locked section, we'd write it like this:

        if (poll_scheduled)
                if (!force)
                        return;
        else
                poll_scheduled = 1;
Quoted 9 lines

However I don't think this condition is possible. We set force=true only when we skipped resetting poll_schedule to 0 and on initial wakeup we always reset poll_schedule. How about changing the comment to this:

/*
* atomic_xchg should be called even when !force to provide a
* full memory barrier (see the comment inside psi_poll_work). */

Quoted 6 lines

Personally, I still find this more confusing than no comment on !force, because when you read it it sort of raises the question what the alternatives would be. And the alternatives appear to be nonsensical code rather than legitimate options.

But I won't insist if you prefer to leave it in. Your call.

I would like to keep it as a precaution, if you don't mind. In case someone in the future thinks about "optimizing" this by flipping the condition, hopefully the comment will give them a pause to think about it :)

Quoted 25 lines
Quoted 25 lines
Quoted 24 lines

/*
* A task change can race with the poll worker that is supposed to
* report on it. To avoid missing events, ensure ordering between
* poll_scheduled and the task state accesses, such that if the poll
* worker misses the state update, the task change is guaranteed to
* reschedule the poll worker:
*
* poll worker:

 *   atomic_set(poll_scheduled, 0)
 *   smp_mb()
 *   LOAD states

*
* task change:
* STORE states
* if atomic_xchg(poll_scheduled, 1) == 0:
* schedule poll worker
*
* The atomic_xchg() implies a full barrier.
*/
smp_mb();

This gives a high-level view of what's happening but it can still be mapped to the code by following the poll_scheduled variable.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

This looks really good to me.
If you agree on the first comment modification, should I respin the next version?

Yeah, sounds good to me!

Thanks! I'll post an update shortly.

v4 is posted at https://lore.kernel.org/patchwork/patch/1455172/