nohz: Fix racy sleeptime stats
Subsystems (3)
cover nohz: Fix racy sleeptime stats
Hi,
This is a resend to include Oleg in Cc, no modification since last version
https://lkml.org/lkml/2013/8/8/638.
Tree is still:
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
timers/nohz-sleeptime-fix
Thanks.
Frederic Weisbecker (4):
nohz: Only update sleeptime stats locally
nohz: Synchronize sleep time stats with seqlock
nohz: Consolidate sleep time stats read code
nohz: Convert a few places to use local per cpu accesses
include/linux/tick.h | 2 + kernel/time/tick-sched.c | 124 +++++++++++++++++++-------------------------- 2 files changed, 54 insertions(+), 72 deletions(-)
Signature · 1 line
1.7.5.4
PATCH 1/4 nohz: Only update sleeptime stats locally
The idle and io sleeptime stats can be updated concurrently from callers of get_cpu_idle_time_us(), get_cpu_iowait_time_us() and tick_nohz_stop_idle().
Updaters can easily race and mess up with internal datas coherency , for example when a governor calls a get_cpu_*_time_us() API and the target CPU exits idle at the same time, because no locking or whatsoever is there to protect against this concurrency.
To fix this, lets only update the sleeptime stats locally when the CPU exits from idle. This is the only place where a real update is truly needed. The callers of get_cpu_*_time_us() can simply add up the pending sleep time delta to the last sleeptime snapshot in order to get a coherent result. There is no need for them to also update the stats.
--- kernel/time/tick-sched.c | 65 +++++++++++++++------------------------------ 1 files changed, 22 insertions(+), 43 deletions(-)
kernel/time/tick-sched.c +22 -43
@@ -408,33 +408,18 @@ static void tick_nohz_update_jiffies(ktime_t now)touch_softlockup_watchdog (); } -/* -* Updates the per cpu time idle statistics counters -*/ -static void -update_ts_time_stats (int cpu,struct tick_sched *ts,ktime_t now,u64 *last_update_time) -{ -ktime_t delta; - -if (ts->idle_active) { - delta =ktime_sub (now, ts->idle_entrytime); -if (nr_iowait_cpu (cpu) >0 ) - ts->iowait_sleeptime =ktime_add (ts->iowait_sleeptime, delta); -else - ts->idle_sleeptime =ktime_add (ts->idle_sleeptime, delta); - ts->idle_entrytime = now; - } - -if (last_update_time) - *last_update_time =ktime_to_us (now); - -} -static void tick_nohz_stop_idle (int cpu,ktime_t now) {struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +ktime_t delta; -update_ts_time_stats (cpu, ts, now,NULL ); +/* Updates the per cpu time idle statistics counters */ + delta =ktime_sub (now, ts->idle_entrytime); +if (nr_iowait_cpu (cpu) >0 ) + ts->iowait_sleeptime =ktime_add (ts->iowait_sleeptime, delta); +else + ts->idle_sleeptime =ktime_add (ts->idle_sleeptime, delta); + ts->idle_entrytime = now; ts->idle_active =0 ;sched_clock_idle_wakeup_event (0 );
@@ -473,17 +458,14 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time)return -1 ; now =ktime_get (); -if (last_update_time) { -update_ts_time_stats (cpu, ts, now, last_update_time); - idle = ts->idle_sleeptime; - }else { -if (ts->idle_active && !nr_iowait_cpu (cpu)) { -ktime_t delta =ktime_sub (now, ts->idle_entrytime); +if (last_update_time) + *last_update_time =ktime_to_us (now); - idle =ktime_add (ts->idle_sleeptime, delta); - }else { - idle = ts->idle_sleeptime; - } +if (ts->idle_active && !nr_iowait_cpu (cpu)) { +ktime_t delta =ktime_sub (now, ts->idle_entrytime); + idle =ktime_add (ts->idle_sleeptime, delta); + }else { + idle = ts->idle_sleeptime; }return ktime_to_us (idle);
@@ -514,17 +496,14 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time)return -1 ; now =ktime_get (); -if (last_update_time) { -update_ts_time_stats (cpu, ts, now, last_update_time); - iowait = ts->iowait_sleeptime; - }else { -if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { -ktime_t delta =ktime_sub (now, ts->idle_entrytime); +if (last_update_time) + *last_update_time =ktime_to_us (now); - iowait =ktime_add (ts->iowait_sleeptime, delta); - }else { - iowait = ts->iowait_sleeptime; - } +if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +ktime_t delta =ktime_sub (now, ts->idle_entrytime); + iowait =ktime_add (ts->iowait_sleeptime, delta); + }else { + iowait = ts->iowait_sleeptime; }return ktime_to_us (iowait);
Signature · 1 line
1.7.5.4
Re: [PATCH 1/4] nohz: Only update sleeptime stats locally
Quoted 3 lines · Frederic Weisbecker
To fix this, lets only update the sleeptime stats locally when the CPU exits from idle.
I am in no position to ack the changes in this area, but I like this change very much. Because, as a code reader, I was totally confused by
if (last_update_time)
update_ts_time_stats()
code and it looks "obviously wrong".
I added more cc's. It seems to me that 9366d840 "cpufreq: governors:
Calculate iowait time only when necessary" doesn't realize what
- u64 idle_time = get_cpu_idle_time_us(cpu, NULL);
+ u64 idle_time = get_cpu_idle_time_us(cpu, io_busy ? wall : NULL);
actually means. OTOH, get_cpu_iowait_time_us() was called with last_update_time != NULL even before this patch...
In short. This looks like the clear fix to me, but I do not understand this code enough, and I think that cpufreq should know about this change.
Quoted 4 lines
static void
tick_nohz_stop_idle(int cpu, ktime_t now){kernel/time/tick-sched.c +1
-0struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +ktime_t delta;
kernel/time/tick-sched.c static void tick_nohz_update_jiffies(ktime_t now) · line 423
Quoted 10 lines
kernel/time/tick-sched.c +7
-1-update_ts_time_stats (cpu, ts, now,NULL ); +/* Updates the per cpu time idle statistics counters */ + delta =ktime_sub (now, ts->idle_entrytime); +if (nr_iowait_cpu (cpu) >0 ) + ts->iowait_sleeptime =ktime_add (ts->iowait_sleeptime, delta); +else + ts->idle_sleeptime =ktime_add (ts->idle_sleeptime, delta); + ts->idle_entrytime = now; ts->idle_active =0 ;
With or without this change, why we update ->idle_entrytime in this case? Looks harmless, but a bit confusing.
While this doesn't really matter, we could probably even kill ->idle_active and use !!ts->idle_entrytime instead.
Review u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time)
Quoted 1 line
(quoted diff) +0
-0@@ -473,17 +458,14 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time)
And I think that we should kill this last_update_time argument later.
Oleg.
Re: [PATCH 1/4] nohz: Only update sleeptime stats locally
Quoted 16 lines · Oleg Nesterov
Quoted 3 lines · Frederic Weisbecker
To fix this, lets only update the sleeptime stats locally when the CPU exits from idle.
I am in no position to ack the changes in this area, but I like this change very much. Because, as a code reader, I was totally confused by
if (
last_update_time)
update_ts_time_stats()code and it looks "obviously wrong".
I added more cc's. It seems to me that
9366d840"cpufreq: governors:
Calculate iowait time only when necessary" doesn't realize what
Quoted 8 lines
- u64
idle_time=get_cpu_idle_time_us(cpu, NULL);
+ u64idle_time=get_cpu_idle_time_us(cpu, io_busy ? wall : NULL);actually means. OTOH,
get_cpu_iowait_time_us()was called withlast_update_time!= NULL even before this patch...In short. This looks like the clear fix to me, but I do not understand this code enough, and I think that cpufreq should know about this change.
Good point, and this time I'm really adding the Cc :)
Quoted 6 lines
Quoted 6 lines
static void
tick_nohz_stop_idle(int cpu, ktime_t now){kernel/time/tick-sched.c +1
-0struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +ktime_t delta;
Quoted 12 lines
kernel/time/tick-sched.c static void tick_nohz_update_jiffies(ktime_t now) · line 423
Quoted 9 lines
kernel/time/tick-sched.c +7
-1-update_ts_time_stats (cpu, ts, now,NULL ); +/* Updates the per cpu time idle statistics counters */ + delta =ktime_sub (now, ts->idle_entrytime); +if (nr_iowait_cpu (cpu) >0 ) + ts->iowait_sleeptime =ktime_add (ts->iowait_sleeptime, delta); +else + ts->idle_sleeptime =ktime_add (ts->idle_sleeptime, delta); + ts->idle_entrytime = now; ts->idle_active =0 ;With or without this change, why we update ->idle_entrytime in this case? Looks harmless, but a bit confusing.
Oh indeed I missed that. It's a leftover from the copy-paste of update_ts_time_stats() content.
Well spotted, I'll fix.
Quoted 3 lines
While this doesn't really matter, we could probably even kill ->idle_active and use !!ts->idle_entrytime instead.
We could but it would be slightly more overhead in the irq entry path (cf: tick_check_nohz()) and it makes the code also a little bit harder to review IMHO.
Quoted 4 lines
Review u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time)
Quoted 2 lines
(quoted diff) +0
-0@@ -473,17 +458,14 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time)And I think that we should kill this
last_update_timeargument later.
Agreed.
Thanks.
Re: [PATCH 1/4] nohz: Only update sleeptime stats locally
In reply to Frederic WeisbeckerSorry for double post. forgot to cc cpufreq maintainers.
Quoted 3 lines · Frederic Weisbecker
To fix this, lets only update the sleeptime stats locally when the CPU exits from idle.
I am in no position to ack the changes in this area, but I like this change very much. Because, as a code reader, I was totally confused by
if (last_update_time)
update_ts_time_stats()
code and it looks "obviously wrong".
I added more cc's. It seems to me that 9366d840 "cpufreq: governors:
Calculate iowait time only when necessary" doesn't realize what
- u64 idle_time = get_cpu_idle_time_us(cpu, NULL);
+ u64 idle_time = get_cpu_idle_time_us(cpu, io_busy ? wall : NULL);
actually means. OTOH, get_cpu_iowait_time_us() was called with last_update_time != NULL even before this patch...
In short. This looks like the clear fix to me, but I do not understand this code enough, and I think that cpufreq should know about this change.
Quoted 4 lines
static void
tick_nohz_stop_idle(int cpu, ktime_t now){kernel/time/tick-sched.c +1
-0struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +ktime_t delta;
kernel/time/tick-sched.c static void tick_nohz_update_jiffies(ktime_t now) · line 423
Quoted 10 lines
kernel/time/tick-sched.c +7
-1-update_ts_time_stats (cpu, ts, now,NULL ); +/* Updates the per cpu time idle statistics counters */ + delta =ktime_sub (now, ts->idle_entrytime); +if (nr_iowait_cpu (cpu) >0 ) + ts->iowait_sleeptime =ktime_add (ts->iowait_sleeptime, delta); +else + ts->idle_sleeptime =ktime_add (ts->idle_sleeptime, delta); + ts->idle_entrytime = now; ts->idle_active =0 ;
With or without this change, why we update ->idle_entrytime in this case? Looks harmless, but a bit confusing.
While this doesn't really matter, we could probably even kill ->idle_active and use !!ts->idle_entrytime instead.
Review u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time)
Quoted 1 line
(quoted diff) +0
-0@@ -473,17 +458,14 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time)
And I think that we should kill this last_update_time argument later.
Oleg.
Re: [PATCH 1/4] nohz: Only update sleeptime stats locally
Quoted 15 lines · Oleg Nesterov
Sorry for double post. forgot to cc cpufreq maintainers.
Quoted 3 lines · Frederic Weisbecker
To fix this, lets only update the sleeptime stats locally when the CPU exits from idle.
I am in no position to ack the changes in this area, but I like this change very much. Because, as a code reader, I was totally confused by
if (
last_update_time)
update_ts_time_stats()code and it looks "obviously wrong".
Quoted 8 lines
I added more cc's. It seems to me that
9366d840"cpufreq: governors:
Calculate iowait time only when necessary" doesn't realize what- u64
idle_time=get_cpu_idle_time_us(cpu, NULL);
+ u64idle_time=get_cpu_idle_time_us(cpu, io_busy ? wall : NULL);actually means. OTOH,
get_cpu_iowait_time_us()was called withlast_update_time!= NULL even before this patch...
To be honest, I am unfamiliar with tick-sched code.
With patch 9366d840, I was trying to avoid duplicate calls to get_cpu_iowait_time_us function. I just saw that the original code was calling update_ts_time_stats within get_cpu_idle_time_us and get_cpu_iowait_time_us and I thought that I should keep calling these functions with non NULL parameter to update the time stats.
In fact the original patch submission was without this:
- u64 idle_time = get_cpu_idle_time_us(cpu, NULL);
+ u64 idle_time = get_cpu_idle_time_us(cpu, io_busy ? wall : NULL);
and the idle time calculation was wrong (ondemand couldn't increase to max freq)
For your convenience the call paths before and after this patch:
Before patch
get_cpu_idle_time(j, &cur_wall_time); u64 idle_time = get_cpu_idle_time_us(cpu, NULL); idle_time += get_cpu_iowait_time_us(cpu, wall); update_ts_time_stats(cpu, ts, now, last_update_time); ... get_cpu_iowait_time_us(j, &cur_wall_time); update_ts_time_stats(cpu, ts, now, last_update_time);
After patch (io_busy = 1)
cur_idle_time = get_cpu_idle_time(j, &cur_wall_time, io_busy); u64 idle_time = get_cpu_idle_time_us(cpu, io_busy ? wall : NULL); update_ts_time_stats(cpu, ts, now, last_update_time);
After patch (io_busy = 0)
cur_idle_time = get_cpu_idle_time(j, &cur_wall_time, io_busy); u64 idle_time = get_cpu_idle_time_us(cpu, io_busy ? wall : NULL); idle_time += get_cpu_iowait_time_us(cpu, wall); update_ts_time_stats(cpu, ts, now, last_update_time);
Regards,
Stratos
PATCH 2/4 nohz: Synchronize sleep time stats with seqlock
In reply to Frederic WeisbeckerWhen some call site uses get_cpu_*_time_us() to read a sleeptime stat, it deduces the total sleeptime by adding the pending time to the last sleeptime snapshot if the CPU target is idle.
Namely this sums up to:
sleeptime = ts($CPU)->idle_sleeptime;
if (ts($CPU)->idle_active)
sleeptime += NOW() - ts($CPU)->idle_entrytimeBut this only works if idle_sleeptime, idle_entrytime and idle_active are read and updated under some disciplined order.
Lets consider the following scenario:
CPU 0 CPU 1
(seq 1) ts(CPU 0)->idle_active = 1
ts(CPU 0)->idle_entrytime = NOW()
(seq 2) sleeptime = NOW() - ts(CPU 0)->idle_entrytime
ts(CPU 0)->idle_sleeptime += sleeptime sleeptime = ts(CPU 0)->idle_sleeptime;
if (ts(CPU 0)->idle_active)
ts(CPU 0)->idle_entrytime = NOW() sleeptime += NOW() - ts(CPU 0)->idle_entrytimeThe resulting value of sleeptime in CPU 1 can vary depending of some ordering scenario:
* If it sees the value of idle_entrytime after seq 1 and the value of idle_sleeptime after seq 2, the value of sleeptime will be buggy because it accounts the delta twice, so it will be too high.
* If it sees the value of idle_entrytime after seq 2 and the value of idle_sleeptime after seq 1, the value of sleeptime will be buggy because it misses the delta, so it will be too low.
* If it sees the value of idle_entrytime and idle_sleeptime, both as seen after seq 1 or 2, the value will be correct.
Some more tricky scenario can also happen if idle_active value is read from a former sequence.
Hence we must honour the following constraints:
- idle_sleeptime, idle_active and idle_entrytime must be updated and read under some correctly enforced SMP ordering
- The three variable values as read by CPU 1 must belong to the same update sequences from CPU 0. The update sequences must be delimited such that the resulting three values after a sequence completion produce a coherent result together when read from the CPU 1.
- We need to prevent from fetching middle-state sequence values.
The ideal solution to implement this synchronization is to use a seqcount. Lets use one here around these three values to enforce sequence synchronization between updates and read.
This fixes a reported bug where non-monotonic sleeptime stats are returned by /proc/stat when it is frequently read. And potential cpufreq governor bugs.
--- include/linux/tick.h | 2 ++ kernel/time/tick-sched.c | 37 +++++++++++++++++++++++++------------ 2 files changed, 27 insertions(+), 12 deletions(-)
include/linux/tick.h +2 -0
@@ -10,6 +10,7 @@#include <linux/irqflags.h> #include <linux/percpu.h> #include <linux/hrtimer.h> +#include <linux/seqlock.h> #ifdef CONFIG_GENERIC_CLOCKEVENTS
@@ -70,6 +71,7 @@ struct tick_sched {unsigned long next_jiffies;ktime_t idle_expires;int do_timer_last; +seqcount_t sleeptime_seq; };extern void __init tick_init (void );
kernel/time/tick-sched.c +25 -12
@@ -414,6 +414,7 @@ static void tick_nohz_stop_idle(int cpu, ktime_t now)ktime_t delta;/* Updates the per cpu time idle statistics counters */ +write_seqcount_begin (&ts->sleeptime_seq); delta =ktime_sub (now, ts->idle_entrytime);if (nr_iowait_cpu (cpu) >0 ) ts->iowait_sleeptime =ktime_add (ts->iowait_sleeptime, delta);
@@ -421,6 +422,7 @@ static void tick_nohz_stop_idle(int cpu, ktime_t now) ts->idle_sleeptime =ktime_add (ts->idle_sleeptime, delta); ts->idle_entrytime = now; ts->idle_active =0 ; +write_seqcount_end (&ts->sleeptime_seq);sched_clock_idle_wakeup_event (0 ); }
@@ -429,8 +431,11 @@ static ktime_t tick_nohz_start_idle(int cpu, struct tick_sched *ts) {ktime_t now =ktime_get (); +write_seqcount_begin (&ts->sleeptime_seq); ts->idle_entrytime = now; ts->idle_active =1 ; +write_seqcount_end (&ts->sleeptime_seq); +sched_clock_idle_sleep_event ();return now; }
@@ -453,6 +458,7 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time) {struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu);ktime_t now, idle; +unsigned int seq;if (!tick_nohz_enabled)return -1 ;
@@ -461,12 +467,15 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time)if (last_update_time) *last_update_time =ktime_to_us (now); -if (ts->idle_active && !nr_iowait_cpu (cpu)) { -ktime_t delta =ktime_sub (now, ts->idle_entrytime); - idle =ktime_add (ts->idle_sleeptime, delta); - }else { - idle = ts->idle_sleeptime; - } +do { + seq =read_seqcount_begin (&ts->sleeptime_seq); +if (ts->idle_active && !nr_iowait_cpu (cpu)) { +ktime_t delta =ktime_sub (now, ts->idle_entrytime); + idle =ktime_add (ts->idle_sleeptime, delta); + }else { + idle = ts->idle_sleeptime; + } + }while (read_seqcount_retry (&ts->sleeptime_seq, seq));return ktime_to_us (idle);
@@ -491,6 +500,7 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) {struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu);ktime_t now, iowait; +unsigned int seq;if (!tick_nohz_enabled)return -1 ;
@@ -499,12 +509,15 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time)if (last_update_time) *last_update_time =ktime_to_us (now); -if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { -ktime_t delta =ktime_sub (now, ts->idle_entrytime); - iowait =ktime_add (ts->iowait_sleeptime, delta); - }else { - iowait = ts->iowait_sleeptime; - } +do { + seq =read_seqcount_begin (&ts->sleeptime_seq); +if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +ktime_t delta =ktime_sub (now, ts->idle_entrytime); + iowait =ktime_add (ts->iowait_sleeptime, delta); + }else { + iowait = ts->iowait_sleeptime; + } + }while (read_seqcount_retry (&ts->sleeptime_seq, seq));return ktime_to_us (iowait); }
Signature · 1 line
1.7.5.4
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Thanks Frederic!
I'll try to read this series carefully later. Not that I think I can help, you certainly understand this much better.
Just one question below,
Quoted 10 lines · Frederic Weisbecker
kernel/time/tick-sched.c +0
-5@@ -499,12 +509,15 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time)if (last_update_time) *last_update_time =ktime_to_us (now); -if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { -ktime_t delta =ktime_sub (now, ts->idle_entrytime); - iowait =ktime_add (ts->iowait_sleeptime, delta); - }else { - iowait = ts->iowait_sleeptime;
kernel/time/tick-sched.c u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) · line 520
Quoted 10 lines
kernel/time/tick-sched.c +9
-1- } +do { + seq =read_seqcount_begin (&ts->sleeptime_seq); +if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +ktime_t delta =ktime_sub (now, ts->idle_entrytime); + iowait =ktime_add (ts->iowait_sleeptime, delta); + }else { + iowait = ts->iowait_sleeptime; + } + }while (read_seqcount_retry (&ts->sleeptime_seq, seq));
Unless I missread this patch, this is still racy a bit.
Suppose it is called on CPU_0 and cpu == 1. Suppose that ts->idle_active == T and nr_iowait_cpu(cpu) == 1.
So we return iowait_sleeptime + delta.
Suppose that we call get_cpu_iowait_time_us() again. By this time the task which incremented ->nr_iowait can be woken up on another CPU, and it can do atomic_dec(rq->nr_iowait). So the next time we return iowait_sleeptime, and this is not monotonic again.
No?
Oleg.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 34 lines · Oleg Nesterov
Thanks Frederic!
I'll try to read this series carefully later. Not that I think I can help, you certainly understand this much better.
Just one question below,
Quoted 10 lines · Frederic Weisbecker
kernel/time/tick-sched.c +0
-5@@ -499,12 +509,15 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time)if (last_update_time) *last_update_time =ktime_to_us (now); -if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { -ktime_t delta =ktime_sub (now, ts->idle_entrytime); - iowait =ktime_add (ts->iowait_sleeptime, delta); - }else { - iowait = ts->iowait_sleeptime;kernel/time/tick-sched.c u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) · line 520
Quoted 10 lines
kernel/time/tick-sched.c +9
-1- } +do { + seq =read_seqcount_begin (&ts->sleeptime_seq); +if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +ktime_t delta =ktime_sub (now, ts->idle_entrytime); + iowait =ktime_add (ts->iowait_sleeptime, delta); + }else { + iowait = ts->iowait_sleeptime; + } + }while (read_seqcount_retry (&ts->sleeptime_seq, seq));Unless I missread this patch, this is still racy a bit.
Suppose it is called on
CPU_0and cpu == 1. Suppose thatts->idle_active== T andnr_iowait_cpu(cpu)== 1.
Quoted 6 lines
So we return
iowait_sleeptime+ delta.Suppose that we call
get_cpu_iowait_time_us()again. By this time the task which incremented ->nr_iowait can be woken up on another CPU, and it can doatomic_dec(rq->nr_iowait). So the next time we returniowait_sleeptime, and this is not monotonic again.
Hmm, by the time it decrements nr_iowait, it returned from schedule() and so idle had flushed the pending iowait sleeptime.
May be you have some scenario in mind that I'm missing?
Quoted 5 lines
No?
Oleg.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 19 lines · Frederic Weisbecker
Quoted 17 lines · Oleg Nesterov
kernel/time/tick-sched.c u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) · line 520
Quoted 9 lines
kernel/time/tick-sched.c +9
-0+do { + seq =read_seqcount_begin (&ts->sleeptime_seq); +if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +ktime_t delta =ktime_sub (now, ts->idle_entrytime); + iowait =ktime_add (ts->iowait_sleeptime, delta); + }else { + iowait = ts->iowait_sleeptime; + } + }while (read_seqcount_retry (&ts->sleeptime_seq, seq));Unless I missread this patch, this is still racy a bit.
Suppose it is called on
CPU_0and cpu == 1. Suppose thatts->idle_active== T andnr_iowait_cpu(cpu)== 1.So we return
iowait_sleeptime+ delta.
Quoted 7 lines
Quoted 4 lines
Suppose that we call
get_cpu_iowait_time_us()again. By this time the task which incremented ->nr_iowait can be woken up on another CPU, and it can doatomic_dec(rq->nr_iowait). So the next time we returniowait_sleeptime, and this is not monotonic again.Hmm, by the time it decrements
nr_iowait, it returned fromschedule()and so idle had flushed the pending iowait sleeptime.
Suppose a task does io_schedule() on CPU_0, and increments the counter. This CPU becomes idle and nr_iowait_cpu(0) == 1.
Then this task is woken up, but try_to_wake_up() selects another CPU != 0.
It returns from schedule() and decrements the same counter, it doesn't do raw_rq/etc again. nr_iowait_cpu(0) becomes 0.
In fact the task can even migrate to another CPU right after raw_rq().
Quoted 1 line
May be you have some scenario in mind that I'm missing?
Or me...
Oleg.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 31 lines · Oleg Nesterov
Quoted 19 lines · Frederic Weisbecker
Quoted 17 lines · Oleg Nesterov
kernel/time/tick-sched.c u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) · line 520
Quoted 9 lines
kernel/time/tick-sched.c +9
-0+do { + seq =read_seqcount_begin (&ts->sleeptime_seq); +if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +ktime_t delta =ktime_sub (now, ts->idle_entrytime); + iowait =ktime_add (ts->iowait_sleeptime, delta); + }else { + iowait = ts->iowait_sleeptime; + } + }while (read_seqcount_retry (&ts->sleeptime_seq, seq));Unless I missread this patch, this is still racy a bit.
Suppose it is called on
CPU_0and cpu == 1. Suppose thatts->idle_active== T andnr_iowait_cpu(cpu)== 1.So we return
iowait_sleeptime+ delta.Quoted 7 lines
Quoted 4 lines
Suppose that we call
get_cpu_iowait_time_us()again. By this time the task which incremented ->nr_iowait can be woken up on another CPU, and it can doatomic_dec(rq->nr_iowait). So the next time we returniowait_sleeptime, and this is not monotonic again.Hmm, by the time it decrements
nr_iowait, it returned fromschedule()and so idle had flushed the pending iowait sleeptime.Suppose a task does
io_schedule()onCPU_0, and increments the counter. This CPU becomes idle andnr_iowait_cpu(0)== 1.
Quoted 6 lines
Then this task is woken up, but
try_to_wake_up()selects another CPU != 0.It returns from
schedule()and decrements the same counter, it doesn't do raw_rq/etc again.nr_iowait_cpu(0)becomes 0.In fact the task can even migrate to another CPU right after
raw_rq().
Ah I see now. So that indeed yet another race.
Should we flush that iowait to the src CPU? But then it means we must handle concurrent updates to iowait_sleeptime, idle_sleeptime from the migration code and from idle enter / exit.
So I fear we need a seqlock.
Or we can live with that and still account the whole idle time slept until tick_nohz_stop_idle() to iowait if we called tick_nohz_start_idle() with nr_iowait > 0. All we need is just a new field in ts-> that records on which state we entered idle.
What do you think?
Ingo, Thomas?
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 4 lines · Frederic Weisbecker
tick_nohz_stop_idle()to iowait if we calledtick_nohz_start_idle()withnr_iowait> 0. All we need is just a new field in ts-> that records on which state we entered idle.
Or we can turn ->idle_active into enum. And all other nr_iowait_cpu's in this code should go away.
Personally I am fine either way.
Oleg.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 10 lines · Oleg Nesterov
Quoted 4 lines · Frederic Weisbecker
tick_nohz_stop_idle()to iowait if we calledtick_nohz_start_idle()withnr_iowait> 0. All we need is just a new field in ts-> that records on which state we entered idle.Or we can turn ->idle_active into enum. And all other nr_iowait_cpu's in this code should go away.
Personally I am fine either way.
Me too.
So my proposition is that we can keep the existing patches as they fix other distinct races (and we add fixes on what peterz just reported) and send them to Ingo. Ah and I'll wait for your review first.
Then if all goes well on the pull request we describe him the nr_iowait race and we let him choose what to do with that nr_iowait migration race: either we ignore the migration and always account to what we saw on idle start, or we flush that time accounting on iowait migration, but that requires seqlocks on the idle path.
Or may be Peter could tell us as well. Peter, do you have a preference?
Thanks.
Quoted 3 lines
Oleg.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 8 lines · Frederic Weisbecker
Quoted 2 lines · Oleg Nesterov
Personally I am fine either way.
Me too.
So my proposition is that we can keep the existing patches as they fix other distinct races
perhaps... but it would be nice to fix the "goes backward" problem.
This "race" is not theoretical, at least for get_cpu_iowait_time_us(). nr_iowait_cpu() can change from !0 to 0 very easily.
And just in case, note that get_cpu_idle_time_us() has the same problem too, because it can also change from 0 to !0 although this case is much less likely.
However, right now I do not see a simple solution. It seems that, if get_cpu_idle_time_us() does ktime_add(ts->idle_sleeptime) it should actually update ts->idle_sleeptime/entrytime to avoid these races (the same for ->idle_sleeptime), but then we need the locking.
Quoted 1 line
(and we add fixes on what peterz just reported)
Do you mean his comments about 4/4 or I missed something else?
Quoted 2 lines
Ah and I'll wait for
your review first.
If only I could understand this code ;) It looks really simple and I hope I can understand what it does. But not why. I simply can't understand why idle/iowait are "mutually exclusive".
And if we do this, then perhaps io_schedule() should do "if (atomic_dec(&rq->nr_iowait)) update_iowait_sleeptime()" but this means the locking again.
Quoted 2 lines
Then if all goes well on the pull request we describe him the
nr_iowaitrace and we let him choose what to do with thatnr_iowaitmigration race:
OK, agreed.
Quoted 1 line
Or may be Peter could tell us as well. Peter, do you have a preference?
Yes, it would be nice to know what Peter thinks ;)
Oleg.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 6 lines · Oleg Nesterov
Quoted 5 lines · Frederic Weisbecker
Quoted 2 lines · Oleg Nesterov
Personally I am fine either way.
Quoted 8 lines
Quoted 3 lines
Me too.
So my proposition is that we can keep the existing patches as they fix other distinct races
perhaps... but it would be nice to fix the "goes backward" problem.
This "race" is not theoretical, at least for
get_cpu_iowait_time_us().nr_iowait_cpu()can change from !0 to 0 very easily.
Actually yes. Now I think we should fix the iowait race in the same changeset because the bugs I'm fixing in this patchset were probably lowering the iowait issue in some case.
Quoted 4 lines
And just in case, note that
get_cpu_idle_time_us()has the same problem too, because it can also change from 0 to !0 although this case is much less likely.
Right.
Quoted 5 lines
However, right now I do not see a simple solution. It seems that, if
get_cpu_idle_time_us()doesktime_add(ts->idle_sleeptime)it should actually update ts->idle_sleeptime/entrytime to avoid these races (the same for ->idle_sleeptime), but then we need the locking.
It seems that wouldn't solve the issue. Imagine that task A waits for IO on CPU 0. It waits 10 seconds. Then it's finally woken on CPU 1. A further call to get_cpu_idle_time_us() on CPU 0, say 5 seconds later, will miss the io sleeptime part.
For now I can't figure out how to avoid flushing the io sleeptime when the iowait task is moved to another CPU. This requires a lock (moving from seqcount to seqlock) and may be involve quite some cache issues in the idle path, resulting in higher latencies on wakeup. We really need another solution.
Quoted 4 lines
Quoted 2 lines
(and we add fixes on what peterz just reported)
Do you mean his comments about 4/4 or I missed something else?
Yep, just removing the cpu arguments from the functions that only use ts locally.
Quoted 6 lines
Quoted 2 lines
Ah and I'll wait for
your review first.If only I could understand this code ;) It looks really simple and I hope I can understand what it does. But not why. I simply can't understand why idle/iowait are "mutually exclusive".
I believe this can be changed such that iowait is included in the idle sleeptime. We can do that if that's needed.
Quoted 4 lines
And if we do this, then perhaps
io_schedule()should do "if (atomic_dec(&rq->nr_iowait))update_iowait_sleeptime()" but this means the locking again.
Yep.
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Frederic WeisbeckerQuoted 1 line · Frederic Weisbecker
Or may be Peter could tell us as well. Peter, do you have a preference?
Still trying to wrap my head around it, but conceptually get_cpu_iowait_time_us() doesn't make any kind of sense. iowait isn't per cpu since effectively tasks that aren't running aren't assigned a cpu (as Oleg already pointed out).
The fact that cpufreq 'needs' this just means that cpufreq is broken -- but I think I've said as much previously; cpufreq needs to stop living in the partitioned-mp era and get dragged (kicking and screaming) into the smp era.
I'm also not entirely clear on the 'desired' semantics here. Do we count iowait time as idle or not?
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 2 lines · Peter Zijlstra
I'm also not entirely clear on the 'desired' semantics here. Do we count iowait time as idle or not?
cpufreq only looks at this because it does not want to count this as idle... other than that it could not care less.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Peter ZijlstraQuoted 8 lines · Peter Zijlstra
Quoted 1 line · Frederic Weisbecker
Or may be Peter could tell us as well. Peter, do you have a preference?
Still trying to wrap my head around it, but conceptually
get_cpu_iowait_time_us()doesn't make any kind of sense. iowait isn't per cpu since effectively tasks that aren't running aren't assigned a cpu (as Oleg already pointed out).
Quoted 7 lines
The fact that cpufreq 'needs' this just means that cpufreq is broken -- but I think I've said as much previously; cpufreq needs to stop living in the partitioned-mp era and get dragged (kicking and screaming) into the smp era.
I'm also not entirely clear on the 'desired' semantics here. Do we count iowait time as idle or not?
fwiw only some Intel cpu's use this, and those by and large no longer use cpufreq so this can just go away as far as I am concerned.
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Frederic WeisbeckerOn Fri, Aug 16, 2013 at 06:46:28PM +0200, Frederic Weisbecker wrote:
Option A:
Quoted 5 lines
Should we flush that iowait to the src CPU? But then it means we must handle concurrent updates to
iowait_sleeptime,idle_sleeptimefrom the migration code and from idle enter / exit.So I fear we need a seqlock.
Option B:
Quoted 6 lines
Or we can live with that and still account the whole idle time slept until
tick_nohz_stop_idle()to iowait if we calledtick_nohz_start_idle()withnr_iowait> 0. All we need is just a new field in ts-> that records on which state we entered idle.What do you think?
I think option B is unworkable. Afaict it could basically caused unlimited iowait time. Suppose we have a load-balancer that tries it bestestest to sort-left (ie. run a task on the lowest 'free' cpu possible) -- the power aware folks are pondering such schemes.
Now suppose we have a small burst of activity and the rightmost cpu gets to run something that goes to sleep on iowait.
We'd accrue iowait on that cpu until it wakes up, which could be days from now if the load stays low enough, even though the task got to run almost instantly on another cpu.
So no, if we need per-cpu iowait time we have to do A.
Since we already have atomics in the io_schedule*() paths, please replace those with (seq)locks. Also see if you can place the entire iowait accounting thing in a separate cacheline.
That said, I'm still not sure if iowait time counts as both idle and iowait or only iowait.
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 5 lines · Peter Zijlstra
So no, if we need per-cpu iowait time we have to do A.
Since we already have atomics in the io_schedule*() paths, please replace those with (seq)locks. Also see if you can place the entire iowait accounting thing in a separate cacheline.
Also, it completely blows to have these extra atomics in the IO paths, I'm sure that the Mega-IOP/s people would be much pleased if we could remove these.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Peter ZijlstraQuoted 29 lines · Peter Zijlstra
On Fri, Aug 16, 2013 at 06:46:28PM +0200, Frederic Weisbecker wrote:
Option A:
Quoted 5 lines
Should we flush that iowait to the src CPU? But then it means we must handle concurrent updates to
iowait_sleeptime,idle_sleeptimefrom the migration code and from idle enter / exit.So I fear we need a seqlock.
Option B:
Quoted 6 lines
Or we can live with that and still account the whole idle time slept until
tick_nohz_stop_idle()to iowait if we calledtick_nohz_start_idle()withnr_iowait> 0. All we need is just a new field in ts-> that records on which state we entered idle.What do you think?
I think option B is unworkable. Afaict it could basically caused unlimited iowait time. Suppose we have a load-balancer that tries it bestestest to sort-left (ie. run a task on the lowest 'free' cpu possible) -- the power aware folks are pondering such schemes.
Now suppose we have a small burst of activity and the rightmost cpu gets to run something that goes to sleep on iowait.
We'd accrue iowait on that cpu until it wakes up, which could be days from now if the load stays low enough, even though the task got to run almost instantly on another cpu.
Quoted 5 lines
So no, if we need per-cpu iowait time we have to do A.
Since we already have atomics in the io_schedule*() paths, please replace those with (seq)locks. Also see if you can place the entire iowait accounting thing in a separate cacheline.
I considered option A for a while but, fearing it would be considered overkill, took a different approach: create a shadow copy of ->iowait_sleeptime that is always kept monotonic (artificially in some cases) and use that to compute the values exported through /proc.
That said, if deemed acceptable, option A is the one I would choose.
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 2 lines · Fernando Luis Vázquez Cao
That said, if deemed acceptable, option A is the one I would choose.
Right, so I think we can do A without much extra cost mostly because we already have 2 atomics in the io_schedule() path. If we replace those two atomic operations with locks and modify nr_iowait and the other stats under the same lock, and ensure all those variables (including the lock) live in the same cacheline we should have the same cost we have now.
Of course, if we can get away with completely removing all of that (which I think Arjan suggested was a real possibility) then that would be ever so much better still :-)
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 10 lines · Peter Zijlstra
Quoted 2 lines · Fernando Luis Vázquez Cao
That said, if deemed acceptable, option A is the one I would choose.
Right, so I think we can do A without much extra cost mostly because we already have 2 atomics in the
io_schedule()path. If we replace those two atomic operations with locks and modifynr_iowaitand the other stats under the same lock, and ensure all those variables (including the lock) live in the same cacheline we should have the same cost we have now.
I can try that :-)
Quoted 4 lines
Of course, if we can get away with completely removing all of that (which I think Arjan suggested was a real possibility) then that would be ever so much better still :-)
Would be lovely. But I don't know much about cpufreq, I hope somebody who's familiar with that code can handle this. Then once there are no more users of get_cpu_iowait_sleep_time() I can simply zap and clean the tick/time related code.
Surely the overhead that this mess brings to io_schedule() (and it's going to be worth with a seqlock, whether in the same cacheline than nr_iowait or not) may be a good motivation to poke at that cpufreq code part.
Thanks.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 9 lines · Frederic Weisbecker
Quoted 4 lines
Of course, if we can get away with completely removing all of that (which I think Arjan suggested was a real possibility) then that would be ever so much better still :-)
Would be lovely. But I don't know much about cpufreq, I hope somebody who's familiar with that code can handle this. Then once there are no more users of
get_cpu_iowait_sleep_time()I can simply zap and clean the tick/time related code.
it's just doing the "idle = 100 - busy% - iowait%" calculation. (with the later part only for Intel cpus iirc)
in a perfect world the scheduler would be doing that calculation in the first place ;-)
removing the later part will impact performance some on specific workloads, but most Intel cpus that this applies to should not be using cpufreq anymore anyway.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 12 lines · Arjan van de Ven
Quoted 9 lines · Frederic Weisbecker
Quoted 4 lines
Of course, if we can get away with completely removing all of that (which I think Arjan suggested was a real possibility) then that would be ever so much better still :-)
Would be lovely. But I don't know much about cpufreq, I hope somebody who's familiar with that code can handle this. Then once there are no more users of
get_cpu_iowait_sleep_time()I can simply zap and clean the tick/time related code.
Quoted 8 lines
it's just doing the "idle = 100 - busy% - iowait%" calculation. (with the later part only for Intel cpus iirc)
in a perfect world the scheduler would be doing that calculation in the first place ;-)
removing the later part will impact performance some on specific workloads, but most Intel cpus that this applies to should not be using cpufreq anymore anyway.
Are there other users than intel?
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 16 lines · Frederic Weisbecker
Quoted 15 lines · Arjan van de Ven
Quoted 9 lines · Frederic Weisbecker
Quoted 4 lines
Of course, if we can get away with completely removing all of that (which I think Arjan suggested was a real possibility) then that would be ever so much better still :-)
Would be lovely. But I don't know much about cpufreq, I hope somebody who's familiar with that code can handle this. Then once there are no more users of
get_cpu_iowait_sleep_time()I can simply zap and clean the tick/time related code.it's just doing the "idle = 100 - busy% - iowait%" calculation. (with the later part only for Intel cpus iirc)
Quoted 8 lines
Quoted 5 lines
in a perfect world the scheduler would be doing that calculation in the first place ;-)
removing the later part will impact performance some on specific workloads, but most Intel cpus that this applies to should not be using cpufreq anymore anyway.
Are there other users than intel?
http://lxr.linux.no/linux+v3.10.7/drivers/cpufreq/cpufreq_ondemand.c#L69
nope
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Peter ZijlstraQuoted 11 lines · Peter Zijlstra
Quoted 2 lines · Fernando Luis Vázquez Cao
That said, if deemed acceptable, option A is the one I would choose.
Right, so I think we can do A without much extra cost mostly because we already have 2 atomics in the
io_schedule()path. If we replace those two atomic operations with locks and modifynr_iowaitand the other stats under the same lock, and ensure all those variables (including the lock) live in the same cacheline we should have the same cost we have now.
Quoted 3 lines
Of course, if we can get away with completely removing all of that (which I think Arjan suggested was a real possibility) then that would be ever so much better still :-)
I'm quite ok with removing that.
however note that "top" also reports per cpu iowait... and that's a userspace expectation
Quoted 1 line
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 10 lines · Arjan van de Ven
Quoted 3 lines · Peter Zijlstra
Of course, if we can get away with completely removing all of that (which I think Arjan suggested was a real possibility) then that would be ever so much better still :-)
I'm quite ok with removing that.
however note that "top" also reports per cpu iowait... and that's a userspace expectation
Right, broken as that maybe :/ OK that looks like CPUTIME_IOWAIT which is tick based and not the ns based accounting.
Still it needs the per-cpu nr_iowait accounting which pretty much requires the atomics so no big gains there.
Which means that if Frederic can make the ns thing as expensive as the existing atomics we might as well keep the ns thing too.
Hmm, would something like the below make sense? I suppose this can be done even for the ns case, you'd have to duplicate all stats though.
At the very least the below reduces the number of atomics, not entirely sure it all matters much though, some benchmarking would be in order I suppose.
---
kernel/sched/core.c +34 -11
@@ -2167,8 +2167,12 @@ unsigned long nr_iowait(void) {unsigned long i, sum =0 ; -for_each_possible_cpu (i) - sum +=atomic_read (&cpu_rq (i)->nr_iowait); +for_each_possible_cpu (i) { +struct rq *rq =cpu_rq (i); + + sum += rq->nr_iowait_local; + sum +=atomic_read (&rq->nr_iowait_remote); + }return sum; }
@@ -2176,7 +2180,7 @@ unsigned long nr_iowait(void)unsigned long nr_iowait_cpu (int cpu) {struct rq *this =cpu_rq (cpu); -return atomic_read (&this->nr_iowait); +return atomic_read (&this->nr_iowait_remote) + this->nr_iowait_local; }#ifdef CONFIG_SMP
@@ -4086,31 +4090,49 @@ EXPORT_SYMBOL_GPL(yield_to);*/ void __sched io_schedule(void) { -struct rq *rq =raw_rq (); +struct rq *rq;delayacct_blkio_start (); -atomic_inc (&rq->nr_iowait);blk_flush_plug (current); + +preempt_disable (); + rq =this_rq (); + rq->nr_iowait_local++; current->in_iowait =1 ; -schedule (); +schedule_preempt_disabled (); current->in_iowait =0 ; -atomic_dec (&rq->nr_iowait); +if (likely (task_cpu (current) ==cpu_of (rq))) + rq->nr_iowait_local--; +else +atomic_dec (&rq->nr_iowait_remote); +preempt_enable (); +delayacct_blkio_end (); }EXPORT_SYMBOL (io_schedule);long __sched io_schedule_timeout (long timeout) { -struct rq *rq =raw_rq (); +struct rq *rq;long ret;delayacct_blkio_start (); -atomic_inc (&rq->nr_iowait);blk_flush_plug (current); + +preempt_disable (); + rq =this_rq (); + rq->nr_iowait_local++; current->in_iowait =1 ; +preempt_enable_no_resched (); ret =schedule_timeout (timeout); +preempt_disable (); current->in_iowait =0 ; -atomic_dec (&rq->nr_iowait); +if (likely (task_cpu (current) ==cpu_of (rq))) + rq->nr_iowait_local--; +else +atomic_dec (&rq->nr_iowait_remote); +preempt_enable (); +delayacct_blkio_end ();return ret; }
@@ -6650,7 +6672,8 @@ void __init sched_init(void)#endif #endif init_rq_hrtick (rq); -atomic_set (&rq->nr_iowait,0 ); + rq->nr_iowait_local =0 ; +atomic_set (&rq->nr_iowait_remote,0 ); }set_load_weight (&init_task);
kernel/sched/sched.h +2 -1
@@ -453,7 +453,8 @@ struct rq {u64 clock;u64 clock_task; -atomic_t nr_iowait; +int nr_iowait_local; +atomic_t nr_iowait_remote;#ifdef CONFIG_SMP struct root_domain *rd;
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
kernel/sched/sched.h struct rq { · line 457
Quoted 10 lines · Peter Zijlstra
kernel/sched/sched.h +2
-1@@ -453,7 +453,8 @@ struct rq {u64 clock;u64 clock_task; -atomic_t nr_iowait; +int nr_iowait_local; +atomic_t nr_iowait_remote;
I am wondering how the extra lock(rq)/unlock(rq) in schedule() is bad compared to atomic_dec.
IOW, what if we simply make rq->nr_iowait "int" and change schedule() to update it? Something like below. Just curious.
As for nr_iowait_local + nr_iowait_remote, this doesn't look safe... in theory nr_iowait_cpu() or even nr_iowait() can return a negative number.
Oleg.
kernel/sched/core.c +9 -7
@@ -2435,6 +2435,9 @@ need_resched: rq->curr = next; ++*switch_count; +if (unlikely (prev->in_iowait)) + rq->nr_iowait++; +context_switch (rq, prev, next);/* unlocks the rq */ /* * The context switch have flipped the stack from under us
@@ -2442,6 +2445,12 @@ need_resched:* this task called schedule() in the past. prev == current * is still correct, but it can be moved to another cpu/rq. */ +if (unlikely (prev->in_iowait)) { +raw_spin_lock_irq (&rq->lock); + rq->nr_iowait--; +raw_spin_unlock_irq (&rq->lock); + } + cpu =smp_processor_id (); rq =cpu_rq (cpu); }else
@@ -3939,31 +3948,24 @@ EXPORT_SYMBOL_GPL(yield_to);*/ void __sched io_schedule(void) { -struct rq *rq =raw_rq (); -delayacct_blkio_start (); -atomic_inc (&rq->nr_iowait);blk_flush_plug (current); current->in_iowait =1 ;schedule (); current->in_iowait =0 ; -atomic_dec (&rq->nr_iowait);delayacct_blkio_end (); }EXPORT_SYMBOL (io_schedule);long __sched io_schedule_timeout (long timeout) { -struct rq *rq =raw_rq ();long ret;delayacct_blkio_start (); -atomic_inc (&rq->nr_iowait);blk_flush_plug (current); current->in_iowait =1 ; ret =schedule_timeout (timeout); current->in_iowait =0 ; -atomic_dec (&rq->nr_iowait);delayacct_blkio_end ();return ret; }
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 11 lines · Oleg Nesterov
On 08/20, Peter Zijlstra wrote:
I am wondering how the extra lock(rq)/unlock(rq) in
schedule()is bad compared toatomic_dec.IOW, what if we simply make
rq->nr_iowait"int" and changeschedule()to update it? Something like below. Just curious.As for
nr_iowait_local+nr_iowait_remote, this doesn't look safe... in theorynr_iowait_cpu()or evennr_iowait()can return a negative number.
I'm too tired to see how its different, I'll think on it :-)
Quoted 13 lines · That
kernel/sched/core.c +3
-0@@ -2435,6 +2435,9 @@ need_resched: rq->curr = next; ++*switch_count; +if (unlikely (prev->in_iowait)) + rq->nr_iowait++; +context_switch (rq, prev, next);/* unlocks the rq */ /* * The context switch have flipped the stack from under us
kernel/sched/core.c need_resched: · line 2452
Quoted 8 lines
kernel/sched/core.c +5
-0@@ -2442,6 +2445,12 @@ need_resched:* this task called schedule() in the past. prev == current * is still correct, but it can be moved to another cpu/rq. */ +if (unlikely (prev->in_iowait)) { +raw_spin_lock_irq (&rq->lock); + rq->nr_iowait--; +raw_spin_unlock_irq (&rq->lock); + }
This seems like the wrong place, this is where you return from schedule() running another task, not where the task you just send to sleep wakes up.
If you want to to this, the nr_iowait decrement needs to be in the wakeup path someplace.
Quoted 3 lines
kernel/sched/core.c +0
-0cpu =smp_processor_id (); rq =cpu_rq (cpu); }else
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 18 lines · Peter Zijlstra
Quoted 16 lines · Oleg Nesterov
kernel/sched/core.c +3
-0@@ -2435,6 +2435,9 @@ need_resched: rq->curr = next; ++*switch_count; +if (unlikely (prev->in_iowait)) + rq->nr_iowait++; +context_switch (rq, prev, next);/* unlocks the rq */ /* * The context switch have flipped the stack from under us @@ -2442,6 +2445,12 @@ need_resched:* this task called schedule() in the past. prev == current * is still correct, but it can be moved to another cpu/rq. */
Quoted 8 lines
kernel/sched/core.c need_resched: · line 2452
Quoted 5 lines
kernel/sched/core.c +5
-0+if (unlikely (prev->in_iowait)) { +raw_spin_lock_irq (&rq->lock); + rq->nr_iowait--; +raw_spin_unlock_irq (&rq->lock); + }This seems like the wrong place, this is where you return from
schedule()running another task,
Yes, but prev is current, and rq should be "correct" for rq->nr_iowait-- ?
This local var should be equal to its value when this task called context_switch() in the past.
Like any other variable, like "rq = raw_rq()" in io_schedule().
Quoted 2 lines
not where the task you just send to
sleep wakes up.
sure, but currently io_schedule() does the same.
Btw. Whatever we do, can't we unify io_schedule/io_schedule_timeout?
Oleg.
kernel/sched/core.c +1 -10
@@ -3939,16 +3939,7 @@ EXPORT_SYMBOL_GPL(yield_to);*/ void __sched io_schedule(void) { -struct rq *rq =raw_rq (); - -delayacct_blkio_start (); -atomic_inc (&rq->nr_iowait); -blk_flush_plug (current); - current->in_iowait =1 ; -schedule (); - current->in_iowait =0 ; -atomic_dec (&rq->nr_iowait); -delayacct_blkio_end (); +io_schedule_timeout (MAX_SCHEDULE_TIMEOUT ); }EXPORT_SYMBOL (io_schedule);
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 13 lines · Oleg Nesterov
Quoted 11 lines · Peter Zijlstra
Review
Quoted 5 lines · Oleg Nesterov
(quoted diff) +5
-0+ if (unlikely(prev->in_iowait)) { + raw_spin_lock_irq(&rq->lock); + rq->nr_iowait--; + raw_spin_unlock_irq(&rq->lock); + }This seems like the wrong place, this is where you return from
schedule()running another task,
Quoted 2 lines
Yes, but prev is current, and rq should be "correct" for rq->nr_iowait-- ?
Yes its the right rq, but the wrong time.
Quoted 9 lines
This local var should be equal to its value when this task called
context_switch()in the past.Like any other variable, like "rq =
raw_rq()" inio_schedule().Quoted 2 lines
not where the task you just send to
sleep wakes up.sure, but currently
io_schedule()does the same.
No it doesn't. It only does the decrement when the task is woken back up. Not right after it switches out.
Quoted 1 line
Btw. Whatever we do, can't we unify io_schedule/io_schedule_timeout?
I suppose we could, a timeout of MAX_SCHEDULE_TIMEOUT will act like a regular schedule, but it gets all the overhead of doing schedule_timeout(). So I don't think its a win.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 15 lines · Peter Zijlstra
Quoted 13 lines · Oleg Nesterov
Quoted 11 lines · Peter Zijlstra
Review
Quoted 5 lines · Oleg Nesterov
(quoted diff) +5
-0+ if (unlikely(prev->in_iowait)) { + raw_spin_lock_irq(&rq->lock); + rq->nr_iowait--; + raw_spin_unlock_irq(&rq->lock); + }This seems like the wrong place, this is where you return from
schedule()running another task,
Quoted 4 lines
Quoted 2 lines
Yes, but prev is current, and rq should be "correct" for rq->nr_iowait-- ?
Yes its the right rq, but the wrong time.
Hmm. Just in case, it is not that I think this patch really makes sense, but I'd like to understand why do you think it is wrong.
Quoted 5 lines
Quoted 5 lines
This local var should be equal to its value when this task called
context_switch()in the past.Like any other variable, like "rq =
raw_rq()" inio_schedule().
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
not where the task you just send to
sleep wakes up.sure, but currently
io_schedule()does the same.No it doesn't. It only does the decrement when the task is woken back up. Not right after it switches out.
But it is not "after it switches out", it is after it switched back.
Lets ignore the locking,
if (prev->in_iowait) rq->nr_iowait++; context_switch(prev, next); if (prev->in_iowait) rq->nr_iowait--;
Quoted 1 line
From the task_struct's (current's) pov prev/rq are the same, before or
after context_switch().
But from the CPU's pov they differ. And ignoring more details on UP the code above is equivalent to
if (prev->in_iowait) rq->nr_iowait++; if (next->in_iowait) rq->nr_iowait--; context_switch(prev, next);
No?
Yes, need_resched()/preemption can trigger more inc/dec's than io_schedule() does, but I don't think this was your concern.
Quoted 5 lines
Quoted 1 line
Btw. Whatever we do, can't we unify io_schedule/io_schedule_timeout?
I suppose we could, a timeout of
MAX_SCHEDULE_TIMEOUTwill act like a regular schedule, but it gets all the overhead of doingschedule_timeout(). So I don't think its a win.
Well, the only overhead is "if(to == MAX_SCHEDULE_TIMEOUT)" at the start. I don't think it makes sense to copy-and-paste the identical code to avoid it. But please ignore, this is really minor and off-topic.
Oleg.
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 8 lines · Oleg Nesterov
Quoted 1 line · Peter Zijlstra
Yes its the right rq, but the wrong time.
Hmm. Just in case, it is not that I think this patch really makes sense, but I'd like to understand why do you think it is wrong.
But it is not "after it switches out", it is after it switched back.
D'uh I was being particularly dense it seems :/
Yes I think it is correct. You're now trading two atomic ops on a different cacheline than rq->lock for one atomic op on the rq->lock.
Might be a win, esp. since hopefully there's a fair chance its the same runqueue.
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Oleg Nesterov
Quoted 1 line · Peter Zijlstra
Yes its the right rq, but the wrong time.
Hmm. Just in case, it is not that I think this patch really makes sense, but I'd like to understand why do you think it is wrong.
But it is not "after it switches out", it is after it switched back.
Quoted 7 lines
D'uh I was being particularly dense it seems :/
Yes I think it is correct. You're now trading two atomic ops on a different cacheline than
rq->lockfor one atomic op on therq->lock.Might be a win, esp. since hopefully there's a fair chance its the same runqueue.
The other consideration is that this adds two branches to the normal schedule path. I really don't know what the regular ratio between schedule() and io_schedule() is -- and I suspect it can very much depend on workload -- but it might be a net loss due to that, even if it makes io_schedule() 'lots' cheaper.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 6 lines · Peter Zijlstra
The other consideration is that this adds two branches to the normal schedule path. I really don't know what the regular ratio between
schedule()andio_schedule()is -- and I suspect it can very much depend on workload -- but it might be a net loss due to that, even if it makesio_schedule()'lots' cheaper.
Yes, agreed. Please ignore it for now, I didn't try to actually suggest this change. And even if this is fine perfomance wise, this needs some benchmarking.
Well. actually I have a vague feeling that _perhaps_ this change can help to solve other problems we are discussing, but I am not sure and right now I can't even explain the idea to me.
In short: please ignore ;)
Oleg.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 12 lines · Oleg Nesterov
Quoted 6 lines · Peter Zijlstra
The other consideration is that this adds two branches to the normal schedule path. I really don't know what the regular ratio between
schedule()andio_schedule()is -- and I suspect it can very much depend on workload -- but it might be a net loss due to that, even if it makesio_schedule()'lots' cheaper.Yes, agreed. Please ignore it for now, I didn't try to actually suggest this change. And even if this is fine perfomance wise, this needs some benchmarking.
Quoted 8 lines
Well. actually I have a vague feeling that _perhaps_ this change can help to solve other problems we are discussing, but I am not sure and right now I can't even explain the idea to me.
In short: please ignore ;)
Oleg.
Ok, the discussion is hard to synthesize but I think I now have more clues to send a better new iteration.
So we have the choice between keeping atomics or using rq->lock. It seems that using rq->lock is going to be worrisome for several reasons. So let's stick to atomics (but tell me if you prefer the other way).
So the idea for the next try is to do something along the lines of:
struct cpu_idletime {
nr_iowait,
seqlock,
idle_start,
idle_time,
iowait_time,
} __cacheline_aligned_in_smp;
DEFINE_PER_CPU(struct cpu_idletime, cpu_idletime);
io_schedule()
{
int prev_cpu;
preempt_disable();
prev_cpu_idletime = __this_cpu_ptr(&cpu_idletime);
atomic_inc(prev_cpu_idletime->nr_iowait);
WARN_ON_ONCE(is_idle_task(current));
preempt_enable_no_resched();
schedule();
write_seqlock(prev_cpu_idletime->seqlock)
if (!atomic_dec_return(prev_cpu_idletime->nr_iowait))
flush_cpu_idle_time(prev_cpu_idletime, 1)
write_sequnlock(prev_cpu_idletime->seqlock)}
flush_cpu_idle_time(cpu_idletime, iowait)
{
if (!cpu_idletime->idle_start)
return;
if (nr_iowait)
cpu_idletime->iowait_time = NOW() - cpu_idletime->idle_start;
else
cpu_idletime->idle_time = NOW() - cpu_idletime->idle_start;
}idle_entry()
{
write_seqlock(this_cpu_idletime->seqlock)
this_cpu_idletime->idle_start = NOW();
write_sequnlock(iowait(cpu)->seqlock)
}idle_exit()
{
write_seqlock(this_cpu_idletime->seqlock)
flush_cpu_idle_time(this_cpu_idletime, atomic_read(&this_cpu_idletime->nr_iowait));
this_cpu_idletime->idle_start = 0;
write_sequnlock(this_cpu_idletime->seqlock)
}Now this all realy on the fact that atomic_inc(cpu_idletime->nr_iowait) can't happen in a CPU that is already idle. So it can't happen between idle_entry() and idle_exit(). Hence the WARN_ON(is_idle_task(current)) below after the inc.
Hmm?
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
2013/10/1 Frederic Weisbecker <[email protected]>:
Quoted 42 lines · Oleg Nesterov
Quoted 12 lines · Peter Zijlstra
Quoted 6 lines
The other consideration is that this adds two branches to the normal schedule path. I really don't know what the regular ratio between
schedule()andio_schedule()is -- and I suspect it can very much depend on workload -- but it might be a net loss due to that, even if it makesio_schedule()'lots' cheaper.Yes, agreed. Please ignore it for now, I didn't try to actually suggest this change. And even if this is fine perfomance wise, this needs some benchmarking.
Quoted 8 lines
Well. actually I have a vague feeling that _perhaps_ this change can help to solve other problems we are discussing, but I am not sure and right now I can't even explain the idea to me.
In short: please ignore ;)
Oleg.
Ok, the discussion is hard to synthesize but I think I now have more clues to send a better new iteration.
So we have the choice between keeping atomics or using
rq->lock. It seems that usingrq->lockis going to be worrisome for several reasons. So let's stick to atomics (but tell me if you prefer the other way).So the idea for the next try is to do something along the lines of:
struct
cpu_idletime{
nr_iowait,
seqlock,
idle_start,
idle_time,
iowait_time,
}__cacheline_aligned_in_smp;
DEFINE_PER_CPU(struct cpu_idletime, cpu_idletime);
io_schedule()
Quoted 14 lines
{
int prev_cpu;
preempt_disable();
prev_cpu_idletime = __this_cpu_ptr(&cpu_idletime);
atomic_inc(prev_cpu_idletime->nr_iowait);
WARN_ON_ONCE(is_idle_task(current));
preempt_enable_no_resched();
schedule();
write_seqlock(prev_cpu_idletime->seqlock)
if (!atomic_dec_return(prev_cpu_idletime->nr_iowait))
flush_cpu_idle_time(prev_cpu_idletime, 1)I forgot...
cpu_idletime->idle_start;
after the update.
Also now I wonder if we actually should lock the inc part. Otherwise it may be hard to get the readers right...
Thanks.
Quoted 36 lines
write_sequnlock(prev_cpu_idletime->seqlock)}
flush_cpu_idle_time(cpu_idletime, iowait){ if (!cpu_idletime->idle_start) return; if (nr_iowait) cpu_idletime->iowait_time = NOW() - cpu_idletime->idle_start; else cpu_idletime->idle_time = NOW() - cpu_idletime->idle_start; }
idle_entry(){ write_seqlock(this_cpu_idletime->seqlock) this_cpu_idletime->idle_start = NOW(); write_sequnlock(iowait(cpu)->seqlock) }
idle_exit(){ write_seqlock(this_cpu_idletime->seqlock) flush_cpu_idle_time(this_cpu_idletime, atomic_read(&this_cpu_idletime->nr_iowait)); this_cpu_idletime->idle_start = 0; write_sequnlock(this_cpu_idletime->seqlock) }Now this all realy on the fact that
atomic_inc(cpu_idletime->nr_iowait)can't happen in a CPU that is already idle. So it can't happen betweenidle_entry()andidle_exit(). Hence the WARN_ON(is_idle_task(current)) below after the inc.Hmm?
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
2013/10/1 Frederic Weisbecker <[email protected]>:
Quoted 2 lines
I forgot...
cpu_idletime->idle_start;
cpu_idletime->idle_start = NOW();
grrr.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Frederic Weisbecker2013/10/1 Frederic Weisbecker <[email protected]>:
Quoted 59 lines
2013/10/1 Frederic Weisbecker <[email protected]>:
Quoted 42 lines · Oleg Nesterov
Quoted 12 lines · Peter Zijlstra
Quoted 6 lines
The other consideration is that this adds two branches to the normal schedule path. I really don't know what the regular ratio between
schedule()andio_schedule()is -- and I suspect it can very much depend on workload -- but it might be a net loss due to that, even if it makesio_schedule()'lots' cheaper.Yes, agreed. Please ignore it for now, I didn't try to actually suggest this change. And even if this is fine perfomance wise, this needs some benchmarking.
Quoted 8 lines
Well. actually I have a vague feeling that _perhaps_ this change can help to solve other problems we are discussing, but I am not sure and right now I can't even explain the idea to me.
In short: please ignore ;)
Oleg.
Ok, the discussion is hard to synthesize but I think I now have more clues to send a better new iteration.
So we have the choice between keeping atomics or using
rq->lock. It seems that usingrq->lockis going to be worrisome for several reasons. So let's stick to atomics (but tell me if you prefer the other way).So the idea for the next try is to do something along the lines of:
struct
cpu_idletime{
nr_iowait,
seqlock,
idle_start,
idle_time,
iowait_time,
}__cacheline_aligned_in_smp;
DEFINE_PER_CPU(struct cpu_idletime, cpu_idletime);
io_schedule()Quoted 14 lines
{ int prev_cpu; preempt_disable(); prev_cpu_idletime = __this_cpu_ptr(&cpu_idletime); atomic_inc(prev_cpu_idletime->nr_iowait); WARN_ON_ONCE(is_idle_task(current)); preempt_enable_no_resched(); schedule(); write_seqlock(prev_cpu_idletime->seqlock) if (!atomic_dec_return(prev_cpu_idletime->nr_iowait)) flush_cpu_idle_time(prev_cpu_idletime, 1)I forgot...
Quoted 45 lines
cpu_idletime->idle_start;after the update.
Also now I wonder if we actually should lock the inc part. Otherwise it may be hard to get the readers right...
Thanks.
Quoted 36 lines
write_sequnlock(prev_cpu_idletime->seqlock)}
flush_cpu_idle_time(cpu_idletime, iowait){ if (!cpu_idletime->idle_start) return; if (nr_iowait) cpu_idletime->iowait_time = NOW() - cpu_idletime->idle_start; else cpu_idletime->idle_time = NOW() - cpu_idletime->idle_start; }
idle_entry(){ write_seqlock(this_cpu_idletime->seqlock) this_cpu_idletime->idle_start = NOW(); write_sequnlock(iowait(cpu)->seqlock) }
idle_exit(){ write_seqlock(this_cpu_idletime->seqlock) flush_cpu_idle_time(this_cpu_idletime, atomic_read(&this_cpu_idletime->nr_iowait)); this_cpu_idletime->idle_start = 0; write_sequnlock(this_cpu_idletime->seqlock) }Now this all realy on the fact that
atomic_inc(cpu_idletime->nr_iowait)can't happen in a CPU that is already idle. So it can't happen betweenidle_entry()andidle_exit(). Hence the WARN_ON(is_idle_task(current)) below after the inc.Hmm?
Actually if we record the source of the idle entry time (whether we enter idle as iowait or idlewait) and protect it inside the seqlock, we can probably make the readers safe. So they would rely on that instead of the nr_io_wait which is only used by the updaters.
I'm probably forgetting something obvious. Anyway I'll try to make a patchset out of that, unless somebody points me out the mistakes I'm missing here.
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Frederic WeisbeckerQuoted 27 lines · Frederic Weisbecker
struct
cpu_idletime{
nr_iowait,
seqlock,
idle_start,
idle_time,
iowait_time,
}__cacheline_aligned_in_smp;
DEFINE_PER_CPU(struct cpu_idletime, cpu_idletime);
io_schedule(){ int prev_cpu; preempt_disable(); prev_cpu_idletime = __this_cpu_ptr(&cpu_idletime); atomic_inc(prev_cpu_idletime->nr_iowait); WARN_ON_ONCE(is_idle_task(current)); preempt_enable_no_resched(); schedule(); write_seqlock(prev_cpu_idletime->seqlock) if (!atomic_dec_return(prev_cpu_idletime->nr_iowait)) flush_cpu_idle_time(prev_cpu_idletime, 1) write_sequnlock(prev_cpu_idletime->seqlock)
Quoted 1 line
}
This is at least 3 atomic ops and a whole bunch of branches extra. It used to be 2 atomics and no branches.
What again are we solving any why?
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 29 lines · Peter Zijlstra
Quoted 27 lines · Frederic Weisbecker
struct
cpu_idletime{
nr_iowait,
seqlock,
idle_start,
idle_time,
iowait_time,
}__cacheline_aligned_in_smp;
DEFINE_PER_CPU(struct cpu_idletime, cpu_idletime);
io_schedule(){ int prev_cpu; preempt_disable(); prev_cpu_idletime = __this_cpu_ptr(&cpu_idletime); atomic_inc(prev_cpu_idletime->nr_iowait); WARN_ON_ONCE(is_idle_task(current)); preempt_enable_no_resched(); schedule(); write_seqlock(prev_cpu_idletime->seqlock) if (!atomic_dec_return(prev_cpu_idletime->nr_iowait)) flush_cpu_idle_time(prev_cpu_idletime, 1) write_sequnlock(prev_cpu_idletime->seqlock)
Quoted 4 lines
Quoted 1 line
}
This is at least 3 atomic ops and a whole bunch of branches extra. It used to be 2 atomics and no branches.
Yeah. If somebody has a better proposition, I'm all for it. Of course the best would be to remove these stats if we can. I think we already concluded that the idea of per CPU iowait stats is broken since sleeping tasks aren't assigned a particular CPU.
Quoted 2 lines
What again are we solving any why?
Fernando summerized it better than I could
* https://lkml.org/lkml/2013/3/18/962
* and http://marc.info/?l=linux-kernel&m=137273800916899&w=2
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Frederic WeisbeckerSo what's wrong with something like:
struct cpu_idletime {
seqlock_t seqlock;
unsigned long nr_iowait;
u64 start;
u64 idle_time,
u64 iowait_time,
} __cacheline_aligned_in_smp;DEFINE_PER_CPU(struct cpu_idletime, cpu_idletime);
void io_schedule(void)
{
struct cpu_idletime *it = __raw_get_cpu_var(cpu_idletime);
write_seqlock(&it->seqlock);
if (!it->nr_iowait++)
it->start = local_clock();
write_sequnlock(&it->seqlock);
current->in_iowait = 1;
schedule();
current->in_iowait = 0;
write_seqlock(&it->seqlock);
if (!--it->nr_iowait)
it->iowait_time += local_clock() - it->start;
write_sequnlock(&it->seqlock);
}Afaict you don't need the preempt disable and atomic muck at all.
It will all get a little more complicated to deal with overlapping idle and iowait times, but the idea is the same.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 32 lines · Peter Zijlstra
So what's wrong with something like:
struct cpu_idletime { seqlock_t seqlock; unsigned long nr_iowait; u64 start; u64 idle_time, u64 iowait_time, } __cacheline_aligned_in_smp;
DEFINE_PER_CPU(struct cpu_idletime, cpu_idletime);void
io_schedule(void){ struct cpu_idletime *it = __raw_get_cpu_var(cpu_idletime); write_seqlock(&it->seqlock); if (!it->nr_iowait++) it->start = local_clock(); write_sequnlock(&it->seqlock); current->in_iowait = 1; schedule(); current->in_iowait = 0; write_seqlock(&it->seqlock); if (!--it->nr_iowait) it->iowait_time += local_clock() - it->start; write_sequnlock(&it->seqlock); }
Quoted 1 line
Afaict you don't need the preempt disable and atomic muck at all.
Yeah thinking more about it, the preempt disable was probably not necessary. Now that's trading 2 atomics + 1 Lock/Unlock with 2 Lock/Unlock.
OTOH it computes iowait time seperately from idle time, so we probably don't need to lock the idle time anymore.
Plus this solution is much much more simple. So if nobody sees a flaw there, I'll try this.
Thanks.
Quoted 3 lines
It will all get a little more complicated to deal with overlapping idle and iowait times, but the idea is the same.
Probably no big deal.
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 2 lines · Frederic Weisbecker
Yeah thinking more about it, the preempt disable was probably not necessary. Now that's trading 2 atomics + 1 Lock/Unlock with 2 Lock/Unlock.
It trades the current 2 atomics for 2 LOCK/UNLOCK. And on x86_64 that's 2 atomics.
So all we get is some extra branches; which we must hope for don't actually mess things up too bad.
Ohh.. wait a sec.. we also call local_clock() which includes another atomic :/ Bugger..
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Frederic Weisbecker
Yeah thinking more about it, the preempt disable was probably not necessary. Now that's trading 2 atomics + 1 Lock/Unlock with 2 Lock/Unlock.
It trades the current 2 atomics for 2 LOCK/UNLOCK. And on
x86_64that's 2 atomics.
Do you mean 2 atomics for LOCK/UNLOCK? Or is that pair optimized somehow in x86?
Quoted 6 lines
So all we get is some extra branches; which we must hope for don't actually mess things up too bad.
Ohh.. wait a sec.. we also call
local_clock()which includes another atomic :/ Bugger..
Yeah. Anyway, I'm going to try something on top of that. May be we'll get a fresher mind and ideas on how to optimize that all after.
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 10 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Frederic Weisbecker
Yeah thinking more about it, the preempt disable was probably not necessary. Now that's trading 2 atomics + 1 Lock/Unlock with 2 Lock/Unlock.
It trades the current 2 atomics for 2 LOCK/UNLOCK. And on
x86_64that's 2 atomics.Do you mean 2 atomics for LOCK/UNLOCK? Or is that pair optimized somehow in x86?
Unlock isn't an atomic on x86_64; it can be on certain i386 builds. See UNLOCK_LOCK_PREFIX.
Quoted 9 lines
Quoted 6 lines
So all we get is some extra branches; which we must hope for don't actually mess things up too bad.
Ohh.. wait a sec.. we also call
local_clock()which includes another atomic :/ Bugger..Yeah. Anyway, I'm going to try something on top of that. May be we'll get a fresher mind and ideas on how to optimize that all after.
Fair enough.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Frederic WeisbeckerQuoted 10 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Frederic Weisbecker
Yeah thinking more about it, the preempt disable was probably not necessary. Now that's trading 2 atomics + 1 Lock/Unlock with 2 Lock/Unlock.
It trades the current 2 atomics for 2 LOCK/UNLOCK. And on
x86_64that's 2 atomics.Do you mean 2 atomics for LOCK/UNLOCK? Or is that pair optimized somehow in x86?
unlock is not actually an atomic.
and on some modern machines, neither is the lock, for the uncontended case ;-)
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 9 lines · Arjan van de Ven
Quoted 8 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Frederic Weisbecker
Yeah thinking more about it, the preempt disable was probably not necessary. Now that's trading 2 atomics + 1 Lock/Unlock with 2 Lock/Unlock.
It trades the current 2 atomics for 2 LOCK/UNLOCK. And on
x86_64that's 2 atomics.
Quoted 7 lines
Quoted 2 lines
Do you mean 2 atomics for LOCK/UNLOCK? Or is that pair optimized somehow in x86?
unlock is not actually an atomic.
and on some modern machines, neither is the lock, for the uncontended case ;-)
Yeah, we are never getting short on black magic as the time goes :)
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Oleg NesterovQuoted 9 lines · Oleg Nesterov
Quoted 5 lines
Quoted 1 line
Btw. Whatever we do, can't we unify io_schedule/io_schedule_timeout?
I suppose we could, a timeout of
MAX_SCHEDULE_TIMEOUTwill act like a regular schedule, but it gets all the overhead of doingschedule_timeout(). So I don't think its a win.Well, the only overhead is "
if(to == MAX_SCHEDULE_TIMEOUT)" at the start. I don't think it makes sense to copy-and-paste the identical code to avoid it. But please ignore, this is really minor and off-topic.
Ah, so the 'problem' is that schedule_timeout() doesn't live in kernel/sched/core.c and we thus get an extra function call (which are somewhat expensive on some archs).
That said, I suppose we could do something like the below, that should allow the compiler to DTRT.
--- include/linux/sched.h | 35 ++++++++++++++++++++++--- kernel/sched/core.c | 22 +++++----------- kernel/timer.c | 67 ++++++++++-------------------------------------- 3 files changed, 52 insertions(+), 72 deletions(-)
include/linux/sched.h +31 -4
@@ -303,10 +303,37 @@ extern char __sched_text_start[], __sched_text_end[];extern int in_sched_functions (unsigned long addr);#define MAX_SCHEDULE_TIMEOUT LONG_MAX -extern signed long schedule_timeout (signed long timeout); -extern signed long schedule_timeout_interruptible (signed long timeout); -extern signed long schedule_timeout_killable (signed long timeout); -extern signed long schedule_timeout_uninterruptible (signed long timeout); +extern long __schedule_timeout (long timeout); + +static inline long schedule_timeout (long timeout) +{ +if (timeout ==MAX_SCHEDULE_TIMEOUT ) { +schedule (); +return timeout; + } +return __schedule_timeout (timeout); +} + +/* +* We can use __set_current_state() here because schedule_timeout() calls +* schedule() unconditionally. +*/ +static inline long schedule_timeout_interruptible (long timeout) +{ +__set_current_state (TASK_INTERRUPTIBLE ); +return schedule_timeout (timeout); +} +static inline long schedule_timeout_killable (long timeout) +{ +__set_current_state (TASK_KILLABLE ); +return schedule_timeout (timeout); +} +static inline long schedule_timeout_uninterruptible (long timeout) +{ +__set_current_state (TASK_UNINTERRUPTIBLE ); +return schedule_timeout (timeout); +} + asmlinkage void schedule(void);extern void schedule_preempt_disabled (void );
kernel/sched/core.c +7 -15
@@ -3937,21 +3937,6 @@ EXPORT_SYMBOL_GPL(yield_to);* This task is about to go to sleep on IO. Increment rq->nr_iowait so * that process accounting knows that this is a task in IO wait state. */ -void __sched io_schedule(void) -{ -struct rq *rq =raw_rq (); - -delayacct_blkio_start (); -atomic_inc (&rq->nr_iowait); -blk_flush_plug (current); - current->in_iowait =1 ; -schedule (); - current->in_iowait =0 ; -atomic_dec (&rq->nr_iowait); -delayacct_blkio_end (); -} -EXPORT_SYMBOL (io_schedule); -long __sched io_schedule_timeout (long timeout) {struct rq *rq =raw_rq ();
@@ -3968,6 +3953,13 @@ long __sched io_schedule_timeout(long timeout)return ret; } +void __sched io_schedule(void) +{ +io_schedule_timeout (MAX_SCHEDULE_TIMEOUT ); +} +EXPORT_SYMBOL (io_schedule); + +/** * sys_sched_get_priority_max - return maximum RT priority. * @policy: scheduling class.
kernel/timer.c +14 -53
84 lines not on this page · raw message
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 10 lines · Peter Zijlstra
Quoted 4 lines · Oleg Nesterov
Well, the only overhead is "
if(to == MAX_SCHEDULE_TIMEOUT)" at the start. I don't think it makes sense to copy-and-paste the identical code to avoid it. But please ignore, this is really minor and off-topic.Ah, so the 'problem' is that
schedule_timeout()doesn't live inkernel/sched/core.cand we thus get an extra function call (which are somewhat expensive on some archs).
So, unlike me, you like -02 more than -Os ;)
include/linux/sched.h extern char __sched_text_start[], __sched_text_end[]; · line 315
Quoted 8 lines
include/linux/sched.h +8
-0+static inline long schedule_timeout (long timeout) +{ +if (timeout ==MAX_SCHEDULE_TIMEOUT ) { +schedule (); +return timeout; + } +return __schedule_timeout (timeout); +}
Well this means that every caller will do the MAX_SCHEDULE_TIMEOUT check inline, and this case is unlikely. And you are also going to make schedule_timeout_*() inline...
But,
Quoted 27 lines
(quoted diff) +15
-11+static inline long schedule_timeout_interruptible(long timeout) +{ + __set_current_state(TASK_INTERRUPTIBLE); + return schedule_timeout(timeout); +} +static inline long schedule_timeout_killable(long timeout) +{ + __set_current_state(TASK_KILLABLE); + return schedule_timeout(timeout); +} +static inline long schedule_timeout_uninterruptible(long timeout) +{ + __set_current_state(TASK_UNINTERRUPTIBLE); + return schedule_timeout(timeout); +} -signed long __sched schedule_timeout_interruptible(signed long timeout) -{ - __set_current_state(TASK_INTERRUPTIBLE); - return schedule_timeout(timeout); -} -EXPORT_SYMBOL(schedule_timeout_interruptible); - -signed long __sched schedule_timeout_killable(signed long timeout) -{ - __set_current_state(TASK_KILLABLE); - return schedule_timeout(timeout);
Review
Quoted 10 lines
(quoted diff) +1
-9-} -EXPORT_SYMBOL(schedule_timeout_killable); - -signed long __sched schedule_timeout_uninterruptible(signed long timeout) -{ - __set_current_state(TASK_UNINTERRUPTIBLE); - return schedule_timeout(timeout); -} -EXPORT_SYMBOL(schedule_timeout_uninterruptible); +EXPORT_SYMBOL(__schedule_timeout);
personally I think this particular change is fine.
Or we can export a single schedule_timeout_state(timeout, state) to factor out get_current().
But just in case, of course I won't argue in any case.
Oleg.
Peter Zijlstra
maintainerRe: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 1 line · Oleg Nesterov
So, unlike me, you like -02 more than -Os ;)
I haven't checked the actual flags they enable in a while, but I think I prefer something in the middle.
Esp. -freorder-blocks and the various -falign flags are something you really want with -Os.
Quoted 11 lines
include/linux/sched.h extern char __sched_text_start[], __sched_text_end[]; · line 315
Quoted 8 lines
include/linux/sched.h +8
-0+static inline long schedule_timeout (long timeout) +{ +if (timeout ==MAX_SCHEDULE_TIMEOUT ) { +schedule (); +return timeout; + } +return __schedule_timeout (timeout); +}Well this means that every caller will do the
MAX_SCHEDULE_TIMEOUTcheck inline, and this case is unlikely.
OK, so do not remove the MAX_SCHEDULE_TIMEOUT check from __schedule_timeout() and change the above to:
static __always_inline long schedule_timeout(long timeout)
{
if (__builtin_constant_p(timeout) && timeout == MAX_SCHEDULE_TIMEOUT) {
schedule();
return timeout;
}
return __schedule_timeout(timeout);
}That should avoid extra code generation for the runtime sites while still allowing what we set out to do.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 4 lines · Peter Zijlstra
OK, so do not remove the
MAX_SCHEDULE_TIMEOUTcheck from__schedule_timeout()and change the above to:
Quoted 8 lines
static
__always_inlinelongschedule_timeout(long timeout){ if (__builtin_constant_p(timeout) && timeout == MAX_SCHEDULE_TIMEOUT) { schedule(); return timeout; } return __schedule_timeout(timeout); }
Agreed, this looks nice.
Hmm. And this can simplify some changes in linux/wait.h I am going to resend.
Oleg.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Oleg NesterovQuoted 38 lines · Oleg Nesterov
kernel/sched/sched.h struct rq { · line 457
Quoted 10 lines · Peter Zijlstra
kernel/sched/sched.h +2
-1@@ -453,7 +453,8 @@ struct rq {u64 clock;u64 clock_task; -atomic_t nr_iowait; +int nr_iowait_local; +atomic_t nr_iowait_remote;I am wondering how the extra lock(rq)/unlock(rq) in
schedule()is bad compared toatomic_dec.IOW, what if we simply make
rq->nr_iowait"int" and changeschedule()to update it? Something like below. Just curious.As for
nr_iowait_local+nr_iowait_remote, this doesn't look safe... in theorynr_iowait_cpu()or evennr_iowait()can return a negative number.Oleg.
kernel/sched/core.c +3
-0@@ -2435,6 +2435,9 @@ need_resched: rq->curr = next; ++*switch_count; +if (unlikely (prev->in_iowait)) + rq->nr_iowait++; +context_switch (rq, prev, next);/* unlocks the rq */ /* * The context switch have flipped the stack from under us
kernel/sched/core.c need_resched: · line 2453
Quoted 9 lines
kernel/sched/core.c +6
-0@@ -2442,6 +2445,12 @@ need_resched:* this task called schedule() in the past. prev == current * is still correct, but it can be moved to another cpu/rq. */ +if (unlikely (prev->in_iowait)) { +raw_spin_lock_irq (&rq->lock); + rq->nr_iowait--; +raw_spin_unlock_irq (&rq->lock); + } +
It seems that with this solution rq->nr_iowait is only ever modified locally. Can't we just disable irqs for rq->nr_iowait-- ?
Also if this is only updated locally, no need for a lock to update ts->iowait_sleep_time anymore, we can flush the io sleeptime in place. Great thing.
Now just in case, it might be worth noting as well that the time elapsing from the rq task enqueue until it is finally running on the CPU is not accounted anymore there. This can make a little difference if the task is woken up but a higher priority task, possibly SCHED_FIFO is already running on the CPU. Now do we care...
Thanks.
Quoted 36 lines
kernel/sched/core.c +0
-7cpu =smp_processor_id (); rq =cpu_rq (cpu); }else @@ -3939,31 +3948,24 @@ EXPORT_SYMBOL_GPL(yield_to);*/ void __sched io_schedule(void) { -struct rq *rq =raw_rq (); -delayacct_blkio_start (); -atomic_inc (&rq->nr_iowait);blk_flush_plug (current); current->in_iowait =1 ;schedule (); current->in_iowait =0 ; -atomic_dec (&rq->nr_iowait);delayacct_blkio_end (); }EXPORT_SYMBOL (io_schedule);long __sched io_schedule_timeout (long timeout) { -struct rq *rq =raw_rq ();long ret;delayacct_blkio_start (); -atomic_inc (&rq->nr_iowait);blk_flush_plug (current); current->in_iowait =1 ; ret =schedule_timeout (timeout); current->in_iowait =0 ; -atomic_dec (&rq->nr_iowait);delayacct_blkio_end ();return ret; }
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 20 lines · Frederic Weisbecker
Quoted 18 lines · Oleg Nesterov
kernel/sched/core.c +4
-0@@ -2435,6 +2435,9 @@ need_resched: rq->curr = next; ++*switch_count; +if (unlikely (prev->in_iowait)) + rq->nr_iowait++; +context_switch (rq, prev, next);/* unlocks the rq */ /* * The context switch have flipped the stack from under us @@ -2442,6 +2445,12 @@ need_resched:* this task called schedule() in the past. prev == current * is still correct, but it can be moved to another cpu/rq. */ +if (unlikely (prev->in_iowait)) {
Quoted 8 lines
kernel/sched/core.c need_resched: · line 2453
Quoted 5 lines
kernel/sched/core.c +5
-0+raw_spin_lock_irq (&rq->lock); + rq->nr_iowait--; +raw_spin_unlock_irq (&rq->lock); + } +It seems that with this solution
rq->nr_iowaitis only ever modified locally. Can't we just disable irqs for rq->nr_iowait-- ?
Not really. rq holds the old value, which was used when this task called context_switch() in the past.
IOW, if a task T does io_schedule() on CPU_0, then cpu_of(rq) is still 0 after context_switch(), even if T runs on another cpu.
Quoted 1 line
Also if this is only updated locally,
Unfortunately, I don't see how we can do this.
Oleg.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Frederic WeisbeckerQuoted 28 lines · Frederic Weisbecker
Quoted 27 lines · Oleg Nesterov
Quoted 15 lines · Frederic Weisbecker
Quoted 14 lines · Oleg Nesterov
kernel/time/tick-sched.c u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) · line 520
Quoted 9 lines
kernel/time/tick-sched.c +9
-0+do { + seq =read_seqcount_begin (&ts->sleeptime_seq); +if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +ktime_t delta =ktime_sub (now, ts->idle_entrytime); + iowait =ktime_add (ts->iowait_sleeptime, delta); + }else { + iowait = ts->iowait_sleeptime; + } + }while (read_seqcount_retry (&ts->sleeptime_seq, seq));Unless I missread this patch, this is still racy a bit.
Suppose it is called on
CPU_0and cpu == 1. Suppose thatts->idle_active== T andnr_iowait_cpu(cpu)== 1.Quoted 8 lines
Quoted 6 lines
So we return
iowait_sleeptime+ delta.Suppose that we call
get_cpu_iowait_time_us()again. By this time the task which incremented ->nr_iowait can be woken up on another CPU, and it can doatomic_dec(rq->nr_iowait). So the next time we returniowait_sleeptime, and this is not monotonic again.Hmm, by the time it decrements
nr_iowait, it returned fromschedule()and so idle had flushed the pending iowait sleeptime.Suppose a task does
io_schedule()onCPU_0, and increments the counter. This CPU becomes idle andnr_iowait_cpu(0)== 1.
Quoted 7 lines
Quoted 6 lines
Then this task is woken up, but
try_to_wake_up()selects another CPU != 0.It returns from
schedule()and decrements the same counter, it doesn't do raw_rq/etc again.nr_iowait_cpu(0)becomes 0.In fact the task can even migrate to another CPU right after
raw_rq().Ah I see now. So that indeed yet another race.
I am sorry for chiming in late.
That precisely the race I described here:
https://lkml.org/lkml/2013/7/2/3
I should have been more concise in my explanation. I apologize.
Quoted 10 lines
Should we flush that iowait to the src CPU? But then it means we must handle concurrent updates to
iowait_sleeptime,idle_sleeptimefrom the migration code and from idle enter / exit.So I fear we need a seqlock.
Or we can live with that and still account the whole idle time slept until
tick_nohz_stop_idle()to iowait if we calledtick_nohz_start_idle()withnr_iowait> 0. All we need is just a new field in ts-> that records on which state we entered idle.
Another approach could be to shadow ->iowait_sleeptime as suggested here: https://lkml.org/lkml/2013/7/2/165
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 37 lines · Fernando Luis Vázquez Cao
Quoted 28 lines · Frederic Weisbecker
Quoted 27 lines · Oleg Nesterov
Quoted 15 lines · Frederic Weisbecker
Quoted 14 lines · Oleg Nesterov
kernel/time/tick-sched.c u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) · line 520
Quoted 9 lines
kernel/time/tick-sched.c +9
-0+do { + seq =read_seqcount_begin (&ts->sleeptime_seq); +if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +ktime_t delta =ktime_sub (now, ts->idle_entrytime); + iowait =ktime_add (ts->iowait_sleeptime, delta); + }else { + iowait = ts->iowait_sleeptime; + } + }while (read_seqcount_retry (&ts->sleeptime_seq, seq));Unless I missread this patch, this is still racy a bit.
Suppose it is called on
CPU_0and cpu == 1. Suppose thatts->idle_active== T andnr_iowait_cpu(cpu)== 1.Quoted 8 lines
Quoted 6 lines
So we return
iowait_sleeptime+ delta.Suppose that we call
get_cpu_iowait_time_us()again. By this time the task which incremented ->nr_iowait can be woken up on another CPU, and it can doatomic_dec(rq->nr_iowait). So the next time we returniowait_sleeptime, and this is not monotonic again.Hmm, by the time it decrements
nr_iowait, it returned fromschedule()and so idle had flushed the pending iowait sleeptime.Suppose a task does
io_schedule()onCPU_0, and increments the counter. This CPU becomes idle andnr_iowait_cpu(0)== 1.Quoted 7 lines
Quoted 6 lines
Then this task is woken up, but
try_to_wake_up()selects another CPU != 0.It returns from
schedule()and decrements the same counter, it doesn't do raw_rq/etc again.nr_iowait_cpu(0)becomes 0.In fact the task can even migrate to another CPU right after
raw_rq().Ah I see now. So that indeed yet another race.
Quoted 7 lines
I am sorry for chiming in late.
That precisely the race I described here:
https://lkml.org/lkml/2013/7/2/3
I should have been more concise in my explanation. I apologize.
Reading that again, the description was good. It's rather me who didn't read that not carefully enough :)
Quoted 7 lines
Quoted 7 lines
Should we flush that iowait to the src CPU? But then it means we must handle concurrent updates to
iowait_sleeptime,idle_sleeptimefrom the migration code and from idle enter / exit.So I fear we need a seqlock.
Quoted 7 lines
Quoted 4 lines
Or we can live with that and still account the whole idle time slept until
tick_nohz_stop_idle()to iowait if we calledtick_nohz_start_idle()withnr_iowait> 0. All we need is just a new field in ts-> that records on which state we entered idle.Another approach could be to shadow ->iowait_sleeptime as suggested here: https://lkml.org/lkml/2013/7/2/165
Hmm, yeah it seems that it would enforce the monotonicity but the wrong forward jumps due to bad ordering could still happen.
Oh and I realize you already suggested to account the iowait time on task migration more than one month ago. Grr I should really sit down before reading emails.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Oleg NesterovQuoted 34 lines · Oleg Nesterov
Thanks Frederic!
I'll try to read this series carefully later. Not that I think I can help, you certainly understand this much better.
Just one question below,
Quoted 10 lines · Frederic Weisbecker
kernel/time/tick-sched.c +0
-5@@ -499,12 +509,15 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time)if (last_update_time) *last_update_time =ktime_to_us (now); -if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { -ktime_t delta =ktime_sub (now, ts->idle_entrytime); - iowait =ktime_add (ts->iowait_sleeptime, delta); - }else { - iowait = ts->iowait_sleeptime;kernel/time/tick-sched.c u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) · line 520
Quoted 10 lines
kernel/time/tick-sched.c +9
-1- } +do { + seq =read_seqcount_begin (&ts->sleeptime_seq); +if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +ktime_t delta =ktime_sub (now, ts->idle_entrytime); + iowait =ktime_add (ts->iowait_sleeptime, delta); + }else { + iowait = ts->iowait_sleeptime; + } + }while (read_seqcount_retry (&ts->sleeptime_seq, seq));Unless I missread this patch, this is still racy a bit.
Suppose it is called on
CPU_0and cpu == 1. Suppose thatts->idle_active== T andnr_iowait_cpu(cpu)== 1.
Quoted 8 lines
So we return
iowait_sleeptime+ delta.Suppose that we call
get_cpu_iowait_time_us()again. By this time the task which incremented ->nr_iowait can be woken up on another CPU, and it can doatomic_dec(rq->nr_iowait). So the next time we returniowait_sleeptime, and this is not monotonic again.No?
OTOH, io_schedule() does:
atomic_inc(&rq->nr_iowait);
schedule();
atomic_dec(&rq->nr_iowait);How do we handle that when the task is migrated after it goes to sleep? I don't see the nr_iowait is decreased from the src CPU and increased on the dest CPU.
I don't either see that iowait tasks can't be migrated.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
2013/8/16 Frederic Weisbecker <[email protected]>:
Quoted 46 lines · Oleg Nesterov
Quoted 34 lines
Thanks Frederic!
I'll try to read this series carefully later. Not that I think I can help, you certainly understand this much better.
Just one question below,
Quoted 10 lines · Frederic Weisbecker
kernel/time/tick-sched.c +0
-5@@ -499,12 +509,15 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time)if (last_update_time) *last_update_time =ktime_to_us (now); -if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { -ktime_t delta =ktime_sub (now, ts->idle_entrytime); - iowait =ktime_add (ts->iowait_sleeptime, delta); - }else { - iowait = ts->iowait_sleeptime;kernel/time/tick-sched.c u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) · line 520
Quoted 10 lines
kernel/time/tick-sched.c +9
-1- } +do { + seq =read_seqcount_begin (&ts->sleeptime_seq); +if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +ktime_t delta =ktime_sub (now, ts->idle_entrytime); + iowait =ktime_add (ts->iowait_sleeptime, delta); + }else { + iowait = ts->iowait_sleeptime; + } + }while (read_seqcount_retry (&ts->sleeptime_seq, seq));Unless I missread this patch, this is still racy a bit.
Suppose it is called on
CPU_0and cpu == 1. Suppose thatts->idle_active== T andnr_iowait_cpu(cpu)== 1.Quoted 8 lines
So we return
iowait_sleeptime+ delta.Suppose that we call
get_cpu_iowait_time_us()again. By this time the task which incremented ->nr_iowait can be woken up on another CPU, and it can doatomic_dec(rq->nr_iowait). So the next time we returniowait_sleeptime, and this is not monotonic again.No?
OTOH,
io_schedule()does:
Quoted 8 lines
atomic_inc(&rq->nr_iowait); schedule(); atomic_dec(&rq->nr_iowait);How do we handle that when the task is migrated after it goes to sleep? I don't see the
nr_iowaitis decreased from the src CPU and increased on the dest CPU.I don't either see that iowait tasks can't be migrated.
My bad, The decrement happens on the src CPU anyway.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Frederic WeisbeckerQuoted 17 lines · Frederic Weisbecker
Quoted 6 lines · Oleg Nesterov
Unless I missread this patch, this is still racy a bit.
Suppose it is called on
CPU_0and cpu == 1. Suppose thatts->idle_active== T andnr_iowait_cpu(cpu)== 1.Quoted 8 lines
So we return
iowait_sleeptime+ delta.Suppose that we call
get_cpu_iowait_time_us()again. By this time the task which incremented ->nr_iowait can be woken up on another CPU, and it can doatomic_dec(rq->nr_iowait). So the next time we returniowait_sleeptime, and this is not monotonic again.No?
Quoted 7 lines
OTOH,
io_schedule()does:atomic_inc(&rq->nr_iowait); schedule(); atomic_dec(&rq->nr_iowait);How do we handle that when the task is migrated after it goes to sleep?
or even before it goes to sleep. This is what I meant.
Quoted 1 line
I don't either see that iowait tasks can't be migrated.
But probably this is fine? This is just the non-precise accounting.
But otoh, I agree. The whole idea about per-cpu nr_iowait looks a bit strange.
Oleg.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 26 lines · Oleg Nesterov
Quoted 17 lines · Frederic Weisbecker
Quoted 6 lines · Oleg Nesterov
Unless I missread this patch, this is still racy a bit.
Suppose it is called on
CPU_0and cpu == 1. Suppose thatts->idle_active== T andnr_iowait_cpu(cpu)== 1.Quoted 8 lines
So we return
iowait_sleeptime+ delta.Suppose that we call
get_cpu_iowait_time_us()again. By this time the task which incremented ->nr_iowait can be woken up on another CPU, and it can doatomic_dec(rq->nr_iowait). So the next time we returniowait_sleeptime, and this is not monotonic again.No?
Quoted 7 lines
OTOH,
io_schedule()does:atomic_inc(&rq->nr_iowait); schedule(); atomic_dec(&rq->nr_iowait);How do we handle that when the task is migrated after it goes to sleep?
Quoted 8 lines
or even before it goes to sleep. This is what I meant.
Quoted 1 line
I don't either see that iowait tasks can't be migrated.
But probably this is fine? This is just the non-precise accounting.
But otoh, I agree. The whole idea about per-cpu
nr_iowaitlooks a bit strange.
My bad, I thought it was doing:
inc(this_rq->nr_iowait)
schedule()
dec(this_rq->nr_iowait)
But it actually uses the src CPU all along.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
In reply to Frederic WeisbeckerQuoted 3 lines · Frederic Weisbecker
This fixes a reported bug where non-monotonic sleeptime stats are returned by
/proc/statwhen it is frequently read.
Plus it fixes the problems with 32bit machines reading u64 values.
But perhaps the changelog should mention other races we discussed.
Oleg.
Re: [PATCH 2/4] nohz: Synchronize sleep time stats with seqlock
Quoted 6 lines · Oleg Nesterov
Quoted 3 lines · Frederic Weisbecker
This fixes a reported bug where non-monotonic sleeptime stats are returned by
/proc/statwhen it is frequently read.Plus it fixes the problems with 32bit machines reading u64 values.
Ah indeed.
Quoted 2 lines
But perhaps the changelog should mention other races we discussed.
Agreed, I'll detail that.
thanks.
Quoted 3 lines
Oleg.
PATCH 3/4 nohz: Consolidate sleep time stats read code
In reply to Frederic Weisbeckerget_cpu_idle_time_us() and get_cpu_iowait_time_us() mostly share the same code. Lets consolidate both implementations.
--- kernel/time/tick-sched.c | 76 ++++++++++++++++++++-------------------------- 1 files changed, 33 insertions(+), 43 deletions(-)
kernel/time/tick-sched.c +33 -43
@@ -440,24 +440,10 @@ static ktime_t tick_nohz_start_idle(int cpu, struct tick_sched *ts)return now; } -/** -* get_cpu_idle_time_us - get the total idle time of a cpu -* @cpu: CPU number to query -* @last_update_time: variable to store update time in. Do not update -* counters if NULL. -* -* Return the cummulative idle time (since boot) for a given -* CPU, in microseconds. -* -* This time is measured via accounting rather than sampling, -* and is as accurate as ktime_get() is. -* -* This function returns -1 if NOHZ is not enabled. -*/ -u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time) +static u64 get_cpu_sleep_time_us (int cpu,bool io,u64 *last_update_time) {struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); -ktime_t now, idle; +ktime_t now, sleep;unsigned int seq;if (!tick_nohz_enabled)
@@ -469,16 +455,41 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time)do { seq =read_seqcount_begin (&ts->sleeptime_seq); -if (ts->idle_active && !nr_iowait_cpu (cpu)) { +if (io) + sleep = ts->iowait_sleeptime; +else + sleep = ts->idle_sleeptime; + +if (ts->idle_active) +continue ; + +if ((io &&nr_iowait_cpu (cpu)) || (!io && !nr_iowait_cpu (cpu))) {ktime_t delta =ktime_sub (now, ts->idle_entrytime); - idle =ktime_add (ts->idle_sleeptime, delta); - }else { - idle = ts->idle_sleeptime; + sleep =ktime_add (sleep, delta); } }while (read_seqcount_retry (&ts->sleeptime_seq, seq)); -return ktime_to_us (idle); +return ktime_to_us (sleep); + +} +/** +* get_cpu_idle_time_us - get the total idle time of a cpu +* @cpu: CPU number to query +* @last_update_time: variable to store update time in. Do not update +* counters if NULL. +* +* Return the cummulative idle time (since boot) for a given +* CPU, in microseconds. +* +* This time is measured via accounting rather than sampling, +* and is as accurate as ktime_get() is. +* +* This function returns -1 if NOHZ is not enabled. +*/ +u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time) +{ +return get_cpu_sleep_time_us (cpu, false, last_update_time); }EXPORT_SYMBOL_GPL (get_cpu_idle_time_us);
@@ -498,28 +509,7 @@ EXPORT_SYMBOL_GPL(get_cpu_idle_time_us);*/ u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time) { -struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); -ktime_t now, iowait; -unsigned int seq; - -if (!tick_nohz_enabled) -return -1 ; - - now =ktime_get (); -if (last_update_time) - *last_update_time =ktime_to_us (now); - -do { - seq =read_seqcount_begin (&ts->sleeptime_seq); -if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { -ktime_t delta =ktime_sub (now, ts->idle_entrytime); - iowait =ktime_add (ts->iowait_sleeptime, delta); - }else { - iowait = ts->iowait_sleeptime; - } - }while (read_seqcount_retry (&ts->sleeptime_seq, seq)); - -return ktime_to_us (iowait); +return get_cpu_sleep_time_us (cpu, true, last_update_time); }EXPORT_SYMBOL_GPL (get_cpu_iowait_time_us);
Signature · 1 line
1.7.5.4
Re: [PATCH 3/4] nohz: Consolidate sleep time stats read code
Quoted 3 lines · Frederic Weisbecker
get_cpu_idle_time_us()andget_cpu_iowait_time_us()mostly share the same code. Lets consolidate both implementations.
Personally I like every patch which consolidates the code ;)
Quoted 9 lines
kernel/time/tick-sched.c +6
-1do { seq = read_seqcount_begin(&ts->sleeptime_seq); - if (ts->idle_active && !nr_iowait_cpu(cpu)) { + if (io) + sleep = ts->iowait_sleeptime; + else + sleep = ts->idle_sleeptime; + + if (ts->idle_active)
kernel/time/tick-sched.c u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time) · line 470
Quoted 10 lines
kernel/time/tick-sched.c +4
-3+continue ; + +if ((io &&nr_iowait_cpu (cpu)) || (!io && !nr_iowait_cpu (cpu))) {ktime_t delta =ktime_sub (now, ts->idle_entrytime); - idle =ktime_add (ts->idle_sleeptime, delta); - }else { - idle = ts->idle_sleeptime; + sleep =ktime_add (sleep, delta); } }while (read_seqcount_retry (&ts->sleeptime_seq, seq));
Cosmetic/subjective, feel free to ignore. but perhaps
if (ts->idle_active && io == !!nr_iowait_cpu(cpu)) {
...
}looks more understandable.
Oleg.
Re: [PATCH 3/4] nohz: Consolidate sleep time stats read code
Quoted 29 lines · Oleg Nesterov
Quoted 3 lines · Frederic Weisbecker
get_cpu_idle_time_us()andget_cpu_iowait_time_us()mostly share the same code. Lets consolidate both implementations.Personally I like every patch which consolidates the code ;)
Quoted 9 lines
kernel/time/tick-sched.c +6
-1do { seq = read_seqcount_begin(&ts->sleeptime_seq); - if (ts->idle_active && !nr_iowait_cpu(cpu)) { + if (io) + sleep = ts->iowait_sleeptime; + else + sleep = ts->idle_sleeptime; + + if (ts->idle_active)kernel/time/tick-sched.c u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time) · line 470
Quoted 10 lines
kernel/time/tick-sched.c +4
-3+continue ; + +if ((io &&nr_iowait_cpu (cpu)) || (!io && !nr_iowait_cpu (cpu))) {ktime_t delta =ktime_sub (now, ts->idle_entrytime); - idle =ktime_add (ts->idle_sleeptime, delta); - }else { - idle = ts->idle_sleeptime; + sleep =ktime_add (sleep, delta); } }while (read_seqcount_retry (&ts->sleeptime_seq, seq));Cosmetic/subjective, feel free to ignore. but perhaps
Quoted 5 lines
if (ts->idle_active && io == !!nr_iowait_cpu(cpu)) { ... }looks more understandable.
Agreed, that's much better.
thanks!
Quoted 3 lines
Oleg.
PATCH 4/4 nohz: Convert a few places to use local per cpu accesses
In reply to Frederic WeisbeckerA few functions use remote per CPU access APIs when they deal with local values.
Just to the right conversion to improve performance, code readability and debug checks.
--- kernel/time/tick-sched.c | 10 ++++------ 1 files changed, 4 insertions(+), 6 deletions(-)
kernel/time/tick-sched.c +4 -6
@@ -395,11 +395,9 @@ __setup("nohz=", setup_tick_nohz);*/ static void tick_nohz_update_jiffies (ktime_t now) { -int cpu =smp_processor_id (); -struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu);unsigned long flags; - ts->idle_waketime = now; +__this_cpu_write (tick_cpu_sched.idle_waketime, now);local_irq_save (flags);tick_do_update_jiffies64 (now);
@@ -410,7 +408,7 @@ static void tick_nohz_update_jiffies(ktime_t now)static void tick_nohz_stop_idle (int cpu,ktime_t now) { -struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);ktime_t delta;/* Updates the per cpu time idle statistics counters */
@@ -901,7 +899,7 @@ static void tick_nohz_account_idle_ticks(struct tick_sched *ts)void tick_nohz_idle_exit (void ) {int cpu =smp_processor_id (); -struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);ktime_t now;local_irq_disable ();
@@ -1020,7 +1018,7 @@ static void tick_nohz_kick_tick(int cpu, ktime_t now)static inline void tick_check_nohz (int cpu) { -struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);ktime_t now;if (!ts->idle_active && !ts->tick_stopped)
Signature · 1 line
1.7.5.4
Peter Zijlstra
maintainerRe: [PATCH 4/4] nohz: Convert a few places to use local per cpu accesses
OK so these patches look ok to me -- didn't read in detail though.
kernel/time/tick-sched.c static void tick_nohz_update_jiffies(ktime_t now) · line 411
Quoted 12 lines · Frederic Weisbecker
A few functions use remote per CPU access APIs when they deal with local values.
Just to the right conversion to improve performance, code readability and debug checks.
kernel/time/tick-sched.c +1
-1@@ -410,7 +408,7 @@ static void tick_nohz_update_jiffies(ktime_t now)static void tick_nohz_stop_idle (int cpu,ktime_t now) { -struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);
What's there from stopping someone from calling this with cpu != smp_processor_id() ?
kernel/time/tick-sched.c static void tick_nohz_kick_tick(int cpu, ktime_t now) · line 1024
Quoted 9 lines
kernel/time/tick-sched.c +1
-1@@ -1020,7 +1018,7 @@ static void tick_nohz_kick_tick(int cpu, ktime_t now)static inline void tick_check_nohz (int cpu) { -struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);ktime_t now;if (!ts->idle_active && !ts->tick_stopped)
idem
Re: [PATCH 4/4] nohz: Convert a few places to use local per cpu accesses
Quoted 11 lines · Peter Zijlstra
OK so these patches look ok to me -- didn't read in detail though.
Quoted 8 lines · Frederic Weisbecker
A few functions use remote per CPU access APIs when they deal with local values.
Just to the right conversion to improve performance, code readability and debug checks.
(quoted diff) +0
-0@@ -410,7 +408,7 @@ static void tick_nohz_update_jiffies(ktime_t now)
Quoted 7 lines
kernel/time/tick-sched.c static void tick_nohz_update_jiffies(ktime_t now) · line 411
Quoted 4 lines
kernel/time/tick-sched.c +1
-1static void tick_nohz_stop_idle (int cpu,ktime_t now) { -struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);What's there from stopping someone from calling this with cpu !=
smp_processor_id()?
May be I should remove the cpu arg yeah.
Quoted 9 lines
Quoted 9 lines
kernel/time/tick-sched.c +1
-1@@ -1020,7 +1018,7 @@ static void tick_nohz_kick_tick(int cpu, ktime_t now)static inline void tick_check_nohz (int cpu) { -struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);ktime_t now;
Quoted 3 lines
kernel/time/tick-sched.c static void tick_nohz_kick_tick(int cpu, ktime_t now) · line 1024
Quoted 1 line
kernel/time/tick-sched.c +0
-0if (!ts->idle_active && !ts->tick_stopped)idem
Yep same here. Will fix.
Re: [PATCH 4/4] nohz: Convert a few places to use local per cpu accesses
In reply to Peter ZijlstraQuoted 12 lines · Peter Zijlstra
OK so these patches look ok to me -- didn't read in detail though.
Quoted 8 lines · Frederic Weisbecker
A few functions use remote per CPU access APIs when they deal with local values.
Just to the right conversion to improve performance, code readability and debug checks.
(quoted diff) +0
-0@@ -410,7 +408,7 @@ static void tick_nohz_update_jiffies(ktime_t now)
Quoted 7 lines
kernel/time/tick-sched.c static void tick_nohz_update_jiffies(ktime_t now) · line 411
Quoted 4 lines
kernel/time/tick-sched.c +1
-1static void tick_nohz_stop_idle (int cpu,ktime_t now) { -struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);What's there from stopping someone from calling this with cpu !=
smp_processor_id()?
I _guess_ this should not happen, but in this case we should probably remove the "cpu" argument. And smp_processor_id() in tick_nohz_idle_exit().
tick_check_idle/tick_check_nohz doesn't need "int cpu" too, it seems.
Oleg.
Re: [PATCH 4/4] nohz: Convert a few places to use local per cpu accesses
Quoted 21 lines · Oleg Nesterov
Quoted 12 lines · Peter Zijlstra
OK so these patches look ok to me -- didn't read in detail though.
Quoted 8 lines · Frederic Weisbecker
A few functions use remote per CPU access APIs when they deal with local values.
Just to the right conversion to improve performance, code readability and debug checks.
(quoted diff) +0
-0@@ -410,7 +408,7 @@ static void tick_nohz_update_jiffies(ktime_t now)Quoted 7 lines
kernel/time/tick-sched.c static void tick_nohz_update_jiffies(ktime_t now) · line 411
Quoted 4 lines
kernel/time/tick-sched.c +1
-1static void tick_nohz_stop_idle (int cpu,ktime_t now) { -struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);What's there from stopping someone from calling this with cpu !=
smp_processor_id()?
Quoted 4 lines
I _guess_ this should not happen, but in this case we should probably remove the "cpu" argument. And
smp_processor_id()intick_nohz_idle_exit().tick_check_idle/tick_check_nohz doesn't need "int cpu" too, it seems.
Yep, indeed it seems right to remove the cpu argument to prevent from further calls to these APIs on remote CPUs, which would otherwise result in random disaster.
Quoted 3 lines
Oleg.
While at it.
I do not also understand the cpu_online() checks in fs/proc/stat.c.
OK, I agree, if cpu is offline it should not participate in cpu summary. But if it goes offline, why it should switch from ->iowait_sleeptime + cpustat[CPUTIME_IDLE] as it seen by /proc/stat?
This can be another source of "idle goes backward", no?
IOW. Ignoring the other problems we have, perhaps something like below makes sense?
Oleg.
fs/proc/stat.c +4 -6
@@ -45,10 +45,9 @@ static cputime64_t get_iowait_time(int cstatic u64 get_idle_time (int cpu) { -u64 idle, idle_time =-1ULL ; +u64 idle, idle_time; -if (cpu_online (cpu)) - idle_time =get_cpu_idle_time_us (cpu,NULL ); + idle_time =get_cpu_idle_time_us (cpu,NULL );if (idle_time ==-1ULL )/* !NO_HZ or cpu offline so we can rely on cpustat.idle */
@@ -61,10 +60,9 @@ static u64 get_idle_time(int cpu)static u64 get_iowait_time (int cpu) { -u64 iowait, iowait_time =-1ULL ; +u64 iowait, iowait_time; -if (cpu_online (cpu)) - iowait_time =get_cpu_iowait_time_us (cpu,NULL ); + iowait_time =get_cpu_iowait_time_us (cpu,NULL );if (iowait_time ==-1ULL )/* !NO_HZ or cpu offline so we can rely on cpustat.iowait */
kernel/time/tick-sched.c +2 -2
@@ -477,7 +477,7 @@ u64 get_cpu_idle_time_us(int cpu, u64 *lupdate_ts_time_stats (cpu, ts, now, last_update_time); idle = ts->idle_sleeptime; }else { -if (ts->idle_active && !nr_iowait_cpu (cpu)) { +if (ts->idle_active &&cpu_online (cpu) && !nr_iowait_cpu (cpu)) {ktime_t delta =ktime_sub (now, ts->idle_entrytime); idle =ktime_add (ts->idle_sleeptime, delta);
@@ -518,7 +518,7 @@ u64 get_cpu_iowait_time_us(int cpu, u64update_ts_time_stats (cpu, ts, now, last_update_time); iowait = ts->iowait_sleeptime; }else { -if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +if (ts->idle_active &&cpu_online (cpu) &&nr_iowait_cpu (cpu)) {ktime_t delta =ktime_sub (now, ts->idle_entrytime); iowait =ktime_add (ts->iowait_sleeptime, delta);
Peter Zijlstra
maintainerQuoted 4 lines · Oleg Nesterov
While at it.
I do not also understand the
cpu_online()checks infs/proc/stat.c.
Quoted 8 lines
OK, I agree, if cpu is offline it should not participate in cpu summary. But if it goes offline, why it should switch from ->iowait_sleeptime + cpustat[CPUTIME_IDLE] as it seen by
/proc/stat?This can be another source of "idle goes backward", no?
IOW. Ignoring the other problems we have, perhaps something like below makes sense?
Agreed, however
Quoted 13 lines
+++
x/kernel/time/tick-sched.ckernel/time/tick-sched.c +1
-1@@ -477,7 +477,7 @@ u64 get_cpu_idle_time_us(int cpu, u64 *lupdate_ts_time_stats (cpu, ts, now, last_update_time); idle = ts->idle_sleeptime; }else { -if (ts->idle_active && !nr_iowait_cpu (cpu)) { +if (ts->idle_active &&cpu_online (cpu) && !nr_iowait_cpu (cpu)) {ktime_t delta =ktime_sub (now, ts->idle_entrytime); idle =ktime_add (ts->idle_sleeptime, delta);
kernel/time/tick-sched.c u64 get_cpu_iowait_time_us(int cpu, u64 · line 524
Quoted 8 lines
kernel/time/tick-sched.c +1
-1@@ -518,7 +518,7 @@ u64 get_cpu_iowait_time_us(int cpu, u64update_ts_time_stats (cpu, ts, now, last_update_time); iowait = ts->iowait_sleeptime; }else { -if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +if (ts->idle_active &&cpu_online (cpu) &&nr_iowait_cpu (cpu)) {ktime_t delta =ktime_sub (now, ts->idle_entrytime); iowait =ktime_add (ts->iowait_sleeptime, delta);
That's still mighty odd, but I guess that's in part due to the whacky semantics. We could simply transfer any open nr_iowait to the cpu doing the hotplug and then we have offline cpus that have nr_iowait == 0 and the above becomes simpler again.
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Oleg Nesterov
While at it.
I do not also understand the
cpu_online()checks infs/proc/stat.c.OK, I agree, if cpu is offline it should not participate in cpu summary. But if it goes offline, why it should switch from ->iowait_sleeptime + cpustat[CPUTIME_IDLE] as it seen by
/proc/stat?
Quoted 7 lines
Quoted 4 lines
This can be another source of "idle goes backward", no?
IOW. Ignoring the other problems we have, perhaps something like below makes sense?
Agreed, however
OK, good,
Quoted 18 lines
Quoted 18 lines
+++
x/kernel/time/tick-sched.ckernel/time/tick-sched.c +2
-2@@ -477,7 +477,7 @@ u64 get_cpu_idle_time_us(int cpu, u64 *lupdate_ts_time_stats (cpu, ts, now, last_update_time); idle = ts->idle_sleeptime; }else { -if (ts->idle_active && !nr_iowait_cpu (cpu)) { +if (ts->idle_active &&cpu_online (cpu) && !nr_iowait_cpu (cpu)) {ktime_t delta =ktime_sub (now, ts->idle_entrytime); idle =ktime_add (ts->idle_sleeptime, delta);@@ -518,7 +518,7 @@ u64 get_cpu_iowait_time_us(int cpu, u64update_ts_time_stats (cpu, ts, now, last_update_time); iowait = ts->iowait_sleeptime; }else { -if (ts->idle_active &&nr_iowait_cpu (cpu) >0 ) { +if (ts->idle_active &&cpu_online (cpu) &&nr_iowait_cpu (cpu)) {ktime_t delta =ktime_sub (now, ts->idle_entrytime);
Quoted 7 lines
kernel/time/tick-sched.c u64 get_cpu_iowait_time_us(int cpu, u64 · line 524
Quoted 2 lines
kernel/time/tick-sched.c +0
-0iowait =ktime_add (ts->iowait_sleeptime, delta);That's still mighty odd, but I guess that's in part due to the whacky semantics. We could simply transfer any open
nr_iowaitto the cpu doing the hotplug and then we have offline cpus that havenr_iowait== 0 and the above becomes simpler again.
This won't help get_cpu_idle_time_us().
But anyway we should fix other problems first, then think about this change. I just wanted to verify that I didn't miss something and this iowait_sleeptime -> CPUTIME_IDLE switch is indeed wrong.
Oleg.