sched/cputime: Preserve iowait accounting across waking interrupts in dyntick-idle
PATCH 1/1 sched/cputime: Preserve iowait accounting across waking interrupts in dyntick-idle
In liburing's test suite, the test/iowait.t test case intermittently fails with:
iowait diff too small: 48
test 1 failed
on tickless systems where IRQ time accounting (CONFIG_IRQ_TIME_ACCOUNTING) is not enabled.
Commit 7198e3927a14 ("sched/cputime: Handle idle irqtime gracefully") optimized dyntick-idle cputime accounting by avoiding pausing idle accounting on IRQ entry when IRQ time accounting is disabled.
However, this causes iowait time to be erroneously accounted as CPUTIME_IDLE if the task waiting for I/O is woken up inside the interrupt handler (such as the 1-second timeout hrtimer in liburing's test/iowait.t):
1. In test/iowait.t, a read request is submitted on an empty pipe and io_uring_wait_cqe_timeout() is called with a 1-second timeout. Because current->in_iowait is set, rq->nr_iowait on the CPU is incremented to 1. When the CPU enters tickless idle, the periodic scheduler tick is stopped.
2. When the 1-second timeout expires, the hrtimer interrupt fires. Because irqtime is disabled, kcpustat_irq_enter() does not pause idle accounting.
3. Inside the interrupt handler, io_cqring_timer_wakeup() calls wake_up_process(). In ttwu_do_activate(), waking up the task immediately decrements rq->nr_iowait from 1 to 0 before the CPU has exited idle.
4. When the CPU returns from the interrupt and calls tick_nohz_idle_exit() -> kcpustat_idle_stop() to settle the elapsed tickless idle duration, it checks:
iowait = nr_iowait_cpu(smp_processor_id()) > 0;
Because rq->nr_iowait was already decremented by ttwu_do_activate(), this evaluates to false. As a result, the entire 1-second idle duration spent waiting for I/O is credited to CPUTIME_IDLE instead of CPUTIME_IOWAIT, causing diff < 50 and failing the test.
Fix this by checking nr_iowait_cpu() in kcpustat_irq_enter(). If the CPU is currently in iowait, pause and settle the elapsed time to CPUTIME_IOWAIT before the interrupt handler runs. On kcpustat_irq_exit(), resume accounting if it was paused.
--- kernel/sched/cputime.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
kernel/sched/cputime.c +7 -2
@@ -496,7 +496,8 @@ void kcpustat_irq_enter(u64 now)struct kernel_cpustat *kc = kcpustat_this_cpu;if (!vtime_generic_enabled_this_cpu () && - (irqtime_enabled () ||vtime_accounting_enabled_this_cpu ())) + (irqtime_enabled () ||vtime_accounting_enabled_this_cpu () || +nr_iowait_cpu (smp_processor_id ()) >0 ))kcpustat_idle_stop (kc, now); }
@@ -510,9 +511,13 @@ void kcpustat_irq_exit(u64 now)* need to pause nohz accounting. Resume nohz accounting as long * as the irqtime config is enabled to handle case where irqtime * accounting got runtime disabled in the middle of an IRQ. +* +* Also resume accounting if it was paused on IRQ enter due to +* iowait. */ if (!vtime_generic_enabled_this_cpu () && - (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING ) ||vtime_accounting_enabled_this_cpu ())) + (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING ) ||vtime_accounting_enabled_this_cpu () || + !kc->idle_elapse))kcpustat_idle_start (kc, now); }
Signature · 1 line
2.55.0
Peter Zijlstra
maintainerQuoted 35 lines · Wei Gao
In liburing's test suite, the test/iowait.t test case intermittently fails with:
iowait diff too small: 48
test 1 failed
on tickless systems where IRQ time accounting (CONFIG_IRQ_TIME_ACCOUNTING) is not enabled.Commit
7198e3927a14("sched/cputime: Handle idle irqtime gracefully") optimized dyntick-idle cputime accounting by avoiding pausing idle accounting on IRQ entry when IRQ time accounting is disabled.However, this causes iowait time to be erroneously accounted as
CPUTIME_IDLEif the task waiting for I/O is woken up inside the interrupt handler (such as the 1-second timeout hrtimer in liburing's test/iowait.t):1. In test/iowait.t, a read request is submitted on an empty pipe and
io_uring_wait_cqe_timeout()is called with a 1-second timeout. Becausecurrent->in_iowaitis set,rq->nr_iowaiton the CPU is incremented to 1. When the CPU enters tickless idle, the periodic scheduler tick is stopped.
2. When the 1-second timeout expires, the hrtimer interrupt fires. Because irqtime is disabled,kcpustat_irq_enter()does not pause idle accounting.
3. Inside the interrupt handler,io_cqring_timer_wakeup()callswake_up_process(). Inttwu_do_activate(), waking up the task immediately decrementsrq->nr_iowaitfrom 1 to 0 before the CPU has exited idle.
4. When the CPU returns from the interrupt and callstick_nohz_idle_exit()->kcpustat_idle_stop()to settle the elapsed tickless idle duration, it checks:iowait = nr_iowait_cpu(smp_processor_id()) > 0;Because
rq->nr_iowaitwas already decremented byttwu_do_activate(), this evaluates to false. As a result, the entire 1-second idle duration spent waiting for I/O is credited toCPUTIME_IDLEinstead ofCPUTIME_IOWAIT, causing diff < 50 and failing the test.
Quoted 4 lines
Fix this by checking
nr_iowait_cpu()inkcpustat_irq_enter(). If the CPU is currently in iowait, pause and settle the elapsed time toCPUTIME_IOWAITbefore the interrupt handler runs. Onkcpustat_irq_exit(), resume accounting if it was paused.
Using nr_iowait_cpu() is fundamentally never a fix. It is rolling a dice and liking random numbers. See the comment that goes with that function.
Quoted 38 lines
Fixes 7198e3927a14 ("sched/cputime: Handle idle irqtime gracefully")Signed-off-by Wei Gao <[email protected]>--- kernel/sched/cputime.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)kernel/sched/cputime.c +7
-2@@ -496,7 +496,8 @@ void kcpustat_irq_enter(u64 now)struct kernel_cpustat *kc = kcpustat_this_cpu;if (!vtime_generic_enabled_this_cpu () && - (irqtime_enabled () ||vtime_accounting_enabled_this_cpu ())) + (irqtime_enabled () ||vtime_accounting_enabled_this_cpu () || +nr_iowait_cpu (smp_processor_id ()) >0 ))kcpustat_idle_stop (kc, now); }@@ -510,9 +511,13 @@ void kcpustat_irq_exit(u64 now)* need to pause nohz accounting. Resume nohz accounting as long * as the irqtime config is enabled to handle case where irqtime * accounting got runtime disabled in the middle of an IRQ. +* +* Also resume accounting if it was paused on IRQ enter due to +* iowait. */ if (!vtime_generic_enabled_this_cpu () && - (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING ) ||vtime_accounting_enabled_this_cpu ())) + (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING ) ||vtime_accounting_enabled_this_cpu () || + !kc->idle_elapse))kcpustat_idle_start (kc, now); }Signature · 1 line
2.55.0
Quoted 36 lines · Peter Zijlstra
Quoted 35 lines · Wei Gao
In liburing's test suite, the test/iowait.t test case intermittently fails with:
iowait diff too small: 48
test 1 failed
on tickless systems where IRQ time accounting (CONFIG_IRQ_TIME_ACCOUNTING) is not enabled.Commit
7198e3927a14("sched/cputime: Handle idle irqtime gracefully") optimized dyntick-idle cputime accounting by avoiding pausing idle accounting on IRQ entry when IRQ time accounting is disabled.However, this causes iowait time to be erroneously accounted as
CPUTIME_IDLEif the task waiting for I/O is woken up inside the interrupt handler (such as the 1-second timeout hrtimer in liburing's test/iowait.t):1. In test/iowait.t, a read request is submitted on an empty pipe and
io_uring_wait_cqe_timeout()is called with a 1-second timeout. Becausecurrent->in_iowaitis set,rq->nr_iowaiton the CPU is incremented to 1. When the CPU enters tickless idle, the periodic scheduler tick is stopped.
2. When the 1-second timeout expires, the hrtimer interrupt fires. Because irqtime is disabled,kcpustat_irq_enter()does not pause idle accounting.
3. Inside the interrupt handler,io_cqring_timer_wakeup()callswake_up_process(). Inttwu_do_activate(), waking up the task immediately decrementsrq->nr_iowaitfrom 1 to 0 before the CPU has exited idle.
4. When the CPU returns from the interrupt and callstick_nohz_idle_exit()->kcpustat_idle_stop()to settle the elapsed tickless idle duration, it checks:iowait = nr_iowait_cpu(smp_processor_id()) > 0;Because
rq->nr_iowaitwas already decremented byttwu_do_activate(), this evaluates to false. As a result, the entire 1-second idle duration spent waiting for I/O is credited toCPUTIME_IDLEinstead ofCPUTIME_IOWAIT, causing diff < 50 and failing the test.
Quoted 8 lines
Quoted 4 lines
Fix this by checking
nr_iowait_cpu()inkcpustat_irq_enter(). If the CPU is currently in iowait, pause and settle the elapsed time toCPUTIME_IOWAITbefore the interrupt handler runs. Onkcpustat_irq_exit(), resume accounting if it was paused.Using
nr_iowait_cpu()is fundamentally never a fix. It is rolling a dice and liking random numbers. See the comment that goes with that function.
Speaking of, that comment is stale, menu doesn't use that heuristic anymore. FWIW the remaining aggregated nr_iowait users seem to be accounting/reporting. (cpufreq still uses it, but it comes directly from p->in_iowait). Given they are exported (/proc/stat procs_blocked and something in s390) I don't see how we can remove this entirely but at least nr_iowait_cpu() as opposed to cpu_has_iowait() could be refactored. The issues are obviously the same still. I'll send a patch.
Peter, if you see a path to ever retire iowait (or at least the CPU iowait for now), I'm more than happy to sink some time into it...
Peter Zijlstra
maintainerQuoted 4 lines · Christian Loehle
Quoted 3 lines
Using
nr_iowait_cpu()is fundamentally never a fix. It is rolling a dice and liking random numbers. See the comment that goes with that function.Speaking of, that comment is stale, menu doesn't use that heuristic anymore.
\o/
Quoted 2 lines
FWIW the remaining aggregated
nr_iowaitusers seem to be accounting/reporting. (cpufreq still uses it, but it comes directly fromp->in_iowait).
I see: get_cpu_iowait_time_us() using nr_cpuwait_cpu(), which in turn is used for both thermal and cpufreq.
Quoted 4 lines
Given they are exported (
/proc/statprocs_blockedand something in s390) I don't see how we can remove this entirely but at leastnr_iowait_cpu()as opposed tocpu_has_iowait()could be refactored. The issues are obviously the same still. I'll send a patch.
Right.
Quoted 2 lines
Peter, if you see a path to ever retire iowait (or at least the CPU iowait for now), I'm more than happy to sink some time into it...
Given the exposure in /proc I'm not sure we can *ever* fully get rid of the per-cpu usage of iowait, but yeah, reducing in-kernel consumers of this number might be somewhat feasible.
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Christian Loehle
Quoted 3 lines
Using
nr_iowait_cpu()is fundamentally never a fix. It is rolling a dice and liking random numbers. See the comment that goes with that function.Speaking of, that comment is stale, menu doesn't use that heuristic anymore.
Quoted 7 lines
\o/
Quoted 2 lines
FWIW the remaining aggregated
nr_iowaitusers seem to be accounting/reporting. (cpufreq still uses it, but it comes directly fromp->in_iowait).I see:
get_cpu_iowait_time_us()usingnr_cpuwait_cpu(), which in turn is used for both thermal and cpufreq.
thermal uses it with io_is_busy=0, so that's okay, ondemand has it sysfs-exposed with a default of 0 except for
return (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL && boot_cpu_data.x86_vfm >= INTEL_PENTIUM_PRO);
:/
If that were the only users I'm sure there would be ways around it though.
Quoted 14 lines
Quoted 5 lines
Given they are exported (
/proc/statprocs_blockedand something in s390) I don't see how we can remove this entirely but at leastnr_iowait_cpu()as opposed tocpu_has_iowait()could be refactored. The issues are obviously the same still. I'll send a patch.Right.
Quoted 2 lines
Peter, if you see a path to ever retire iowait (or at least the CPU iowait for now), I'm more than happy to sink some time into it...
Given the exposure in /proc I'm not sure we can *ever* fully get rid of the per-cpu usage of iowait, but yeah, reducing in-kernel consumers of this number might be somewhat feasible.
Quoted 36 lines · Peter Zijlstra
Quoted 35 lines · Wei Gao
In liburing's test suite, the test/iowait.t test case intermittently fails with:
iowait diff too small: 48
test 1 failed
on tickless systems where IRQ time accounting (CONFIG_IRQ_TIME_ACCOUNTING) is not enabled.Commit
7198e3927a14("sched/cputime: Handle idle irqtime gracefully") optimized dyntick-idle cputime accounting by avoiding pausing idle accounting on IRQ entry when IRQ time accounting is disabled.However, this causes iowait time to be erroneously accounted as
CPUTIME_IDLEif the task waiting for I/O is woken up inside the interrupt handler (such as the 1-second timeout hrtimer in liburing's test/iowait.t):1. In test/iowait.t, a read request is submitted on an empty pipe and
io_uring_wait_cqe_timeout()is called with a 1-second timeout. Becausecurrent->in_iowaitis set,rq->nr_iowaiton the CPU is incremented to 1. When the CPU enters tickless idle, the periodic scheduler tick is stopped.
2. When the 1-second timeout expires, the hrtimer interrupt fires. Because irqtime is disabled,kcpustat_irq_enter()does not pause idle accounting.
3. Inside the interrupt handler,io_cqring_timer_wakeup()callswake_up_process(). Inttwu_do_activate(), waking up the task immediately decrementsrq->nr_iowaitfrom 1 to 0 before the CPU has exited idle.
4. When the CPU returns from the interrupt and callstick_nohz_idle_exit()->kcpustat_idle_stop()to settle the elapsed tickless idle duration, it checks:iowait = nr_iowait_cpu(smp_processor_id()) > 0;Because
rq->nr_iowaitwas already decremented byttwu_do_activate(), this evaluates to false. As a result, the entire 1-second idle duration spent waiting for I/O is credited toCPUTIME_IDLEinstead ofCPUTIME_IOWAIT, causing diff < 50 and failing the test.
Quoted 8 lines
Quoted 4 lines
Fix this by checking
nr_iowait_cpu()inkcpustat_irq_enter(). If the CPU is currently in iowait, pause and settle the elapsed time toCPUTIME_IOWAITbefore the interrupt handler runs. Onkcpustat_irq_exit(), resume accounting if it was paused.Using
nr_iowait_cpu()is fundamentally never a fix. It is rolling a dice and liking random numbers. See the comment that goes with that function.
Thanks for such quick feedback. I see your point since per-CPU iowait accounting on SMP is fundamentally flawed. It doesn't make sense to create any more patches for current iowait accounting.
For iowait.t test i guess need revalue it or simply skipped it.
Quoted 38 lines
Quoted 38 lines
Fixes 7198e3927a14 ("sched/cputime: Handle idle irqtime gracefully")Signed-off-by Wei Gao <[email protected]>--- kernel/sched/cputime.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)kernel/sched/cputime.c +7
-2@@ -496,7 +496,8 @@ void kcpustat_irq_enter(u64 now)struct kernel_cpustat *kc = kcpustat_this_cpu;if (!vtime_generic_enabled_this_cpu () && - (irqtime_enabled () ||vtime_accounting_enabled_this_cpu ())) + (irqtime_enabled () ||vtime_accounting_enabled_this_cpu () || +nr_iowait_cpu (smp_processor_id ()) >0 ))kcpustat_idle_stop (kc, now); }@@ -510,9 +511,13 @@ void kcpustat_irq_exit(u64 now)* need to pause nohz accounting. Resume nohz accounting as long * as the irqtime config is enabled to handle case where irqtime * accounting got runtime disabled in the middle of an IRQ. +* +* Also resume accounting if it was paused on IRQ enter due to +* iowait. */ if (!vtime_generic_enabled_this_cpu () && - (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING ) ||vtime_accounting_enabled_this_cpu ())) + (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING ) ||vtime_accounting_enabled_this_cpu () || + !kc->idle_elapse))kcpustat_idle_start (kc, now); }Signature · 1 line
2.55.0
Quoted 29 lines · Wei Gao
In liburing's test suite, the test/iowait.t test case intermittently fails with:
iowait diff too small: 48
test 1 failed
on tickless systems where IRQ time accounting (CONFIG_IRQ_TIME_ACCOUNTING) is not enabled.Commit
7198e3927a14("sched/cputime: Handle idle irqtime gracefully") optimized dyntick-idle cputime accounting by avoiding pausing idle accounting on IRQ entry when IRQ time accounting is disabled.However, this causes iowait time to be erroneously accounted as
CPUTIME_IDLEif the task waiting for I/O is woken up inside the interrupt handler (such as the 1-second timeout hrtimer in liburing's test/iowait.t):1. In test/iowait.t, a read request is submitted on an empty pipe and
io_uring_wait_cqe_timeout()is called with a 1-second timeout. Becausecurrent->in_iowaitis set,rq->nr_iowaiton the CPU is incremented to 1. When the CPU enters tickless idle, the periodic scheduler tick is stopped.
2. When the 1-second timeout expires, the hrtimer interrupt fires. Because irqtime is disabled,kcpustat_irq_enter()does not pause idle accounting.
3. Inside the interrupt handler,io_cqring_timer_wakeup()callswake_up_process(). Inttwu_do_activate(), waking up the task immediately decrementsrq->nr_iowaitfrom 1 to 0 before the CPU has exited idle.
4. When the CPU returns from the interrupt and callstick_nohz_idle_exit()->kcpustat_idle_stop()to settle the elapsed tickless idle duration, it checks:
Quoted 5 lines
iowait = nr_iowait_cpu(smp_processor_id()) > 0;Because
rq->nr_iowaitwas already decremented byttwu_do_activate(), this evaluates to false. As a result, the entire 1-second idle duration spent waiting for I/O is credited toCPUTIME_IDLEinstead ofCPUTIME_IOWAIT, causing diff < 50 and failing the test.
So reading the test, the reason this doesn't fail constantly (on a non-isolated machine) is the massive margins?
It pins the io_uring_wait_cqe_timeout() to CPU0 with either in_iowait=1 or in_iowait=0.
It then asserts after 1s that CPU0 got:
a) for in_iowait=0 at most +10 iowait units (<10% CPU time)
b) for in_iowait=1 at least +50 iowait units (>50% CPU time)
So a) would fail on too many unrelated iowaits, b) would fail on unrelated runnable tasks.
I don't think validating the per-task in_iowait setting via the CPU aggregate is really feasible FWIW.
Quoted 41 lines
Fix this by checking
nr_iowait_cpu()inkcpustat_irq_enter(). If the CPU is currently in iowait, pause and settle the elapsed time toCPUTIME_IOWAITbefore the interrupt handler runs. Onkcpustat_irq_exit(), resume accounting if it was paused.Fixes 7198e3927a14 ("sched/cputime: Handle idle irqtime gracefully")Signed-off-by Wei Gao <[email protected]>--- kernel/sched/cputime.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)kernel/sched/cputime.c +7
-2@@ -496,7 +496,8 @@ void kcpustat_irq_enter(u64 now)struct kernel_cpustat *kc = kcpustat_this_cpu;if (!vtime_generic_enabled_this_cpu () && - (irqtime_enabled () ||vtime_accounting_enabled_this_cpu ())) + (irqtime_enabled () ||vtime_accounting_enabled_this_cpu () || +nr_iowait_cpu (smp_processor_id ()) >0 ))kcpustat_idle_stop (kc, now); }@@ -510,9 +511,13 @@ void kcpustat_irq_exit(u64 now)* need to pause nohz accounting. Resume nohz accounting as long * as the irqtime config is enabled to handle case where irqtime * accounting got runtime disabled in the middle of an IRQ. +* +* Also resume accounting if it was paused on IRQ enter due to +* iowait. */ if (!vtime_generic_enabled_this_cpu () && - (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING ) ||vtime_accounting_enabled_this_cpu ())) + (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING ) ||vtime_accounting_enabled_this_cpu () || + !kc->idle_elapse))kcpustat_idle_start (kc, now); }
Quoted 36 lines · Christian Loehle
Quoted 29 lines · Wei Gao
In liburing's test suite, the test/iowait.t test case intermittently fails with:
iowait diff too small: 48
test 1 failed
on tickless systems where IRQ time accounting (CONFIG_IRQ_TIME_ACCOUNTING) is not enabled.Commit
7198e3927a14("sched/cputime: Handle idle irqtime gracefully") optimized dyntick-idle cputime accounting by avoiding pausing idle accounting on IRQ entry when IRQ time accounting is disabled.However, this causes iowait time to be erroneously accounted as
CPUTIME_IDLEif the task waiting for I/O is woken up inside the interrupt handler (such as the 1-second timeout hrtimer in liburing's test/iowait.t):1. In test/iowait.t, a read request is submitted on an empty pipe and
io_uring_wait_cqe_timeout()is called with a 1-second timeout. Becausecurrent->in_iowaitis set,rq->nr_iowaiton the CPU is incremented to 1. When the CPU enters tickless idle, the periodic scheduler tick is stopped.
2. When the 1-second timeout expires, the hrtimer interrupt fires. Because irqtime is disabled,kcpustat_irq_enter()does not pause idle accounting.
3. Inside the interrupt handler,io_cqring_timer_wakeup()callswake_up_process(). Inttwu_do_activate(), waking up the task immediately decrementsrq->nr_iowaitfrom 1 to 0 before the CPU has exited idle.
4. When the CPU returns from the interrupt and callstick_nohz_idle_exit()->kcpustat_idle_stop()to settle the elapsed tickless idle duration, it checks:Quoted 5 lines
iowait = nr_iowait_cpu(smp_processor_id()) > 0;Because
rq->nr_iowaitwas already decremented byttwu_do_activate(), this evaluates to false. As a result, the entire 1-second idle duration spent waiting for I/O is credited toCPUTIME_IDLEinstead ofCPUTIME_IOWAIT, causing diff < 50 and failing the test.
Quoted 2 lines
So reading the test, the reason this doesn't fail constantly (on a non-isolated machine) is the massive margins?
Margins should help, but this case has more chance to failing after recent kernel change. It does help track kernel behaviro.
Quoted 53 lines
It pins the
io_uring_wait_cqe_timeout()to CPU0 with either in_iowait=1 or in_iowait=0.
It then asserts after 1s that CPU0 got:
a) for in_iowait=0 at most +10 iowait units (<10% CPU time)
b) for in_iowait=1 at least +50 iowait units (>50% CPU time) So a) would fail on too many unrelated iowaits, b) would fail on unrelated runnable tasks.I don't think validating the per-task
in_iowaitsetting via the CPU aggregate is really feasible FWIW.Quoted 41 lines
Fix this by checking
nr_iowait_cpu()inkcpustat_irq_enter(). If the CPU is currently in iowait, pause and settle the elapsed time toCPUTIME_IOWAITbefore the interrupt handler runs. Onkcpustat_irq_exit(), resume accounting if it was paused.Fixes 7198e3927a14 ("sched/cputime: Handle idle irqtime gracefully")Signed-off-by Wei Gao <[email protected]>--- kernel/sched/cputime.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)kernel/sched/cputime.c +7
-2@@ -496,7 +496,8 @@ void kcpustat_irq_enter(u64 now)struct kernel_cpustat *kc = kcpustat_this_cpu;if (!vtime_generic_enabled_this_cpu () && - (irqtime_enabled () ||vtime_accounting_enabled_this_cpu ())) + (irqtime_enabled () ||vtime_accounting_enabled_this_cpu () || +nr_iowait_cpu (smp_processor_id ()) >0 ))kcpustat_idle_stop (kc, now); }@@ -510,9 +511,13 @@ void kcpustat_irq_exit(u64 now)* need to pause nohz accounting. Resume nohz accounting as long * as the irqtime config is enabled to handle case where irqtime * accounting got runtime disabled in the middle of an IRQ. +* +* Also resume accounting if it was paused on IRQ enter due to +* iowait. */ if (!vtime_generic_enabled_this_cpu () && - (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING ) ||vtime_accounting_enabled_this_cpu ())) + (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING ) ||vtime_accounting_enabled_this_cpu () || + !kc->idle_elapse))kcpustat_idle_start (kc, now); }