time/tick-sched: enable idle load balancing when nohz_full cpu becomes idle.
Subsystems (2)
PATCH 1/1 time/tick-sched: enable idle load balancing when nohz_full cpu becomes idle.
When nohz_full CPU stops tick in tick_nohz_irq_exit(),
It wouldn't be chosen to perform idle load balancing bacause it doesn't call nohz_balance_enter_idle() in tick_nohz_idle_stop_tick() when it becomes idle.
tick_nohz_idle_stop_tick() is only called in idle state and nohz_balance_enter_idle() tracks the CPU is part of nohz.idle_cpus_mask with rq->nohz_tick_stopped.
So, nohz_balance_enter_idle() could be called safely without !was_stooped check.
--- kernel/time/tick-sched.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/time/tick-sched.c +4 -2
@@ -1228,8 +1228,10 @@ void tick_nohz_idle_stop_tick(void) ts->idle_sleeps++; ts->idle_expires = expires; -if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { - ts->idle_jiffies = ts->last_jiffies; +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +if (!was_stopped) + ts->idle_jiffies = ts->last_jiffies; +nohz_balance_enter_idle (cpu); } }else {
--
2.41.0
Quoted 2 lines
When
nohz_fullCPU stops tick intick_nohz_irq_exit(),
It wouldn't be chosen to perform idle load balancing bacause it doesn't
…
Quoted 2 lines
So,
nohz_balance_enter_idle()could be called safely without !was_stooped check.
* Please avoid typos in this change description.
* How do you think about to add the tag “Fixes”?
* Would you like to use imperative wordings for an improved changelog?
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst?h=v6.9-rc7#n94
Regards,
Markus
Markus, kernel-janitors doesn't need to be a part of these discussions. Could you please stop adding us to the CC list?
regards,
dan carpenter
PATCH v2 1/1 time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
In reply to Levi YunWhen nohz_full CPU stops tick in tick_nohz_irq_exit(),
It wouldn't be chosen to perform idle load balancing bacause it doesn't call nohz_balance_enter_idle() in tick_nohz_idle_stop_tick() when it becomes idle.
tick_nohz_idle_stop_tick() is only called in idle state and nohz_balance_enter_idle() tracks the CPU is part of nohz.idle_cpus_mask with rq->nohz_tick_stopped.
So, nohz_balance_enter_idle() could be called safely without !was_stooped check.
--- kernel/time/tick-sched.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/time/tick-sched.c +4 -2
@@ -1228,8 +1228,10 @@ void tick_nohz_idle_stop_tick(void) ts->idle_sleeps++; ts->idle_expires = expires; -if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { - ts->idle_jiffies = ts->last_jiffies; +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +if (!was_stopped) + ts->idle_jiffies = ts->last_jiffies; +nohz_balance_enter_idle (cpu); } }else {
--
2.41.0
Re: [PATCH v2] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 2 lines
When
nohz_fullCPU stops tick intick_nohz_irq_exit(),
It wouldn't be chosen to perform idle load balancing bacause it doesn't
…
Quoted 2 lines
So,
nohz_balance_enter_idle()could be called safely without !was_stooped check.
* Why did you repeat typos in this patch iteration?
* Would you like to take the relevance of imperative wordings for changelogs better into account?
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst?h=v6.9-rc7#n94
Quoted 2 lines
--- kernel/time/tick-sched.c | 6 ++++--
Will further patch version descriptions become helpful behind the marker line?
Regards,
Markus
Re: [PATCH v2] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Hi Markus.
Quoted 3 lines
* Why did you repeat typos in this patch iteration?
* Would you like to take the relevance of imperative wordings for changelogs
better into account?
Sorry for my mistake, I've fixed it but, I sent an old one :(
Thanks
Quoted 21 lines · Markus Elfring
Quoted 3 lines
When
nohz_fullCPU stops tick intick_nohz_irq_exit(),
It wouldn't be chosen to perform idle load balancing bacause it doesn't…
Quoted 2 lines
So,
nohz_balance_enter_idle()could be called safely without !was_stooped check.* Why did you repeat typos in this patch iteration?
* Would you like to take the relevance of imperative wordings for changelogs better into account?
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst?h=v6.9-rc7#n94Quoted 2 lines
--- kernel/time/tick-sched.c | 6 ++++--Will further patch version descriptions become helpful behind the marker line?
Regards,
Markus
PATCH v3 1/1 time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
In reply to Levi YunWhen nohz_full CPU stops tick in tick_nohz_irq_exit(),
It wouldn't be chosen to perform idle load balancing because it doesn't call nohz_balance_enter_idle() in tick_nohz_idle_stop_tick() when it becomes idle.
tick_nohz_idle_stop_tick() is only called in idle state and nohz_balance_enter_idle() tracks the CPU which is part of nohz.idle_cpus_mask with rq->nohz_tick_stopped.
Change tick_nohz_idle_stop_tick() to call nohz_balance_enter_idle() without checking !was_stopped so that nohz_full cpu can be chosen to perform idle load balancing when it enters idle state.
---
v3:
- Rewording commit message.
v2:
- Fix typos in commit message.
kernel/time/tick-sched.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/time/tick-sched.c +4 -2
@@ -1228,8 +1228,10 @@ void tick_nohz_idle_stop_tick(void) ts->idle_sleeps++; ts->idle_expires = expires; -if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { - ts->idle_jiffies = ts->last_jiffies; +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +if (!was_stopped) + ts->idle_jiffies = ts->last_jiffies; +nohz_balance_enter_idle (cpu); } }else {
--
2.41.0
Re: [PATCH v3] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 5 lines
When
nohz_fullCPU stops tick intick_nohz_irq_exit(),
It wouldn't be chosen to perform idle load balancing because it doesn't callnohz_balance_enter_idle()intick_nohz_idle_stop_tick()when it becomes idle.
Quoted 7 lines
tick_nohz_idle_stop_tick()is only called in idle state andnohz_balance_enter_idle()tracks the CPU which is part of nohz.idle_cpus_mask withrq->nohz_tick_stopped.Change
tick_nohz_idle_stop_tick()to callnohz_balance_enter_idle()without checking !was_stopped so thatnohz_fullcpu can be chosen to perform idle load balancing when it enters idle state.
Would you eventually like to add the tag “Fixes” once more?
…
kernel/time/tick-sched.c void tick_nohz_idle_stop_tick(void) · line 1236
Quoted 13 lines
+++
b/kernel/time/tick-sched.ckernel/time/tick-sched.c +4
-2@@ -1228,8 +1228,10 @@ void tick_nohz_idle_stop_tick(void) ts->idle_sleeps++; ts->idle_expires = expires; -if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { - ts->idle_jiffies = ts->last_jiffies; +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +if (!was_stopped) + ts->idle_jiffies = ts->last_jiffies; +nohz_balance_enter_idle (cpu); }
…
I interpret these diff data in the way that you propose to reorder two condition checks.
But I wonder still how “good” the presented change description fits to the suggested source code adjustment.
Regards,
Markus
Re: [PATCH v3] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 1 line
Would you eventually like to add the tag “Fixes” once more?
Sorry. I forgot :(
Quoted 15 lines
Quoted 13 lines
+++
b/kernel/time/tick-sched.ckernel/time/tick-sched.c +4
-2@@ -1228,8 +1228,10 @@ void tick_nohz_idle_stop_tick(void) ts->idle_sleeps++; ts->idle_expires = expires; -if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { - ts->idle_jiffies = ts->last_jiffies; +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +if (!was_stopped) + ts->idle_jiffies = ts->last_jiffies; +nohz_balance_enter_idle (cpu); }…
Quoted 5 lines
I interpret these diff data in the way that you propose to reorder two condition checks.
But I wonder still how “good” the presented change description fits to the suggested source code adjustment.
FWIW it doesn't need to check !was_stopped to
call nohz_balance_enter_idle() in tick_nohz_idle_stop_tick(). Formerly, __tick_nohz_idle_enter() is called in both tick_nohz_irq_exit() and in do_idle().
That's why it's required for nohz full cpu which already stop the tick, but not idle
to prevent enter idle balance.
(but it makes nohz full cpu enter nohz idle balance as side effect i think?)
but after some reorganizing code tick_nohz_idle_stop_tick() becomes
the code called in
only when enter idle.
What I point is that it doesn't need to check !was_stopped to call nohz_balance_enter_idle()
in tick_nohz_idle_stop_tick().
So, I think it's enough in commit message?
Am I wrong?
Thanks.
Re: [PATCH v3] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
+++
b/kernel/time/tick-sched.ckernel/time/tick-sched.c +4
-2@@ -1228,8 +1228,10 @@ void tick_nohz_idle_stop_tick(void) ts->idle_sleeps++; ts->idle_expires = expires; -if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { - ts->idle_jiffies = ts->last_jiffies; +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +if (!was_stopped) + ts->idle_jiffies = ts->last_jiffies; +nohz_balance_enter_idle (cpu); }
…
Quoted 1 line
So, I think it's enough in commit message?
…
We are trying to clarify special implementation details here. Our corresponding wording preferences are probably different. I hope that a better common understanding can be achieved also for another transformation.
* Thus I became curious how you got interested to adjust this software component further.
* Will any other data representation become more helpful for the circumstances according to calls of a function like “tick_nohz_idle_stop_tick”?
* How do you think about to stress condition ordering concerns around the system configuration “nohz_full”?
* How will related changelogs evolve further?
Regards,
Markus
Re: [PATCH v3] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Hi Markus.
.
Quoted 2 lines
* Will any other data representation become more helpful for the circumstances according to calls of a function like “tick_nohz_idle_stop_tick”?
Maybe not in this commit..?
Quoted 2 lines
* How do you think about stress condition ordering concerns around the system configuration “nohz_full”?
Well.. regardless of the stress condition, it wants to fix the inconsistent behavior
happening when enter "idle state"
Let's think about two cases:
1. nohz_full cpu stop tick in tick_nohz_irq_exiit() while it runs
only 1 cfs task.
2. nohz_full cpu which doesn't stop the tick and switches to idle task.
Without this commit, case (1) wouldn't participate in idle balance when it switches to idle task while its tick is already stopped.
case (2) although nohz_full cpu participates in idle balcancing because former clock isn't stopped yet.
Quoted 2 lines
* How will related changelogs evolve further?
Thanks for the suggestion. But I'll add some more background commit message then.
Thanks again :)
Re: [v3] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 6 lines
Quoted 2 lines
* How do you think about stress condition ordering concerns around the system configuration “nohz_full”?
Well.. regardless of the stress condition, it wants to fix the inconsistent behavior
happening when enter "idle state"
Were any special test cases or related analysis tools involved in the discovery of improvable software behaviour?
Regards,
Markus
PATCH v4 1/1 time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
In reply to Levi YunWhen nohz_full CPU stops tick in tick_nohz_irq_exit(),
It wouldn't be chosen to perform idle load balancing because it doesn't call nohz_balance_enter_idle() in tick_nohz_idle_stop_tick() when it becomes idle.
Formerly, __tick_nohz_idle_enter() is called in both tick_nohz_irq_exit() and in do_idle().
That's why commit a0db971e4eb6 ("nohz: Move idle balancer registration to the idle path") prevents nohz_full cpu which isn't yet idle state but tick is stopped from entering idle balance.
However, this prevents nohz_full cpu which already stops tick from entering idle balacne when this cpu really becomes idle state.
Currently, tick_nohz_idle_stop_tick() is only called in idle state and it calls nohz_balance_enter_idle(). this function tracks the CPU which is part of nohz.idle_cpus_mask with rq->nohz_tick_stopped properly.
Therefore, Change tick_nohz_idle_stop_tick() to call nohz_balance_enter_idle() without checking !was_stopped so that nohz_full cpu can be chosen to perform idle load balancing when it enters idle state.
---
v4:
- Add fixes tags.
v3:
- Rewording commit message.
v2:
- Fix typos in commit message.
kernel/time/tick-sched.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/time/tick-sched.c +4 -2
@@ -1228,8 +1228,10 @@ void tick_nohz_idle_stop_tick(void) ts->idle_sleeps++; ts->idle_expires = expires; -if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { - ts->idle_jiffies = ts->last_jiffies; +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +if (!was_stopped) + ts->idle_jiffies = ts->last_jiffies; +nohz_balance_enter_idle (cpu); } }else {
--
2.41.0
Re: [v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
…
Quoted 3 lines
Currently,
tick_nohz_idle_stop_tick()is only called in idle state and it callsnohz_balance_enter_idle(). this function tracks the CPU which is part of nohz.idle_cpus_mask withrq->nohz_tick_stoppedproperly.
…
* How did you notice improvement possibilities for such data processing?
* I hope that a few remaining wording “weaknesses” can be adjusted accordingly.
Regards,
Markus
Re: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
In reply to Levi YunGentle ping..?
Am I missing something?
Thanks.
Quoted 57 lines · Levi Yun
When
nohz_fullCPU stops tick intick_nohz_irq_exit(),
It wouldn't be chosen to perform idle load balancing because it doesn't callnohz_balance_enter_idle()intick_nohz_idle_stop_tick()when it becomes idle.Formerly,
__tick_nohz_idle_enter()is called in bothtick_nohz_irq_exit()and indo_idle().
That's why commita0db971e4eb6("nohz: Move idle balancer registration to the idle path") preventsnohz_fullcpu which isn't yet idle state but tick is stopped from entering idle balance.However, this prevents
nohz_fullcpu which already stops tick from entering idle balacne when this cpu really becomes idle state.Currently,
tick_nohz_idle_stop_tick()is only called in idle state and it callsnohz_balance_enter_idle(). this function tracks the CPU which is part of nohz.idle_cpus_mask withrq->nohz_tick_stoppedproperly.Therefore, Change
tick_nohz_idle_stop_tick()to callnohz_balance_enter_idle()without checking !was_stopped so thatnohz_fullcpu can be chosen to perform idle load balancing when it enters idle state.Fixes a0db971e4eb6 ("nohz: Move idle balancer registration to the idle path")Signed-off-by Levi Yun <[email protected]>---
v4:
- Add fixes tags.v3:
- Rewording commit message.v2:
- Fix typos in commit message.kernel/time/tick-sched.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)kernel/time/tick-sched.c +4
-3@@ -1228,8 +1228,10 @@ void tick_nohz_idle_stop_tick(void) ts->idle_sleeps++; ts->idle_expires = expires; -if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { - ts->idle_jiffies = ts->last_jiffies; +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +if (!was_stopped) + ts->idle_jiffies = ts->last_jiffies; +nohz_balance_enter_idle (cpu); } }else { --2.41.0
Frederic Weisbecker
maintainerRe: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
In reply to Levi YunQuoted 42 lines · Levi Yun
When
nohz_fullCPU stops tick intick_nohz_irq_exit(),
It wouldn't be chosen to perform idle load balancing because it doesn't callnohz_balance_enter_idle()intick_nohz_idle_stop_tick()when it becomes idle.Formerly,
__tick_nohz_idle_enter()is called in bothtick_nohz_irq_exit()and indo_idle().
That's why commita0db971e4eb6("nohz: Move idle balancer registration to the idle path") preventsnohz_fullcpu which isn't yet idle state but tick is stopped from entering idle balance.However, this prevents
nohz_fullcpu which already stops tick from entering idle balacne when this cpu really becomes idle state.Currently,
tick_nohz_idle_stop_tick()is only called in idle state and it callsnohz_balance_enter_idle(). this function tracks the CPU which is part of nohz.idle_cpus_mask withrq->nohz_tick_stoppedproperly.Therefore, Change
tick_nohz_idle_stop_tick()to callnohz_balance_enter_idle()without checking !was_stopped so thatnohz_fullcpu can be chosen to perform idle load balancing when it enters idle state.Fixes a0db971e4eb6 ("nohz: Move idle balancer registration to the idle path")Signed-off-by Levi Yun <[email protected]>---
v4:
- Add fixes tags.v3:
- Rewording commit message.v2:
- Fix typos in commit message.kernel/time/tick-sched.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)kernel/time/tick-sched.c +0
-0
kernel/time/tick-sched.c void tick_nohz_idle_stop_tick(void) · line 1234
Quoted 9 lines
kernel/time/tick-sched.c +4
-2@@ -1228,8 +1228,10 @@ void tick_nohz_idle_stop_tick(void) ts->idle_sleeps++; ts->idle_expires = expires; -if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { - ts->idle_jiffies = ts->last_jiffies; +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +if (!was_stopped) + ts->idle_jiffies = ts->last_jiffies; +
I've taken some time to respond because your patch has raised more questions while discussing this with Anna-Maria:
1) Is Idle load balancing actually relevant for nohz_full? HK_TYPE_MISC already prevent those CPUs from becoming idle load balancer. They can still be targets for load balancing but nohz_full CPUs are supposed to run only one task.
2) This is related to previous point: HK_TYPE_SCHED is never activated. It would prevent the CPU from even beeing part of idle load balancing. Should we remove it or plug it?
3) nohz_balance_enter_idle() is called when the tick is stopped for the first time and nohz_balance_exit_idle() is called from the tick. But that also applies to idle ticks. So if the load balancing triggers while the tick is stopped, nohz_balance_enter_idle() won't be re-called in the idle loop even though the tick is stopped (that would be fixed with your patch).
4) Why is nohz_balance_exit_idle() called from the tick and not from the idle exit path? Is it to avoid overhead?
I'm adding some scheduler people in Cc who might help answer some of those questions.
Thanks.
Quoted 5 lines
kernel/time/tick-sched.c +0
-1nohz_balance_enter_idle (cpu); } }else { --2.41.0
Re: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
HI. Frederic!
Thanks for your reply.
Quoted 4 lines
1) Is Idle load balancing actually relevant for
nohz_full?HK_TYPE_MISCalready prevent those CPUs from becoming idle load balancer. They can still be targets for load balancing butnohz_fullCPUs are supposed to run only one task.
I'm not sure nohz_full cpu is really relevant with Idle load balancing. However, I couldn't find any reason it shouldn't participate Idle load balancing.
And about HK_TYPE_MISC, this is half true when it sets with "nohz_full=" options,
if nohz_full cpu is set via "isolcpus=nohz," it's still in HK_TYPE_MISC mask so find_ilb() doesn't prevent entering idle load balance.
Quoted 3 lines
2) This is related to previous point:
HK_TYPE_SCHEDis never activated. It would prevent the CPU from even beeing part of idle load balancing. Should we remove it or plug it?
IMHO, If we want to prevent nohz_full cpu entering, I should set with HK_TYPE_TICK | HK_TYPE_SCHED together.
Because, find_new_ilb()'s comment say
" HK_TYPE_MISC CPUs are used for this task, because HK_TYPE_SCHED is not set anywhere yet."
or integrate with HK_TYPE_TICK....? removing HK_TYPE_SCHED in the hk list...
It depends on ilb relevant for nohz_full...
If it is, Idle load balance could be controlled with HK_TYPE_SCHED. If it doesn't ,IMHO, It seems better to remove HK_TYPE_SCHED and check with HK_TYPE_SCHED.
Quoted 5 lines
4) Why is
nohz_balance_exit_idle()called from the tick and not from the idle exit path? Is it to avoid overhead?I'm adding some scheduler people in Cc who might help answer some of those questions.
...
Thanks :)
Re: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
In reply to Frederic WeisbeckerQuoted 72 lines · Frederic Weisbecker
Quoted 42 lines · Levi Yun
When
nohz_fullCPU stops tick intick_nohz_irq_exit(),
It wouldn't be chosen to perform idle load balancing because it doesn't callnohz_balance_enter_idle()intick_nohz_idle_stop_tick()when it becomes idle.Formerly,
__tick_nohz_idle_enter()is called in bothtick_nohz_irq_exit()and indo_idle().
That's why commita0db971e4eb6("nohz: Move idle balancer registration to the idle path") preventsnohz_fullcpu which isn't yet idle state but tick is stopped from entering idle balance.However, this prevents
nohz_fullcpu which already stops tick from entering idle balacne when this cpu really becomes idle state.Currently,
tick_nohz_idle_stop_tick()is only called in idle state and it callsnohz_balance_enter_idle(). this function tracks the CPU which is part of nohz.idle_cpus_mask withrq->nohz_tick_stoppedproperly.Therefore, Change
tick_nohz_idle_stop_tick()to callnohz_balance_enter_idle()without checking !was_stopped so thatnohz_fullcpu can be chosen to perform idle load balancing when it enters idle state.Fixes a0db971e4eb6 ("nohz: Move idle balancer registration to the idle path")Signed-off-by Levi Yun <[email protected]>---
v4:
- Add fixes tags.v3:
- Rewording commit message.v2:
- Fix typos in commit message.kernel/time/tick-sched.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)kernel/time/tick-sched.c +0
-0kernel/time/tick-sched.c void tick_nohz_idle_stop_tick(void) · line 1234
Quoted 9 lines
kernel/time/tick-sched.c +4
-2@@ -1228,8 +1228,10 @@ void tick_nohz_idle_stop_tick(void) ts->idle_sleeps++; ts->idle_expires = expires; -if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { - ts->idle_jiffies = ts->last_jiffies; +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +if (!was_stopped) + ts->idle_jiffies = ts->last_jiffies; +I've taken some time to respond because your patch has raised more questions while discussing this with Anna-Maria:
1) Is Idle load balancing actually relevant for
nohz_full?HK_TYPE_MISCalready prevent those CPUs from becoming idle load balancer. They can still be targets for load balancing butnohz_fullCPUs are supposed to run only one task.2) This is related to previous point:
HK_TYPE_SCHEDis never activated. It would prevent the CPU from even beeing part of idle load balancing. Should we remove it or plug it?3)
nohz_balance_enter_idle()is called when the tick is stopped for the first time andnohz_balance_exit_idle()is called from the tick. But that also applies to idle ticks. So if the load balancing triggers while the tick is stopped,nohz_balance_enter_idle()won't be re-called in the idle loop even though the tick is stopped (that would be fixed with your patch).
Quoted 5 lines
4) Why is
nohz_balance_exit_idle()called from the tick and not from the idle exit path? Is it to avoid overhead?I'm adding some scheduler people in Cc who might help answer some of those questions.
None of that HK nonsense is relevant. The NOHZ_FULL nonsense implies single CPU partitions, and *that* should be avoiding any and all load-balancing.
If there still is, that's a bug, but that's not related to HK goo.
As such, I don't think the HK_TYPE_SCHED check in nohz_balance_enter_idle() actually makes sense, the on_null_omain() check a little below that should already take care of things, no?
Re: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 3 lines
None of that HK nonsense is relevant. The
NOHZ_FULLnonsense implies single CPU partitions, and *that* should be avoiding any and all load-balancing.
Do you mean.. tick_nohz_full cpu (non-HK-ticked cpu) shouldn't belong to any sched_domain?
Quoted 6 lines
If there still is, that's a bug, but that's not related to HK goo.
As such, I don't think the
HK_TYPE_SCHEDcheck innohz_balance_enter_idle()actually makes sense, theon_null_omain()check a little below that should already take care of things, no?
IIUC,
currently, whether cpu belongs on domain or null is determined by HK_DOMAIN_FLAGS
However, when "nohz_full=" is used, it still on HK_DOMAIN, so it belongs to sched_domain
so, it couldn't be filtered out by on_null_domain().
unless "isolcpus=domain" or "isolcpus={cpu_list}", it's on null domain. with "isolcpus=tick", it participates sched_domain.
Re: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 6 lines · Yun Levi
Quoted 3 lines
None of that HK nonsense is relevant. The
NOHZ_FULLnonsense implies single CPU partitions, and *that* should be avoiding any and all load-balancing.Do you mean..
tick_nohz_fullcpu (non-HK-ticked cpu) shouldn't belong to anysched_domain?
AFAIK NOHZ_FULL still hard relies on the isolcpus garbage, so yeah, it should be all single cpu partitions, which don't have a domain.
(this really should migrate to use cpusets partitions)
Quoted 9 lines
Quoted 5 lines
If there still is, that's a bug, but that's not related to HK goo.
As such, I don't think the
HK_TYPE_SCHEDcheck innohz_balance_enter_idle()actually makes sense, theon_null_omain()check a little below that should already take care of things, no?IIUC,
currently, whether cpu belongs on domain or null is determined byHK_DOMAIN_FLAGS
No! you can create NULL domains without any of the HK nonsense. Both isolcpus and cpusets can create single CPU partitions.
Quoted 6 lines
However, when "nohz_full=" is used, it still on
HK_DOMAIN, so it belongs tosched_domain
so, it couldn't be filtered out byon_null_domain().unless "isolcpus=domain" or "isolcpus={cpu_list}", it's on null domain. with "isolcpus=tick", it participates
sched_domain.
Frederic ?!? You can use nohz_full without isolcpus? That makes no sense. If you do that you get to keep the pieces.
Frederic Weisbecker
maintainerRe: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 30 lines · Peter Zijlstra
Quoted 6 lines · Yun Levi
Quoted 3 lines
None of that HK nonsense is relevant. The
NOHZ_FULLnonsense implies single CPU partitions, and *that* should be avoiding any and all load-balancing.Do you mean..
tick_nohz_fullcpu (non-HK-ticked cpu) shouldn't belong to anysched_domain?AFAIK
NOHZ_FULLstill hard relies on the isolcpus garbage, so yeah, it should be all single cpu partitions, which don't have a domain.(this really should migrate to use cpusets partitions)
Quoted 9 lines
Quoted 5 lines
If there still is, that's a bug, but that's not related to HK goo.
As such, I don't think the
HK_TYPE_SCHEDcheck innohz_balance_enter_idle()actually makes sense, theon_null_omain()check a little below that should already take care of things, no?IIUC,
currently, whether cpu belongs on domain or null is determined byHK_DOMAIN_FLAGSNo! you can create NULL domains without any of the HK nonsense. Both isolcpus and cpusets can create single CPU partitions.
Quoted 4 lines
However, when "nohz_full=" is used, it still on
HK_DOMAIN, so it belongs tosched_domain
so, it couldn't be filtered out byon_null_domain().
Quoted 5 lines
Quoted 2 lines
unless "isolcpus=domain" or "isolcpus={cpu_list}", it's on null domain. with "isolcpus=tick", it participates
sched_domain.Frederic ?!? You can use
nohz_fullwithout isolcpus? That makes no sense. If you do that you get to keep the pieces.
I fear you can yes, even though most users combine it with isolcpus. I know, that interface is terrible but it dates from times when we weren't sure about all the potential usecases of nohz_full. There was a possibility that HPC could just want to reduce ticks without all the hard and costly isolation around. But all the usecases I have witnessed so far in ten years involved wanting 0 noise after all...
Re: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
As such, I don't think the
HK_TYPE_SCHEDcheck innohz_balance_enter_idle()actually makes sense, theon_null_omain()check a little below that should already take care of things, no?IIUC,
currently, whether cpu belongs on domain or null is determined byHK_DOMAIN_FLAGSNo! you can create NULL domains without any of the HK nonsense. Both isolcpus and cpusets can create single CPU partitions.
Yes. However what I said, nohz_full cpu isn't on null_domain unless it was configured by cpusets.
even with option "nohz_full="
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
However, when "nohz_full=" is used, it still on
HK_DOMAIN, so it belongs tosched_domain
so, it couldn't be filtered out byon_null_domain().unless "isolcpus=domain" or "isolcpus={cpu_list}", it's on null domain. with "isolcpus=tick", it participates
sched_domain.Frederic ?!? You can use
nohz_fullwithout isolcpus? That makes no sense. If you do that you get to keep the pieces.
Quoted 6 lines
I fear you can yes, even though most users combine it with isolcpus. I know, that interface is terrible but it dates from times when we weren't sure about all the potential usecases of
nohz_full. There was a possibility that HPC could just want to reduce ticks without all the hard and costly isolation around. But all the usecases I have witnessed so far in ten years involved wanting 0 noise after all...
If I make you annoyed I'm sorry in advance but let me clarify please.
1. In case of none-HK-TICK-housekeeping cpu (a.k.a nohz_full cpu), It should be on the null_domain. right?
2. If (1) is true, when none-HK-TICK is set, should it set none-HK-DOMAIN to prevent on any sched_domain (cpusets filter out none-HK-DOMAIN cpu)?
3. If (1) is true, Is HK_SCHED still necessary? There seems to be no use case and the check for this can be replaced by on_null_domain().
Many thanks!
Re: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 8 lines · Yun Levi
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
As such, I don't think the
HK_TYPE_SCHEDcheck innohz_balance_enter_idle()actually makes sense, theon_null_omain()check a little below that should already take care of things, no?IIUC,
currently, whether cpu belongs on domain or null is determined byHK_DOMAIN_FLAGS
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
No! you can create NULL domains without any of the HK nonsense. Both isolcpus and cpusets can create single CPU partitions.
Yes. However what I said,
nohz_fullcpu isn't onnull_domainunless it was configured by cpusets.
even with option "nohz_full="
So if a CPU isn't isolated, either by isolcpus or cpusets, you get to participate in load-balancing -- end of story.
Quoted 20 lines
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
However, when "nohz_full=" is used, it still on
HK_DOMAIN, so it belongs tosched_domain
so, it couldn't be filtered out byon_null_domain().unless "isolcpus=domain" or "isolcpus={cpu_list}", it's on null domain. with "isolcpus=tick", it participates
sched_domain.Frederic ?!? You can use
nohz_fullwithout isolcpus? That makes no sense. If you do that you get to keep the pieces.Quoted 6 lines
I fear you can yes, even though most users combine it with isolcpus. I know, that interface is terrible but it dates from times when we weren't sure about all the potential usecases of
nohz_full. There was a possibility that HPC could just want to reduce ticks without all the hard and costly isolation around. But all the usecases I have witnessed so far in ten years involved wanting 0 noise after all...If I make you annoyed I'm sorry in advance but let me clarify please.
Quoted 8 lines
1. In case of none-HK-TICK-housekeeping cpu (a.k.a
nohz_fullcpu), It should be on thenull_domain. right?2. If (1) is true, when none-HK-TICK is set, should it set none-HK-DOMAIN to prevent on any
sched_domain(cpusets filter out none-HK-DOMAIN cpu)?3. If (1) is true, Is
HK_SCHEDstill necessary? There seems to be no use case and the check for this can be replaced byon_null_domain().
I've no idea about all those HK knobs, it's all insane if you ask me.
Frederic, afaict all the HK_ goo in kernel/sched/fair.c is total nonsense, can you please explain?
If the CPU participates in load-balancing, it gets to fully participate. If you want to get out of load-balancing, you get single CPU partitions.
Frederic Weisbecker
maintainerRe: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 8 lines · Peter Zijlstra
Quoted 8 lines
If I make you annoyed I'm sorry in advance but let me clarify please.
1. In case of none-HK-TICK-housekeeping cpu (a.k.a
nohz_fullcpu), It should be on thenull_domain. right?2. If (1) is true, when none-HK-TICK is set, should it set none-HK-DOMAIN to prevent on any
sched_domain(cpusets filter out none-HK-DOMAIN cpu)?
Quoted 7 lines
Quoted 2 lines
3. If (1) is true, Is
HK_SCHEDstill necessary? There seems to be no use case and the check for this can be replaced byon_null_domain().I've no idea about all those HK knobs, it's all insane if you ask me.
Frederic, afaict all the
HK_goo inkernel/sched/fair.cis total nonsense, can you please explain?
Yes. Lemme unearth this patch:
https://lore.kernel.org/all/[email protected]/
Because all we need now is:
_ HK_TYPE_KERNEL_NOISE: nohz_full= or isolcpus=nohz _ HK_TYPE_DOMAIN: isolcpus=domain (or classic isolcpus= alone) _ HK_TYPE_MANAGED_IRQ: isolcpus=managed_irq
And that's it. Then let's remove HK_TYPE_SCHED that is unused. And then lemme comment the HK_TYPE_* uses within sched/* within the same patchset.
Just a question, correct me if I'm wrong, we don't want nohz_full= to ever take the idle load balancer duty (this is what HK_TYPE_MISC prevents in find_new_ilb) because the nohz_full CPU going back to userspace concurrently doesn't want to be disturbed by a loose IPI telling it to do idle balancing. But we still want nohz_full CPUs to be part of nohz.idle_cpus_mask so that the idle balancing can be performed on them by a non isolated CPU. Right?
Thanks.
Quoted 3 lines
If the CPU participates in load-balancing, it gets to fully participate. If you want to get out of load-balancing, you get single CPU partitions.
Re: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 12 lines · Frederic Weisbecker
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines
If I make you annoyed I'm sorry in advance but let me clarify please.
1. In case of none-HK-TICK-housekeeping cpu (a.k.a
nohz_fullcpu), It should be on thenull_domain. right?2. If (1) is true, when none-HK-TICK is set, should it set none-HK-DOMAIN to prevent on any
sched_domain(cpusets filter out none-HK-DOMAIN cpu)?3. If (1) is true, Is
HK_SCHEDstill necessary? There seems to be no use case and the check for this can be replaced byon_null_domain().
Quoted 7 lines
Quoted 4 lines
I've no idea about all those HK knobs, it's all insane if you ask me.
Frederic, afaict all the
HK_goo inkernel/sched/fair.cis total nonsense, can you please explain?Yes. Lemme unearth this patch:
https://lore.kernel.org/all/[email protected]/
AFAICT we need more cleanups.
Quoted 4 lines
Because all we need now is:
_
HK_TYPE_KERNEL_NOISE: nohz_full= or isolcpus=nohz _HK_TYPE_DOMAIN: isolcpus=domain (or classic isolcpus= alone)
What does this do?
Quoted 5 lines
_
HK_TYPE_MANAGED_IRQ: isolcpus=managed_irqAnd that's it. Then let's remove
HK_TYPE_SCHEDthat is unused. And then lemme comment the HK_TYPE_* uses within sched/* within the same patchset.
Please, I find this MISC and DOMAIN stuff confusing, wth does it do? It can't possibly be right.
Quoted 6 lines
Just a question, correct me if I'm wrong, we don't want nohz_full= to ever take the idle load balancer duty (this is what
HK_TYPE_MISCprevents infind_new_ilb) because thenohz_fullCPU going back to userspace concurrently doesn't want to be disturbed by a loose IPI telling it to do idle balancing. But we still wantnohz_fullCPUs to be part of nohz.idle_cpus_mask so that the idle balancing can be performed on them by a non isolated CPU. Right?
I'm confused, none of that makes sense. If you're part of a load-balancer, you're part of a load-balancer, no ifs buts or other nonsense.
idle load balancer is no different from regular load balancing.
Fundamentally, you can't disable the tick if you're part of a load-balance group, the load-balancer needs the tick.
The only possible way to use nohz_full is to not be part of a load-balancer, and the only way that is so is by having (lots of) single CPU partitions.
Frederic Weisbecker
maintainerRe: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Frederic Weisbecker
Quoted 13 lines · Peter Zijlstra
Quoted 10 lines
If I make you annoyed I'm sorry in advance but let me clarify please.
1. In case of none-HK-TICK-housekeeping cpu (a.k.a
nohz_fullcpu), It should be on thenull_domain. right?2. If (1) is true, when none-HK-TICK is set, should it set none-HK-DOMAIN to prevent on any
sched_domain(cpusets filter out none-HK-DOMAIN cpu)?3. If (1) is true, Is
HK_SCHEDstill necessary? There seems to be no use case and the check for this can be replaced byon_null_domain().I've no idea about all those HK knobs, it's all insane if you ask me.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Frederic, afaict all the
HK_goo inkernel/sched/fair.cis total nonsense, can you please explain?Yes. Lemme unearth this patch:
https://lore.kernel.org/all/[email protected]/AFAICT we need more cleanups.
Well, we need to start somewhere :-)
Quoted 7 lines
Quoted 5 lines
Because all we need now is:
_
HK_TYPE_KERNEL_NOISE: nohz_full= or isolcpus=nohz _HK_TYPE_DOMAIN: isolcpus=domain (or classic isolcpus= alone)What does this do?
So housekeeping_cpumask(HK_TYPE_KERNEL_NOISE) will return all the CPUs not in nohz_full=
That is, all the CPUs that do all the housekeeping work on behalf of nohz_full CPUs (unbound workqueues and timers, etc...).
Then in a similar way housekeeping_cpumask(HK_TYPE_DOMAIN) are all the CPUs not in isolcpus= (on_null_domain()). Perhaps that one is confusing and we should just have a simple isolcpus_cpumask instead?
Quoted 9 lines
Quoted 6 lines
_
HK_TYPE_MANAGED_IRQ: isolcpus=managed_irqAnd that's it. Then let's remove
HK_TYPE_SCHEDthat is unused. And then lemme comment the HK_TYPE_* uses within sched/* within the same patchset.Please, I find this MISC and DOMAIN stuff confusing, wth does it do? It can't possibly be right.
MISC is actually part of what is going to become HK_TYPE_KERNEL_NOISE. It's an artificial microfeature but it's actually the same as _WORKQUEUE, _TICK, _RCU, _TIMER, etc... All of which intended to be merged together.
Quoted 12 lines
Quoted 7 lines
Just a question, correct me if I'm wrong, we don't want nohz_full= to ever take the idle load balancer duty (this is what
HK_TYPE_MISCprevents infind_new_ilb) because thenohz_fullCPU going back to userspace concurrently doesn't want to be disturbed by a loose IPI telling it to do idle balancing. But we still wantnohz_fullCPUs to be part of nohz.idle_cpus_mask so that the idle balancing can be performed on them by a non isolated CPU. Right?I'm confused, none of that makes sense. If you're part of a load-balancer, you're part of a load-balancer, no ifs buts or other nonsense.
Quoted 8 lines
idle load balancer is no different from regular load balancing.
Fundamentally, you can't disable the tick if you're part of a load-balance group, the load-balancer needs the tick.
The only possible way to use
nohz_fullis to not be part of a load-balancer, and the only way that is so is by having (lots of) single CPU partitions.
So you're suggesting that nohz_full should just be part of the whole ilb machinery by default (that is, not fiddle with ilb internals) and then it's up to CPU partitioning (through cpuset or isolcpus) to disable ilb naturally. Right?
Thanks.
Re: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 9 lines · Frederic Weisbecker
Quoted 9 lines
I'm confused, none of that makes sense. If you're part of a load-balancer, you're part of a load-balancer, no ifs buts or other nonsense.
idle load balancer is no different from regular load balancing.
Fundamentally, you can't disable the tick if you're part of a load-balance group, the load-balancer needs the tick.
Quoted 8 lines
Quoted 3 lines
The only possible way to use
nohz_fullis to not be part of a load-balancer, and the only way that is so is by having (lots of) single CPU partitions.So you're suggesting that
nohz_fullshould just be part of the whole ilb machinery by default (that is, not fiddle with ilb internals) and then it's up to CPU partitioning (through cpuset or isolcpus) to disable ilb naturally. Right?
Yes, but stronger, as long as the CPU is part of a load-balance domain, it must not disable the tick while running anything.
that is, NOHZ_FULL must not become active unless it's running on a single CPU partition.
Frederic Weisbecker
maintainerRe: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 20 lines · Peter Zijlstra
Quoted 9 lines · Frederic Weisbecker
Quoted 9 lines
I'm confused, none of that makes sense. If you're part of a load-balancer, you're part of a load-balancer, no ifs buts or other nonsense.
idle load balancer is no different from regular load balancing.
Fundamentally, you can't disable the tick if you're part of a load-balance group, the load-balancer needs the tick.
Quoted 8 lines
Quoted 3 lines
The only possible way to use
nohz_fullis to not be part of a load-balancer, and the only way that is so is by having (lots of) single CPU partitions.So you're suggesting that
nohz_fullshould just be part of the whole ilb machinery by default (that is, not fiddle with ilb internals) and then it's up to CPU partitioning (through cpuset or isolcpus) to disable ilb naturally. Right?
Quoted 5 lines
Yes, but stronger, as long as the CPU is part of a load-balance domain, it must not disable the tick while running anything.
that is,
NOHZ_FULLmust not become active unless it's running on a single CPU partition.
I like the idea but I'm afraid to introduce regressions while doing so, with people currently using nohz_full without proper partionning...
Re: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 8 lines
Quoted 5 lines
Yes, but stronger, as long as the CPU is part of a load-balance domain, it must not disable the tick while running anything.
that is,
NOHZ_FULLmust not become active unless it's running on a single CPU partition.I like the idea but I'm afraid to introduce regressions while doing so, with people currently using
nohz_fullwithout proper partionning...
But I wonder when nohz_full cpu enters *idle".
If the cpuidle governor selects an idle state which wants "stop_tick", If nohz_full cpu is still in sched domain, should it disable tick or not?
Re: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
In reply to Frederic WeisbeckerQuoted 10 lines · Frederic Weisbecker
Quoted 5 lines · Peter Zijlstra
Yes, but stronger, as long as the CPU is part of a load-balance domain, it must not disable the tick while running anything.
that is,
NOHZ_FULLmust not become active unless it's running on a single CPU partition.I like the idea but I'm afraid to introduce regressions while doing so, with people currently using
nohz_fullwithout proper partionning...
There is no regression, if this is possible today it is utterly broken.
This should never have been possible.
Frederic Weisbecker
maintainerRe: [PATCH v4] time/tick-sched: idle load balancing when nohz_full cpu becomes idle.
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Frederic Weisbecker
Quoted 5 lines · Peter Zijlstra
Yes, but stronger, as long as the CPU is part of a load-balance domain, it must not disable the tick while running anything.
that is,
NOHZ_FULLmust not become active unless it's running on a single CPU partition.
Quoted 6 lines
Quoted 2 lines
I like the idea but I'm afraid to introduce regressions while doing so, with people currently using
nohz_fullwithout proper partionning...There is no regression, if this is possible today it is utterly broken.
This should never have been possible.
Ok, I'll try something.
Thanks.