yielding while running SCHED_DEADLINE
Hi all,
We have been playing around with SCHED_DEADLINE and found some discrepancy around the calculation of nr_involuntary_switches and nr_voluntary_switches in /proc/${PID}/sched.
Whenever the task is done with it's work earlier and executes sched_yield() to voluntarily gives up the CPU this increments nr_involuntary_switches. It should have incremented nr_voluntary_switches.
This can be easily demonstrated by running cyclicdeadline task which is part of rt-tests(https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests .git/) and checking the value of nr_voluntary_switches.
Please note that the issue seems to be with sched_yield() and not SCHED_DEADLINE because we have seen similar behavior when we tried switching to other policies. But, we are using SCHED_DEADLINE because it is one of the (very) few scenarios where sched_yield() can be used correctly.
Some analysis:
--------------
I enabled the sched/sched_switch (setting cyclicdeadline as filter) and syscalls/sys_enter_sched_yield events to check whether the sched_yield() call was resulting in a new task running. I got the following results:
cyclicdeadline-3290 [003] ....... 3111.132786: tracing_mark_write: start at 3111125101 off=3 (period=3111125098 next=3111126098) cyclicdeadline-3290 [003] ....1.. 3111.132789: sys_sched_yield()
cyclicdeadline-3290 [003] d...2.. 3111.132797: sched_switch: prev_comm=cyclicdeadline prev_pid=3290 prev_prio=-1 prev_state=R ==> next_comm=swapper/3 next_pid=0 next_prio=120 cyclicdeadline-3290 [003] ....... 3111.133786: tracing_mark_write: start at 3111126101 off=3 (period=3111126098 next=3111127098) cyclicdeadline-3290 [003] ....1.. 3111.133789: sys_sched_yield()
cyclicdeadline-3290 [003] d...2.. 3111.133797: sched_switch: prev_comm=cyclicdeadline prev_pid=3290 prev_prio=-1 prev_state=R ==> next_comm=swapper/3 next_pid=0 next_prio=120 cyclicdeadline-3290 [003] ....... 3111.134786: tracing_mark_write: start at 3111127101 off=3 (period=3111127098 next=3111128098) cyclicdeadline-3290 [003] ....1.. 3111.134789: sys_sched_yield()
cyclicdeadline-3290 [003] d...2.. 3111.134797: sched_switch: prev_comm=cyclicdeadline prev_pid=3290 prev_prio=-1 prev_state=R ==> next_comm=swapper/3 next_pid=0 next_prio=120
....
As seen above, all the sched_yield calls are followed by sched switch. So, we believe that the sched_yield() is actually resulting in a switch. The values for nr_voluntary_switches/nr_involuntary_switches in this scenario:
nr_switches : 138753 nr_voluntary_switches : 1 nr_involuntary_switches : 138752
Looking at __schedule() in kernel/sched/core.c, the switch is counted as part of nr_involuntary_switches if the task has not been preempted and the task is TASK_RUNNING state. This does not seem to happen when sched_yield() is called.
Is there something we are missing over here? OR Is this a known issue and is planned to be fixed later?
Thanks,
Vedang Patel
Hi,
Quoted 10 lines · Vedang
Hi all,
We have been playing around with
SCHED_DEADLINEand found some discrepancy around the calculation of nr_involuntary_switches andnr_voluntary_switchesin/proc/${PID}/sched.Whenever the task is done with it's work earlier and executes
sched_yield()to voluntarily gives up the CPU this incrementsnr_involuntary_switches. It should have incrementednr_voluntary_switches.
Mmm, I see what you are saying.
[...]
Quoted 4 lines
Looking at
__schedule()inkernel/sched/core.c, the switch is counted as part ofnr_involuntary_switchesif the task has not been preempted and the task isTASK_RUNNINGstate. This does not seem to happen whensched_yield()is called.
Mmm,
- nr_voluntary_switches++ if !preempt && !RUNNING
- nr_involuntary_switches++ otherwise (yield fits this as the task is still RUNNING, even though throttled for DEADLINE)
Not sure this is the same as what you say above..
Quoted 2 lines
Is there something we are missing over here? OR Is this a known issue and is planned to be fixed later?
.. however, not sure. Peter, what you say. It looks like we might indeed want to account yield as a voluntary switch, seems to fit. In this case I guess we could use a flag or add a sched_ bit to task_struct to handle the case?
Best,
- Juri
Quoted 31 lines · Juri Lelli
Hi,
Quoted 10 lines · Vedang
Hi all,
We have been playing around with
SCHED_DEADLINEand found some discrepancy around the calculation of nr_involuntary_switches andnr_voluntary_switchesin/proc/${PID}/sched.Whenever the task is done with it's work earlier and executes
sched_yield()to voluntarily gives up the CPU this incrementsnr_involuntary_switches. It should have incrementednr_voluntary_switches.Mmm, I see what you are saying.
[...]
Quoted 4 lines
Looking at
__schedule()inkernel/sched/core.c, the switch is counted as part ofnr_involuntary_switchesif the task has not been preempted and the task isTASK_RUNNINGstate. This does not seem to happen whensched_yield()is called.Mmm,
- nr_voluntary_switches++ if !preempt && !RUNNING
- nr_involuntary_switches++ otherwise (yield fits this as the task is still RUNNING, even though throttled for DEADLINE)Not sure this is the same as what you say above..
Quoted 7 lines
Quoted 2 lines
Is there something we are missing over here? OR Is this a known issue and is planned to be fixed later?
.. however, not sure. Peter, what you say. It looks like we might indeed want to account yield as a voluntary switch, seems to fit. In this case I guess we could use a flag or add a
sched_bit totask_structto handle the case?
It's been like this _forever_ afaict. This isn't deadline specific afaict, all yield callers will end up in non-voluntary switches.
I don't know anybody that cares and I don't think this is something worth fixing. If someone did rely on this behaviour we'd break them, and i'd much rather save a cycle than add more stupid stats crap to the scheduler.
Quoted 48 lines · Peter Zijlstra
Quoted 40 lines · Juri Lelli
Hi,
Quoted 11 lines · Vedang
Hi all,
We have been playing around with
SCHED_DEADLINEand found some discrepancy around the calculation of nr_involuntary_switches andnr_voluntary_switchesin/proc/${PID}/sched.Whenever the task is done with it's work earlier and executes
sched_yield()to voluntarily gives up the CPU this incrementsnr_involuntary_switches. It should have incrementednr_voluntary_switches.Mmm, I see what you are saying.
[...]
Quoted 8 lines
Looking at
__schedule()inkernel/sched/core.c, the switch is counted
as part ofnr_involuntary_switchesif the task has not been preempted
and the task isTASK_RUNNINGstate. This does not seem to happen whensched_yield()is called.Mmm,
- nr_voluntary_switches++ if !preempt && !RUNNING
- nr_involuntary_switches++ otherwise (yield fits this as the task is
still RUNNING, even though throttled for DEADLINE)Not sure this is the same as what you say above..
Quoted 4 lines
Is there something we are missing over here? OR Is this a known issue
and is planned to be fixed later?Quoted 7 lines
.. however, not sure. Peter, what you say. It looks like we might indeed
want to account yield as a voluntary switch, seems to fit. In this case
I guess we could use a flag or add asched_bit totask_structto handle
the case?
Quoted 8 lines
It's been like this _forever_ afaict. This isn't deadline specific afaict, all yield callers will end up in non-voluntary switches.
I don't know anybody that cares and I don't think this is something worth fixing. If someone did rely on this behaviour we'd break them, and
i'd much rather save a cycle than add more stupid stats crap to the scheduler.
Thanks Peter and Juri for the response.
We will try to use a different mechanism to account for this.
-Vedang
I’m fine with not fixing this behavior since Vedang has mentioned, we can use different mechanisms to achieve the same goal. However, I would like to go on the record as someone that cares about this functionality.
Assuming that sched_yield was counted as part of nr_voluntary
switches, then the specific scenario in which I would find this statistic useful is during debug of latency spikes on a realtime thread.
In particular, I could quickly correlate whether or not a latency
spike occurred due to being swapped out by the scheduler.
Quoted 48 lines · Peter Zijlstra
Quoted 40 lines · Juri Lelli
Hi,
Quoted 11 lines · Vedang
Hi all,
We have been playing around with
SCHED_DEADLINEand found some discrepancy around the calculation ofnr_involuntary_switchesandnr_voluntary_switchesin/proc/${PID}/sched.Whenever the task is done with it's work earlier and executes
sched_yield()to voluntarily gives up the CPU this incrementsnr_involuntary_switches. It should have incrementednr_voluntary_switches.Mmm, I see what you are saying.
[...]
Quoted 8 lines
Looking at
__schedule()inkernel/sched/core.c, the switch is counted
as part ofnr_involuntary_switchesif the task has not been preempted
and the task isTASK_RUNNINGstate. This does not seem to happen whensched_yield()is called.Mmm,
- nr_voluntary_switches++ if !preempt && !RUNNING
- nr_involuntary_switches++ otherwise (yield fits this as the task
is
still RUNNING, even though throttled for DEADLINE)Not sure this is the same as what you say above..
Quoted 4 lines
Is there something we are missing over here? OR Is this a known issue
and is planned to be fixed later?Quoted 7 lines
.. however, not sure. Peter, what you say. It looks like we might indeed
want to account yield as a voluntary switch, seems to fit. In this case
I guess we could use a flag or add asched_bit totask_structto handle
the case?
Quoted 8 lines
It's been like this _forever_ afaict. This isn't deadline specific afaict, all yield callers will end up in non-voluntary switches.
I don't know anybody that cares and I don't think this is something worth fixing. If someone did rely on this behaviour we'd break them, and
i'd much rather save a cycle than add more stupid stats crap to the scheduler.
Thanks Peter and Juri for the response.
We will try to use a different mechanism to account for this.
-Vedang