Skip to contentSkip to conversation

SCHED_DEADLINE with CPU affinity

DISCUSSION Philipp Stanner 2 participants 4 messages Nov 19, 2019 – Jan 13, 2020

lore
Share & tools
lore
#0 lore raw

Hey folks,
(please put me in CC when answering, I'm not subscribed)

I'm currently working student in the embedded industry. We have a device where we need to be able to process network data within a certain deadline. At the same time, safety is a primary requirement; that's why we construct everything fully redundant. Meaning: We have two network interfaces, each IRQ then bound to one CPU core and spawn a container (systemd-nspawn, cgroups based) which in turn is bound to the corresponding CPU (CPU affinity masked).

        Container0       Container1
   -----------------  -----------------

   |               |  |               |
   |    Proc. A    |  |   Proc. A'    |
   |    Proc. B    |  |   Proc. B'    |
   |               |  |               |

   -----------------  -----------------
          ^                  ^
          |                  |
        CPU 0              CPU 1
          |                  |
       IRQ eth0           IRQ eth1

Within each container several processes are started. Ranging from systemd (SCHED_OTHER) till two (soft) real-time critical processes: which we want to execute via SCHED_DEADLINE.

Now, I've worked through the manpage describing scheduling policies, and it seems that our scenario is forbidden my the kernel.  I've done some tests with the syscalls sched_setattr and sched_setaffinity, trying to activate SCHED_DEADLINE while also binding to a certain core.  It fails with EINVAL or EINBUSY, depending on the order of the syscalls.

I've read that the kernel accomplishes plausibility checks when you ask for a new deadline task to be scheduled, and I assume this check is what prevents us from implementing our intended architecture.

Now, the questions we're having are:

   1. Why does the kernel do this, what is the problem with scheduling with SCHED_DEADLINE on a certain core? In contrast, how is it handled when you have single core systems etc.? Why this artificial limitation?
   2. How can we possibly implement this? We don't want to use SCHED_FIFO, because out-of-control tasks would freeze the entire container.

SCHED_RR / SCHED_FIFO will probably be a better policy compared to SCHED_OTHER, but SCHED_DEADLINE is exactly what we are looking for.

Cheers,
Philipp

#1 lore raw

Hi Philipp,

Quoted 28 lines · Philipp Stanner

Hey folks,
(please put me in CC when answering, I'm not subscribed)

I'm currently working student in the embedded industry. We have a device where we need to be able to process network data within a certain deadline. At the same time, safety is a primary requirement; that's why we construct everything fully redundant. Meaning: We have two network interfaces, each IRQ then bound to one CPU core and spawn a container (systemd-nspawn, cgroups based) which in turn is bound to the corresponding CPU (CPU affinity masked).

        Container0       Container1
   -----------------  -----------------

   |               |  |               |
   |    Proc. A    |  |   Proc. A'    |
   |    Proc. B    |  |   Proc. B'    |
   |               |  |               |

   -----------------  -----------------
          ^                  ^
          |                  |
        CPU 0              CPU 1
          |                  |
       IRQ eth0           IRQ eth1

Within each container several processes are started. Ranging from systemd (SCHED_OTHER) till two (soft) real-time critical processes: which we want to execute via SCHED_DEADLINE.

Quoted 7 lines

Now, I've worked through the manpage describing scheduling policies, and it seems that our scenario is forbidden my the kernel.  I've done some tests with the syscalls sched_setattr and sched_setaffinity, trying to activate SCHED_DEADLINE while also binding to a certain core.  It fails with EINVAL or EINBUSY, depending on the order of the syscalls.

I've read that the kernel accomplishes plausibility checks when you ask for a

Yeah, admission control.

Quoted 8 lines

new deadline task to be scheduled, and I assume this check is what prevents us from implementing our intended architecture.

Now, the questions we're having are:

   1. Why does the kernel do this, what is the problem with scheduling with SCHED_DEADLINE on a certain core? In contrast, how is it handled when you have single core systems etc.? Why this artificial limitation?

Please have also a look (you only mentioned manpage so, in case you missed it) at

https://elixir.bootlin.com/linux/latest/source/Documentation/scheduler/sched-deadline.rst#L667

and the document in general should hopefully give you the answer about why we need admission control and current limitations regarding affinities.

Quoted 2 lines

   2. How can we possibly implement this? We don't want to use SCHED_FIFO, because out-of-control tasks would freeze the entire container.

I experimented myself a bit with this kind of setup in the past and I think I made it work by pre-configuring exclusive cpusets (similarly as what detailed in the doc above) and then starting containers inside such exclusive sets with podman run --cgroup-parent option.

I don't have proper instructions yet for how to do this (plan to put them together soon-ish), but please see if you can make it work with this hint.

Best,

Juri

#2 lore raw
Quoted 1 line · On Wed, 20.11.2019, 09:50 +0100 Juri Lelli wrote:

Hi Philipp,

Hey Juri,

thanks so far; we indeed could make it work with exclusive CPU-sets.

Quoted 34 lines · Philipp Stanner
Quoted 11 lines

from implementing our intended architecture.

Now, the questions we're having are:

1. Why does the kernel do this, what is the problem with
scheduling with
SCHED_DEADLINE on a certain core? In contrast, how is it
handled when
you have single core systems etc.? Why this artificial
limitation?

Please have also a look (you only mentioned manpage so, in case you missed it) at

https://elixir.bootlin.com/linux/latest/source/Documentation/scheduler/sched-deadline.rst#L667

and the document in general should hopefully give you the answer about
why we need admission control and current limitations regarding affinities.

Quoted 4 lines

2. How can we possibly implement this? We don't want to use
SCHED_FIFO,
because out-of-control tasks would freeze the entire
container.

I experimented myself a bit with this kind of setup in the past and I think I made it work by pre-configuring exclusive cpusets (similarly as
what detailed in the doc above) and then starting containers inside such
exclusive sets with podman run --cgroup-parent option.

Quoted 3 lines

I don't have proper instructions yet for how to do this (plan to put them together soon-ish), but please see if you can make it work with this hint.

I fear I have not understood quite well yet why this "workaround" leads to (presumably) the same results as set_affinity would. From what I have read, I understand it as follows: For sched_dead, admission control tries to guarantee that the requested policy can be executed. To do so, it analyzes the current workload situation, taking especially the number of cores into account.

Now, with a pre-configured set, the kernel knows which tasks will run on which core, therefore it's able to judge wether a process can be deadline scheduled or not. But when using the default way, you could start your processes as SCHED_OTHER, set SCHED_DEADLINE as policy and later many of them could suddenly call set_affinity, desiring to run on the same core, therefore provoking collisions.

Is my understanding of the situation correct?

Merry Christmas,
P.

#3 lore raw

Hi,

Sorry for the delay in repling (Xmas + catching-up w/ emails).

Quoted 6 lines · Philipp Stanner
Quoted 1 line · On Wed, 20.11.2019, 09:50 +0100 Juri Lelli wrote:

Hi Philipp,

Hey Juri,

thanks so far; we indeed could make it work with exclusive CPU-sets.

Good. :-)

Quoted 46 lines · Philipp Stanner
Quoted 34 lines
Quoted 11 lines

from implementing our intended architecture.

Now, the questions we're having are:

1. Why does the kernel do this, what is the problem with
scheduling with
SCHED_DEADLINE on a certain core? In contrast, how is it
handled when
you have single core systems etc.? Why this artificial
limitation?

Please have also a look (you only mentioned manpage so, in case you missed it) at

https://elixir.bootlin.com/linux/latest/source/Documentation/scheduler/sched-deadline.rst#L667

and the document in general should hopefully give you the answer about
why we need admission control and current limitations regarding affinities.

Quoted 4 lines

2. How can we possibly implement this? We don't want to use
SCHED_FIFO,
because out-of-control tasks would freeze the entire
container.

I experimented myself a bit with this kind of setup in the past and I think I made it work by pre-configuring exclusive cpusets (similarly as
what detailed in the doc above) and then starting containers inside such
exclusive sets with podman run --cgroup-parent option.

Quoted 3 lines

I don't have proper instructions yet for how to do this (plan to put them together soon-ish), but please see if you can make it work with this hint.

I fear I have not understood quite well yet why this "workaround" leads to (presumably) the same results as set_affinity would. From what I have read, I understand it as follows: For sched_dead, admission control tries to guarantee that the requested policy can be executed. To do so, it analyzes the current workload situation, taking especially the number of cores into account.

Quoted 6 lines

Now, with a pre-configured set, the kernel knows which tasks will run on which core, therefore it's able to judge wether a process can be deadline scheduled or not. But when using the default way, you could start your processes as SCHED_OTHER, set SCHED_DEADLINE as policy and later many of them could suddenly call set_affinity, desiring to run on the same core, therefore provoking collisions.

But setting affinity would still have to pass admission control, and should fail in the case you are describing (IIUC).

https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L5433

Best,

Juri