Skip to contentSkip to conversation

SCHED_DEADLINE as user

DISCUSSION Tim Blechmann 7 participants 25 messages Aug 15, 2018 – Aug 23, 2018 3 messages not in the archive

lore
Share & tools
lore
#0 lore raw

hi all,

i wonder, what's the preferred way to obtain SCHED_DEADLINE privileges as non-root user?
for SCHED_RR/SCHED_FIFO i'm typically using pam_limits/limits.conf, but i haven't found any resources on how SCHED_DEADLINE can be obtained ...

thanks in advance,
tim

#1 lore raw
Quoted 4 lines · Tim Blechmann

hi all,

i wonder, what's the preferred way to obtain SCHED_DEADLINE privileges as non-root user?

Hi Tim,

It is still not possible to use sched deadline as a non-root user :-(.

Thanks
-- Daniel

Quoted 6 lines

for SCHED_RR/SCHED_FIFO i'm typically using pam_limits/limits.conf, but i haven't found any resources on how SCHED_DEADLINE can be obtained ...

thanks in advance,
tim

#2 lore raw

Hi Tim,

Quoted 11 lines · Daniel Bristot de Oliveira
Quoted 4 lines · Tim Blechmann

hi all,

i wonder, what's the preferred way to obtain SCHED_DEADLINE privileges as non-root user?

Hi Tim,

It is still not possible to use sched deadline as a non-root user :-(.

Indeed, but Tim, is this hindering you from experimenting/using DEADLINE?

Asking because we might want to have a look at how to relax the current privileges requirement.

Best,

- Juri

#3 lore raw

hi juri,

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

i wonder, what's the preferred way to obtain SCHED_DEADLINE privileges as non-root user?

Hi Tim,

It is still not possible to use sched deadline as a non-root user :-(.

Indeed, but Tim, is this hindering you from experimenting/using DEADLINE?

it's a showstopper for using it in audio applications, which are running as user.
audio applications typically use SCHED_FIFO. however they are rate-monotonic by design, so i wanted to evaluate how SCHED_DEADLINE behaves here.

Quoted 2 lines

Asking because we might want to have a look at how to relax the current privileges requirement.

please :)

cheers,
tim

#4 lore raw
Quoted 9 lines · Tim Blechmann

hi juri,

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

i wonder, what's the preferred way to obtain SCHED_DEADLINE privileges as non-root user?

Hi Tim,

It is still not possible to use sched deadline as a non-root user :-(.

Quoted 8 lines
Quoted 2 lines

Indeed, but Tim, is this hindering you from experimenting/using DEADLINE?

it's a showstopper for using it in audio applications, which are running as user.
audio applications typically use SCHED_FIFO. however they are rate-monotonic by design, so i wanted to evaluate how SCHED_DEADLINE behaves here.

Right. Exactly what I started to think about lately as well.

Quoted 5 lines
Quoted 3 lines

Asking because we might want to have a look at how to relax the current privileges requirement.

please :)

Anything to try attracting new users. :-P

More seriously, could you please describe in a bit more details your specific use case? Based on JACK I guess? Is it the application you are interested into public available somewhere?

Thanks,

- Juri

#6 lore raw
In reply to a message not in the archive
Quoted 25 lines · Tim Blechmann
Quoted 10 lines
Quoted 3 lines

audio applications typically use SCHED_FIFO. however they are rate-monotonic by design, so i wanted to evaluate how SCHED_DEADLINE behaves here.

Right. Exactly what I started to think about lately as well.

Quoted 4 lines
Quoted 3 lines

Asking because we might want to have a look at how to relax the current privileges requirement.

Quoted 7 lines
Quoted 1 line

please :)

Anything to try attracting new users. :-P

More seriously, could you please describe in a bit more details your specific use case? Based on JACK I guess? Is it the application you are interested into public available somewhere?

it could be based on JACK or natively on ALSA. furthermore the fun part starts when trying to distribute work to multiple threads: during the time slice of one audio callback (could be as low as 1.3ms), we need to wake helper threads and (busy?)wait to collect all the results. (while one of the applications is open source, it's not entirely straight-forward to use)

Quoted 5 lines

for me it's more an investigation if SCHED_DEADLINE is actually feasible in real-world audio application these days. tommaso used to have a research prototype [1], built directly into JACK. however currently i'd be more interested in using it in ALSA. but yeah, it's more an investigation as SCHED_FIFO does a decent job for my use case ...

I (almost) see. Thanks for the details.

Quoted 1 line

[1] http://retis.sssup.it/~tommaso/publications/LAC-2011.pdf

So, this was IIRC using AQuoSA hierarchical RT on EDF scheduler implementation; which it was handy to easy the task of assigning CBS server parameters to JACK clients (graph).

In your case - assuming you could already use DEADLINE for !root - do you think you would need to use the same interface for the multiple threads you are distributing work among? Or would you be fine with assigning runtime/deadline for each one of those?

BTW, cc-ing Tommaso and Luca. Discussion started from
https://www.spinics.net/lists/linux-rt-users/msg19296.html

#7 lore raw
Quoted 4 lines

In your case - assuming you could already use DEADLINE for !root - do you think you would need to use the same interface for the multiple threads you are distributing work among? Or would you be fine with assigning runtime/deadline for each one of those?

that's a tricky question and depends on the use-case: for me it's "good enough" to use something like:

main audio thread:
while running:
snd_pcm_read // blocks until data is delivered from ALSA wake helper(s)
do work()
sync_with_helper(s)
snd_pcm_write // blocks until data is delivered to ALSA

alsa is basically driven by the hardware, which delivers/requests data every ~1.3ms.

distributing the work is a little tricky: to avoid excessive scheduling overhead, the worker threads are typically woken up (after snd_pcm_read) and the result is collected where sync_with_helper() typically boils down to busy waiting. in this case, the workers as rate-monotonic in a similar manner as the main audio thread.

one could also use lock-free queues with semaphores to wake only as many threads as needed for the graph topology (which can depend on user input). in this case SCHED_FIFO sounds more suitable.

--

from a practical point of view: i'm not targeting a safety critical system. one advantage i'm seeing of DEADLINE over FIFO/RR is that it's easier to prevent lockups (e.g. when a user overloads the system). in the linux audio world this is typically done by a watchdog thread. the other part of the unix world (mach) is using time-constraint threads by default for audio use cases. so i'd assume that DEADLINE would free me from the need to spawn the watchdog thread ...

cheers,
tim

#8 lore raw
Quoted 16 lines · Tim Blechmann
Quoted 4 lines

In your case - assuming you could already use DEADLINE for !root - do you think you would need to use the same interface for the multiple threads you are distributing work among? Or would you be fine with assigning runtime/deadline for each one of those?

that's a tricky question and depends on the use-case: for me it's "good enough" to use something like:

main audio thread:
while running:
snd_pcm_read // blocks until data is delivered from ALSA wake helper(s)
do work()
sync_with_helper(s)
snd_pcm_write // blocks until data is delivered to ALSA

Quoted 2 lines

alsa is basically driven by the hardware, which delivers/requests data every ~1.3ms.

Hi Tim!

Quoted 6 lines

distributing the work is a little tricky: to avoid excessive scheduling overhead, the worker threads are typically woken up (after snd_pcm_read) and the result is collected where sync_with_helper() typically boils down to busy waiting. in this case, the workers as rate-monotonic in a similar manner as the main audio thread.

When you say "rate-monitonic", do you mean that the worker threads with smaller period (high rate, so) have a higher fixed-priority? If so, what are the other periods?

as sync_with_helper() does a busy-wait, the "main audio thread" has a relative lower priority than the worker threads, right?

Quoted 5 lines

one could also use lock-free queues with semaphores to wake only as many threads as needed for the graph topology (which can depend on user input). in this case SCHED_FIFO sounds more suitable.
--

Quoted 7 lines

from a practical point of view: i'm not targeting a safety critical system. one advantage i'm seeing of DEADLINE over FIFO/RR is that it's easier to prevent lockups (e.g. when a user overloads the system). in the linux audio world this is typically done by a watchdog thread. the other part of the unix world (mach) is using time-constraint threads by default for audio use cases. so i'd assume that DEADLINE would free me from the need to spawn the watchdog thread ...

So, your main wish here is to avoid having a watchdog thread to "throttle" misbehaving workload/setup?

For curiosity, what are the "time-constraints" setup used on mach?

-- Daniel

#9 lore raw
Quoted 17 lines
Quoted 8 lines

main audio thread:
while running:
snd_pcm_read // blocks until data is delivered from ALSA wake helper(s)
do work()
sync_with_helper(s)
snd_pcm_write // blocks until data is delivered to ALSA

Quoted 8 lines

alsa is basically driven by the hardware, which delivers/requests data every ~1.3ms.

distributing the work is a little tricky: to avoid excessive scheduling overhead, the worker threads are typically woken up (after snd_pcm_read) and the result is collected where sync_with_helper() typically boils down to busy waiting. in this case, the workers as rate-monotonic in a similar manner as the main audio thread.

Quoted 3 lines

When you say "rate-monitonic", do you mean that the worker threads with smaller period (high rate, so) have a higher fixed-priority? If so, what are the other periods?

the period will be the same period: i wake up the workers once per period of the main thread, distribute the work and collect it at the end. so "period" is basically the same, "runtime" a little lower.

Quoted 2 lines

as sync_with_helper() does a busy-wait, the "main audio thread" has a relative lower priority than the worker threads, right?

typically all threads have the same priority and there will be at most N threads for N physical CPU cores. so each thread will have a CPU to run on and no thread will be preempted.

Quoted 2 lines

So, your main wish here is to avoid having a watchdog thread to "throttle" misbehaving workload/setup?

mainly ... and nerdy research if SCHED_DEADLINE is useful for audio applications ... ;)

Quoted 1 line

For curiosity, what are the "time-constraints" setup used on mach?

it's pretty similar to the SCHED_DEADLINE constraints:

from:
https://opensource.apple.com/source/xnu/xnu-124.7/osfmk/mach/thread_policy.h.auto.html

Quoted 18 lines

/*
* THREAD_TIME_CONSTRAINT_POLICY:
*
* This scheduling mode is for threads which have real time
* constraints on their execution.
*
* Parameters:
*
* period: This is the nominal amount of time between separate
* processing arrivals, specified in absolute time units. A
* value of 0 indicates that there is no inherent periodicity in
* the computation.
*
* computation: This is the nominal amount of computation
* time needed during a separate processing arrival, specified
* in absolute time units.
*
* constraint: This is the maximum amount of real time that

Quoted 8 lines

* may elapse from the start of a separate processing arrival
* to the end of computation for logically correct functioning,
* specified in absolute time units. Must be (>= computation).
* Note that latency = (constraint - computation).
*
* preemptible: This indicates that the computation may be
* interrupted, subject to the constraint specified above. */

cheers,
tim

#15 lore raw
In reply to Tim Blechmann

[copying Alessio]

Hi,

Quoted 11 lines · Tim Blechmann

that's a tricky question and depends on the use-case: for me it's "good enough" to use something like:

main audio thread:
while running:
snd_pcm_read // blocks until data is delivered from ALSA wake helper(s)
do work()
sync_with_helper(s)
snd_pcm_write // blocks until data is delivered to ALSA

Quoted 2 lines

alsa is basically driven by the hardware, which delivers/requests data every ~1.3ms.

this is exactly the use-case we considered in that old paper back in LAC 2011, and let me just add that there is more experimentation and details and results in Giacomo's MSc thesis:

  https://retis.sssup.it/?q=node/77

FYI, we restarted activities on this use-case just recently, considering a couple of further dimensions:

1) support for heterogeneous platforms with the aim of achieving a sound support for Arm big.LITTLE on Android, and apply it to the Android video & audio processing pipelines;

2) proper consideration of power switching capabilities of the platform (assuming the user would block the CPU frequency to its maximum is not really multimedia-friendly);

3) support for multi-threaded processing workflows on multi-cores, as typically needed by (high-performance) audio applications, either with a single multi-threaded audio processing client, or, e.g., with JACK, with a DAG of computations that can take advantage of underlying multi-core processing.

In this context, we've been playing also with the hierarchical extension to SCHED_DEADLINE we sent last year on LKML

  https://lkml.org/lkml/2017/3/31/658

I hope we can share some good write-up & experimental results pretty soon about some of the above...

Quoted 9 lines

distributing the work is a little tricky: to avoid excessive scheduling overhead, the worker threads are typically woken up (after snd_pcm_read) and the result is collected where sync_with_helper() typically boils down to busy waiting. in this case, the workers as rate-monotonic in a similar manner as the main audio thread.

one could also use lock-free queues with semaphores to wake only as many threads as needed for the graph topology (which can depend on user input). in this case SCHED_FIFO sounds more suitable.

proper handling of task dependencies & synchronization, and associated priority inversion issues, is also an aspect we've been and we're still working on, as when you mix DEADLINE tasks in the picture, it's a big mess.

Quoted 7 lines

from a practical point of view: i'm not targeting a safety critical system. one advantage i'm seeing of DEADLINE over FIFO/RR is that it's easier to prevent lockups (e.g. when a user overloads the system). in the linux audio world this is typically done by a watchdog thread. the other part of the unix world (mach) is using time-constraint threads by default for audio use cases. so i'd assume that DEADLINE would free me from the need to spawn the watchdog thread ...

I'm not sure I'm getting this: AFAIU, from this viewpoint, what you could get with DEADLINE, is very similar to what you can get already with RT throttling on RT (RR/FIFO), setting up properly CPU cgroups & limits. Why do you think DEADLINE would allow you to avoid a watchdog thread, that you need with RT instead ?

Thanks,

    T.

Signature · 4 lines
Tommaso Cucinotta, Computer Engineering PhD
Associate Professor at the Real-Time Systems Laboratory (ReTiS)
Scuola Superiore Sant'Anna, Pisa, Italy
http://retis.sssup.it/people/tommaso
#12 lore raw
In reply to a message not in the archive

On 21/08/18 00:45, Tim Blechmann wrote:

[...]

Quoted 4 lines

apart from that, i'm only missing two features:

* using SCHED_DEADLINE from user applications is kind of a showstopper for me

Right. Would you be up for continuing experimenting with DEADLINE (while we get our heads around removing this constraint) by - ugly hack warning
- removing the check at

https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L4216

?

I guess it would be good if you could provide even more feedback than you already did, so that we can plan ahead what needs fixing/changing or implementing.

BTW, thanks a lot already for this discussion! :-)

Quoted 5 lines

* a nicer glibc wrapper, that allows me to use pthreads rather than using syscall: it's all pretty easy to obtain the tid for a calling thread (gettid), but it seems to be impossible to obtain the tid from pthread_t without some hacks which are way too dirty to share in
public ...

Indeed. We have been basically relying on

https://github.com/scheduler-tools/rt-app/tree/master/libdl

so far. But, I guess it's time - it has been time for a while - that we reach glibc guys and try to understand why this is not yet all in the standard libraries.

#14 lore raw
In reply to a message not in the archive

Hi all,

+cc-ing Alessio, who has been working on this just recently on Android (will tell more answering on a different msg).

Quoted 1 line · Tim Blechmann

the mach API also has this notion of "period == 0", but apart from that it's quite similar.

in terms of API, XNU time constrained threads look like DEADLINE, however in terms of kernel behavior/implementation behind that, it's something completely different: last time I had a glance at this, it was .... ehm .... probably just during that old Linux Audio Conference (LAC) back in 2011, where the take away was: 1) in the kernel there's nothing like the CBS or deadline-based scheduling, but just a heuristic boosting a bit more those tasks, accompanied by usual heuristics to de-boost in case the task becomes CPU-intensive (similar to CFS heuristics for interactive-vs-batch); 2) the only user of that API seemed to be some sound daemon on OS-X, that had an almost random value set as "runtime", accompanied by a comment "this just works" ;-P

But, it was long ago, if anyone has a pointer to what XNU does with time constrained threads in these days....

Quoted 4 lines

apart from that, i'm only missing two features:

* using SCHED_DEADLINE from user applications is kind of a showstopper for me

unprivileged tasks using DEADLINE remained a request for quite a long time -- that old single-processor implementation (AQuoSA) we used long ago with JACK had such a feature, that was implementing this model:

  http://retis.sssup.it/~tommaso/papers/rtas08.php

the criticality for DEADLINE is that those tasks preempt RT99 ones, so even with controlled runtime/deadline boundaries....

Quoted 5 lines

* a nicer glibc wrapper, that allows me to use pthreads rather than using syscall: it's all pretty easy to obtain the tid for a calling thread (gettid), but it seems to be impossible to obtain the tid from pthread_t without some hacks which are way too dirty to share in
public ...

well-known "pain" as well, thanks.

    T.

Signature · 4 lines
Tommaso Cucinotta, Computer Engineering PhD
Associate Professor at the Real-Time Systems Laboratory (ReTiS)
Scuola Superiore Sant'Anna, Pisa, Italy
http://retis.sssup.it/people/tommaso
#16 lore raw
In reply to Tommaso Cucinotta
Quoted 26 lines
Quoted 2 lines

alsa is basically driven by the hardware, which delivers/requests data every ~1.3ms.

this is exactly the use-case we considered in that old paper back in LAC 2011, and let me just add that there is more experimentation and details and results in Giacomo's MSc thesis:

  https://retis.sssup.it/?q=node/77

FYI, we restarted activities on this use-case just recently, considering a couple of further dimensions:

1) support for heterogeneous platforms with the aim of achieving a sound support for Arm big.LITTLE on Android, and apply it to the Android video & audio processing pipelines;

2) proper consideration of power switching capabilities of the platform (assuming the user would block the CPU frequency to its maximum is not really multimedia-friendly);

3) support for multi-threaded processing workflows on multi-cores, as typically needed by (high-performance) audio applications, either with a single multi-threaded audio processing client, or, e.g., with JACK, with a DAG of computations that can take advantage of underlying multi-core processing.

Quoted 7 lines

In this context, we've been playing also with the hierarchical extension to SCHED_DEADLINE we sent last year on LKML

  https://lkml.org/lkml/2017/3/31/658

I hope we can share some good write-up & experimental results pretty soon about some of the above...

very interesting! thanks for the links ...

Quoted 7 lines

I'm not sure I'm getting this: AFAIU, from this viewpoint, what you could get
with DEADLINE, is very similar to what you can get already with RT throttling
on RT (RR/FIFO), setting up properly CPU cgroups & limits. Why do you think DEADLINE would allow you to avoid a watchdog thread, that you need with RT instead ?

interesting, wasn't aware of rt throttling ... (haven't done much linux audio developments for almost 10 years). will give it another read ...

cheers,
tim

#13 lore raw
In reply to Juri Lelli
Quoted 29 lines · Juri Lelli

On 21/08/18 00:45, Tim Blechmann wrote:

[...]

Quoted 4 lines

apart from that, i'm only missing two features:

* using SCHED_DEADLINE from user applications is kind of a showstopper for me

Right. Would you be up for continuing experimenting with DEADLINE (while we get our heads around removing this constraint) by - ugly hack warning
- removing the check at

https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L4216

?

I guess it would be good if you could provide even more feedback than you already did, so that we can plan ahead what needs fixing/changing or implementing.

BTW, thanks a lot already for this discussion! :-)

Quoted 5 lines

* a nicer glibc wrapper, that allows me to use pthreads rather than using syscall: it's all pretty easy to obtain the tid for a calling thread (gettid), but it seems to be impossible to obtain the tid from pthread_t without some hacks which are way too dirty to share in
public ...

Quoted 7 lines

Indeed. We have been basically relying on

https://github.com/scheduler-tools/rt-app/tree/master/libdl

so far. But, I guess it's time - it has been time for a while - that we reach glibc guys and try to understand why this is not yet all in the standard libraries.

FYI, I added both points to the list of known issues. Feel free to add details.

https://github.com/jlelli/linux/issues

Best,

- Juri

#18 lore raw
In reply to luca abeni

Hi,

Quoted 4 lines · luca abeni

Hi Juri,

sorry for the late reply (I am just back from vacations), and thanks for cc-ing me.

Welcome back! :-)

Quoted 2 lines

I see you opened a bug on github about this, so I am going to add more details there,

I use github issues to keep track of things, but I guess mailing list discussions are to be preferred though (so that more people can potentially follow).

Quoted 4 lines

but basically I think that in order to use SCHED_DEADLINE as non-root we need to:
1) Disable the boosting mechanism (not the inheritance, just the "soft enforcement behaviour" of tasks holding mutexes)

But then what is a sane inheritance mechanism? Walk the chain and find the next potential deadline to inherit for the current boosted (still runtime enforced) task before throttling it? Not sure it's going to be any easier than the proxy solution. :-/

Quoted 4 lines

2) Implement some mechanism to limit the amount of dl bandwidth a user can allocate to itself (I think the cgroup-based approach we discussed some time ago should be OK... If I remember well, you even had a patch implementing it :)

I think most (all?) distributions today run with CONFIG_RT_GROUP_SCHED disabled, we should also think about a solution that doesn't rely on that interface. Maybe the existing system wide sched_rt_{period, runtime}_us are enough?

Best,

- Juri

#19 lore raw

Hi Juri,

Quoted 4 lines · On Thu, 23 Aug 2018 12:23:50 +0200 Juri Lelli <[email protected]> wrote:
Quoted 2 lines

sorry for the late reply (I am just back from vacations), and thanks for cc-ing me.

Welcome back! :-)

Thanks :)

Quoted 6 lines
Quoted 2 lines

I see you opened a bug on github about this, so I am going to add more details there,

I use github issues to keep track of things, but I guess mailing list discussions are to be preferred though (so that more people can potentially follow).

Ok

Quoted 6 lines
Quoted 4 lines

but basically I think that in order to use SCHED_DEADLINE as non-root we need to:
1) Disable the boosting mechanism (not the inheritance, just the "soft enforcement behaviour" of tasks holding mutexes)

But then what is a sane inheritance mechanism?

In my understanding (please correct me if I am wrong), this is an orthogonal issue: if I understand well, the issue preventing non-root usage of SCHED_DEADLINE is that a task inheriting a dl entity is not throttled (when the current runtime rrives to 0, the deadline is postponed, but the task stays schedulable). So, I think that removing this behaviour should allow to use SCHED_DEADLINE without starving other tasks...

Then, there is the issue about the deadline and runtime to inherit. And I agree that this is important (and the solution is not easy), but you have this issue even if you use the current "dl_boosted" behaviour... No?

Quoted 4 lines

Walk the chain and find
the next potential deadline to inherit for the current boosted (still runtime enforced) task before throttling it? Not sure it's going to be any easier than the proxy solution. :-/

Right; this is not easy... But I think it is not related with the issue we are discussing (if I understand this email thread well). Yes, it has to be fixed, but it does not prevent non-root usage. Or am I missing something?

Quoted 8 lines
Quoted 4 lines

2) Implement some mechanism to limit the amount of dl bandwidth a user can allocate to itself (I think the cgroup-based approach we discussed some time ago should be OK... If I remember well, you
even had a patch implementing it :)

I think most (all?) distributions today run with CONFIG_RT_GROUP_SCHED disabled, we should also think about a solution that doesn't rely on that interface.

I guess CONFIG_RT_GROUP_SCHED is often disabled because it ends up changing the "traditional" SCHED_{RR/FIFO} behaviour. So, maybe the solution is to have a different dl_{runtime,period} interface in control groups (enabled by CONFIG_DL_GROUP_SCHED :). CONFIG_DL_GROUP_SCHED does not change the scheduling behaviour, but only the admission test... So, enabling it could be safer than enabling CONFIG_RT_GROUP_SCHED.

Quoted 2 lines

Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?

I do not know... A cgroup-based interface looks more powerful (and not so difficult to implement... :), and would allow the sysadmin to decide which users can use SCHED_DEADLINE, how much SCHED_DEADLINE bandwidth can each user/group use, etc...

Luca

#17 lore raw
In reply to Juri Lelli

Hi Juri,

sorry for the late reply (I am just back from vacations), and thanks for cc-ing me.

I see you opened a bug on github about this, so I am going to add more details there, but basically I think that in order to use SCHED_DEADLINE as non-root we need to:
1) Disable the boosting mechanism (not the inheritance, just the "soft enforcement behaviour" of tasks holding mutexes)
2) Implement some mechanism to limit the amount of dl bandwidth a user can allocate to itself (I think the cgroup-based approach we discussed some time ago should be OK... If I remember well, you even had a patch implementing it :)

Luca

Quoted 48 lines · On Mon, 20 Aug 2018 10:54:20 +0200 Juri Lelli <[email protected]> wrote:
Quoted 25 lines · Tim Blechmann
Quoted 10 lines
Quoted 3 lines

audio applications typically use SCHED_FIFO. however they are rate-monotonic by design, so i wanted to evaluate how SCHED_DEADLINE behaves here.

Right. Exactly what I started to think about lately as well.

Quoted 4 lines
Quoted 3 lines

Asking because we might want to have a look at how to relax the current privileges requirement.

Quoted 7 lines
Quoted 1 line

please :)

Anything to try attracting new users. :-P

More seriously, could you please describe in a bit more details your specific use case? Based on JACK I guess? Is it the application you are interested into public available somewhere?

it could be based on JACK or natively on ALSA. furthermore the fun part starts when trying to distribute work to multiple threads:
during the time slice of one audio callback (could be as low as 1.3ms), we need to wake helper threads and (busy?)wait to collect all the results. (while one of the applications is open source, it's not entirely straight-forward to use)

Quoted 6 lines

for me it's more an investigation if SCHED_DEADLINE is actually feasible in real-world audio application these days. tommaso used to have a research prototype [1], built directly into JACK. however currently i'd be more interested in using it in ALSA. but yeah, it's more an investigation as SCHED_FIFO does a decent job for my use case ...

I (almost) see. Thanks for the details.

Quoted 1 line

[1] http://retis.sssup.it/~tommaso/publications/LAC-2011.pdf

So, this was IIRC using AQuoSA hierarchical RT on EDF scheduler implementation; which it was handy to easy the task of assigning CBS server parameters to JACK clients (graph).

In your case - assuming you could already use DEADLINE for !root - do you think you would need to use the same interface for the multiple threads you are distributing work among? Or would you be fine with assigning runtime/deadline for each one of those?

BTW, cc-ing Tommaso and Luca. Discussion started from
https://www.spinics.net/lists/linux-rt-users/msg19296.html

#20 lore raw
In reply to luca abeni
Quoted 2 lines · luca abeni

On Thu, 23 Aug 2018 12:23:50 +0200
Juri Lelli <[email protected]> wrote:

[...]

Quoted 9 lines
Quoted 1 line

But then what is a sane inheritance mechanism?

In my understanding (please correct me if I am wrong), this is an orthogonal issue: if I understand well, the issue preventing non-root usage of SCHED_DEADLINE is that a task inheriting a dl entity is not throttled (when the current runtime rrives to 0, the deadline is postponed, but the task stays schedulable). So, I think that removing this behaviour should allow to use SCHED_DEADLINE without starving other tasks...

Right, potential starvation would be gone...

Quoted 4 lines

Then, there is the issue about the deadline and runtime to inherit. And I agree that this is important (and the solution is not easy), but you have this issue even if you use the current "dl_boosted" behaviour... No?

... but, while it's true that current inheritance has problems w/ o w/o boosting behaviour, I wonder if the problem might be more painful for a normal user that it's still under runtime enforcement. Disabling enforcement seems to hide a bit the fact that we need proper inheritance. :-(

Quoted 9 lines
Quoted 4 lines

Walk the chain and find
the next potential deadline to inherit for the current boosted (still runtime enforced) task before throttling it? Not sure it's going to be any easier than the proxy solution. :-/

Right; this is not easy... But I think it is not related with the issue we are discussing (if I understand this email thread well). Yes, it has to be fixed, but it does not prevent non-root usage. Or am I missing something?

It depends on how bad we think it's what I said above I guess.

Quoted 9 lines
Quoted 8 lines
Quoted 4 lines

2) Implement some mechanism to limit the amount of dl bandwidth a user can allocate to itself (I think the cgroup-based approach we discussed some time ago should be OK... If I remember well, you
even had a patch implementing it :)

I think most (all?) distributions today run with CONFIG_RT_GROUP_SCHED disabled, we should also think about a solution that doesn't rely on that interface.

Quoted 7 lines

I guess CONFIG_RT_GROUP_SCHED is often disabled because it ends up changing the "traditional" SCHED_{RR/FIFO} behaviour. So, maybe the solution is to have a different dl_{runtime,period} interface in control groups (enabled by CONFIG_DL_GROUP_SCHED :). CONFIG_DL_GROUP_SCHED does not change the scheduling behaviour, but only the admission test... So, enabling it could be safer than enabling CONFIG_RT_GROUP_SCHED.

Not sure if adding yet another config switch is acceptable. FWIW, I'd prefer not to add it, also looking back at all the problems RT_GROUP_ SCHED seems to pose/have posed, but if turns to be the only option..

Quoted 7 lines
Quoted 2 lines

Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?

I do not know... A cgroup-based interface looks more powerful (and not so difficult to implement... :), and would allow the sysadmin to decide which users can use SCHED_DEADLINE, how much SCHED_DEADLINE bandwidth can each user/group use, etc...

How about extending PAM limits instead? It looks like it's what (e.g.) audio users rely on already [1]. It is maybe possible to add dlruntime, dlperiod, dldeadline parameters in there?

1 - https://fedoraproject.org/wiki/JACK_Audio_Connection_Kit#Running_Jack_in_Realtime_mode

#21 lore raw
Quoted 18 lines · On Thu, 23 Aug 2018 14:37:39 +0200 Juri Lelli <[email protected]> wrote:
Quoted 2 lines · luca abeni

On Thu, 23 Aug 2018 12:23:50 +0200
Juri Lelli <[email protected]> wrote:

[...]

Quoted 9 lines
Quoted 1 line

But then what is a sane inheritance mechanism?

In my understanding (please correct me if I am wrong), this is an orthogonal issue: if I understand well, the issue preventing non-root usage of SCHED_DEADLINE is that a task inheriting a dl entity is not throttled (when the current runtime rrives to 0, the deadline is postponed, but the task stays schedulable). So, I think that removing this behaviour should allow to use SCHED_DEADLINE without starving other tasks...

Right, potential starvation would be gone...

Quoted 8 lines
Quoted 4 lines

Then, there is the issue about the deadline and runtime to inherit. And I agree that this is important (and the solution is not easy), but you have this issue even if you use the current "dl_boosted" behaviour... No?

... but, while it's true that current inheritance has problems w/ o w/o boosting behaviour, I wonder if the problem might be more painful for a normal user that it's still under runtime enforcement.

I think that the only way to have an answer to this doubt is to try :)

Quoted 3 lines

Disabling
enforcement seems to hide a bit the fact that we need proper inheritance. :-(

My impression is that it hides the problem in some situations, but the Murphy law ensures that the issue will appear when the quality of the audio is more important :)

Luca

#22 lore raw
In reply to Juri Lelli

On Thu, 23 Aug 2018 14:37:39 +0200
Juri Lelli <[email protected]> wrote:
[...]

Quoted 11 lines
Quoted 7 lines
Quoted 2 lines

Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?

I do not know... A cgroup-based interface looks more powerful (and not so difficult to implement... :), and would allow the sysadmin to decide which users can use SCHED_DEADLINE, how much SCHED_DEADLINE bandwidth can each user/group use, etc...

How about extending PAM limits instead? It looks like it's what (e.g.) audio users rely on already [1]. It is maybe possible to add dlruntime, dlperiod, dldeadline parameters in there?

Ok, I do not know much about these PAM limits... Where can I find more information? What do we need to implement exactly, to add these dlruntime, dlperiod, dldeadline parameters?

Thanks,
Luca

#23 lore raw
Quoted 11 lines · luca abeni

On Thu, 23 Aug 2018 14:37:39 +0200
Juri Lelli <[email protected]> wrote:
[...]

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?

I do not know... A cgroup-based interface looks more powerful (and not so difficult to implement... :), and would allow the sysadmin to decide which users can use SCHED_DEADLINE, how much SCHED_DEADLINE bandwidth can each user/group use, etc...

Quoted 6 lines
Quoted 3 lines

How about extending PAM limits instead? It looks like it's what (e.g.) audio users rely on already [1]. It is maybe possible to add dlruntime, dlperiod, dldeadline parameters in there?

Ok, I do not know much about these PAM limits... Where can I find more information?

The link I shared already gives some information, man pam_limits and limits.conf also.

Quoted 2 lines

What do we need to implement exactly, to add these dlruntime, dlperiod, dldeadline parameters?

Not sure yet "exactly" :-), but I'd say something along these lines

https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L4198

#24 lore raw
Quoted 22 lines · On Thu, 23 Aug 2018 15:09:57 +0200 Juri Lelli <[email protected]> wrote:
Quoted 11 lines · luca abeni

On Thu, 23 Aug 2018 14:37:39 +0200
Juri Lelli <[email protected]> wrote:
[...]

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?

I do not know... A cgroup-based interface looks more powerful (and not so difficult to implement... :), and would allow the sysadmin to decide which users can use SCHED_DEADLINE, how much SCHED_DEADLINE bandwidth can each user/group use, etc...

Quoted 6 lines
Quoted 3 lines

How about extending PAM limits instead? It looks like it's what (e.g.) audio users rely on already [1]. It is maybe possible to add dlruntime, dlperiod, dldeadline parameters in there?

Ok, I do not know much about these PAM limits... Where can I find more information?

The link I shared already gives some information, man pam_limits and limits.conf also.

Quoted 6 lines
Quoted 2 lines

What do we need to implement exactly, to add these dlruntime, dlperiod, dldeadline parameters?

Not sure yet "exactly" :-), but I'd say something along these lines

https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L4198

Ok, thanks; I think I got it: we need to extend the "rlim" array (in signal_struct) adding fields for "RLIMIT_DLRUNTIME" and friends (actually, I think a "RLIMIT_UTILIZATION" field is also needed). And then we need to add checks for these fields in __sched_setscheduler(), etc...

Maybe I am misunderstanding this stuff, but the cgroup-based approach looks simpler / more elegant :)

Thanks,
Luca

#25 lore raw
In reply to luca abeni
Quoted 16 lines · On Thursday 23 August 2018 09:00:40 luca abeni wrote:

On Thu, 23 Aug 2018 14:37:39 +0200
Juri Lelli <[email protected]> wrote:
[...]

Quoted 11 lines
Quoted 7 lines
Quoted 2 lines

Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?

I do not know... A cgroup-based interface looks more powerful (and not so difficult to implement... :), and would allow the sysadmin to decide which users can use SCHED_DEADLINE, how much SCHED_DEADLINE bandwidth can each user/group use, etc...

How about extending PAM limits instead? It looks like it's what (e.g.) audio users rely on already [1]. It is maybe possible to add dlruntime, dlperiod, dldeadline parameters in there?

Quoted 7 lines

Ok, I do not know much about these PAM limits... Where can I find more information? What do we need to implement exactly, to add these dlruntime, dlperiod, dldeadline parameters?

Thanks,
Luca

Are you folks interested in input from a real world user? On who at the moment is building about half these kernels as you announce them on a pi-3b, the same pi-3b running 1500 lbs of a Sheldon 11"x36" lathe.

I am running linuxcnc on both x86 and armhf machinery, two lathes and two milling machines, all being moved by stepper motors.

Steppers, if the steps are being software generated, are very sensitive to timing wibbles in a stream of steps, with a 5 microsecond wobble in the timing haveing serious effects on the available torque by causing a speedup and slowdown. This severely limits the speed at which the motor can be driven without stalling or losing steps, which in a stepper system is the equivalent of losing its home reference which=wrecked part. I have one machine still depending on software stepping. I could easily triple the speed of that machine by spending another 100 dollars to offload that to an fpga based interface card. As it is the base thread runs at about 28 microseconds. That also represents the granularity of a speed change. So it of late has been relegated to doing very precise work via EDM, which is slow anyway.

In short, linuxcnc, and its best rtai kernels can do pergaps 10"/minute whereas the fpga cards can move that same machine at 100 ipm. Any help in getting the latency down means we can do things faster.

LinuxCNC is also run as a common user, never as root. Thats my $0.02 from a users viewpoint.

Signature · 6 lines
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>
#26 lore raw
Quoted 2 lines · Gene Heskett

LinuxCNC is also run as a common user, never as root. Thats my $0.02 from a users viewpoint.

Given that these are special-purpose machines, can I ask why you wouldn't run as root if it gave better performance?

Chris

#27 lore raw
Quoted 8 lines · On Thursday 23 August 2018 12:01:41 Chris Friesen wrote:
Quoted 2 lines · Gene Heskett

LinuxCNC is also run as a common user, never as root. Thats my $0.02 from a users viewpoint.

Given that these are special-purpose machines, can I ask why you wouldn't run as root if it gave better performance?

Chris

Its a rather bad habit, and if I get used to doing it on this machine, I might do it on one of the metal carvers, which has a tendency to mess with the user perms. Root or known user is all the same to lcnc AFAIK. I have done it 3 or 4 times, but there is no detectable diff in performance. But sometime in the last few years it has now been trained to refuse to run as root.

The kernel argument isolcpus= comes in handy occasionally on really old but still multicore hdwe but has had little or no effect since about kernel 3.4.0 built for SMP and PREEMPT. We don't use more than 2GB of d-ram for the same reason, the patches to give a 4GB capable address are also a performance penalty.

Currently running
Linux GO704 3.4-9-rtai-686-pae #1 SMP PREEMPT Debian 3.4.55-4linuxcnc i686 GNU/Linux, a 32 bit build because its faster at a context switch than the larger stack image the 64 bit build uses.

This machine, my "doitall" machine doesn't drive any of that sort of hardware, and is currently running:3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.39-1+deb8u1~bpo70+1 (2017-02-24) x86_64 GNU/Linux which will run the simulator so I can write cgode from a comfy chair. Then export it with sshfs to the machine that will actually make the swarf, aka metallic sawdust.

Thanks Chris.

Signature · 6 lines
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>