SCHED_DEADLINE as user
hi all,
i wonder, what's the preferred way to obtain SCHED_DEADLINE privileges as non-root user?
for SCHED_RR/SCHED_FIFO i'm typically using pam_limits/limits.conf, but i haven't found any resources on how SCHED_DEADLINE can be obtained ...
thanks in advance,
tim
Quoted 4 lines · Tim Blechmann
hi all,
i wonder, what's the preferred way to obtain
SCHED_DEADLINEprivileges as non-root user?
Hi Tim,
It is still not possible to use sched deadline as a non-root user :-(.
Thanks
-- Daniel
Quoted 6 lines
for SCHED_RR/SCHED_FIFO i'm typically using
pam_limits/limits.conf, but i haven't found any resources on howSCHED_DEADLINEcan be obtained ...thanks in advance,
tim
Hi Tim,
Quoted 11 lines · Daniel Bristot de Oliveira
Quoted 4 lines · Tim Blechmann
hi all,
i wonder, what's the preferred way to obtain
SCHED_DEADLINEprivileges as non-root user?Hi Tim,
It is still not possible to use sched deadline as a non-root user :-(.
Indeed, but Tim, is this hindering you from experimenting/using DEADLINE?
Asking because we might want to have a look at how to relax the current privileges requirement.
Best,
- Juri
hi juri,
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
i wonder, what's the preferred way to obtain
SCHED_DEADLINEprivileges as non-root user?Hi Tim,
It is still not possible to use sched deadline as a non-root user :-(.
Indeed, but Tim, is this hindering you from experimenting/using DEADLINE?
it's a showstopper for using it in audio applications, which are running as user.
audio applications typically use SCHED_FIFO. however they are rate-monotonic by design, so i wanted to evaluate how SCHED_DEADLINE behaves here.
Quoted 2 lines
Asking because we might want to have a look at how to relax the current privileges requirement.
please :)
cheers,
tim
Quoted 9 lines · Tim Blechmann
hi juri,
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
i wonder, what's the preferred way to obtain
SCHED_DEADLINEprivileges as non-root user?Hi Tim,
It is still not possible to use sched deadline as a non-root user :-(.
Quoted 8 lines
Quoted 2 lines
Indeed, but Tim, is this hindering you from experimenting/using DEADLINE?
it's a showstopper for using it in audio applications, which are running as user.
audio applications typically useSCHED_FIFO. however they are rate-monotonic by design, so i wanted to evaluate howSCHED_DEADLINEbehaves here.
Right. Exactly what I started to think about lately as well.
Quoted 5 lines
Quoted 3 lines
Asking because we might want to have a look at how to relax the current privileges requirement.
please :)
Anything to try attracting new users. :-P
More seriously, could you please describe in a bit more details your specific use case? Based on JACK I guess? Is it the application you are interested into public available somewhere?
Thanks,
- Juri
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 25 lines · Tim Blechmann
Quoted 10 lines
Quoted 3 lines
audio applications typically use
SCHED_FIFO. however they are rate-monotonic by design, so i wanted to evaluate howSCHED_DEADLINEbehaves here.Right. Exactly what I started to think about lately as well.
Quoted 4 lines
Quoted 3 lines
Asking because we might want to have a look at how to relax the current privileges requirement.
Quoted 7 lines
Quoted 1 line
please :)
Anything to try attracting new users. :-P
More seriously, could you please describe in a bit more details your specific use case? Based on JACK I guess? Is it the application you are interested into public available somewhere?
it could be based on JACK or natively on ALSA. furthermore the fun part starts when trying to distribute work to multiple threads: during the time slice of one audio callback (could be as low as 1.3ms), we need to wake helper threads and (busy?)wait to collect all the results. (while one of the applications is open source, it's not entirely straight-forward to use)
Quoted 5 lines
for me it's more an investigation if
SCHED_DEADLINEis actually feasible in real-world audio application these days. tommaso used to have a research prototype [1], built directly into JACK. however currently i'd be more interested in using it in ALSA. but yeah, it's more an investigation asSCHED_FIFOdoes a decent job for my use case ...
I (almost) see. Thanks for the details.
Quoted 1 line
[1] http://retis.sssup.it/~tommaso/publications/LAC-2011.pdf
So, this was IIRC using AQuoSA hierarchical RT on EDF scheduler implementation; which it was handy to easy the task of assigning CBS server parameters to JACK clients (graph).
In your case - assuming you could already use DEADLINE for !root - do you think you would need to use the same interface for the multiple threads you are distributing work among? Or would you be fine with assigning runtime/deadline for each one of those?
BTW, cc-ing Tommaso and Luca. Discussion started from
https://www.spinics.net/lists/linux-rt-users/msg19296.html
Quoted 4 lines
In your case - assuming you could already use DEADLINE for !root - do you think you would need to use the same interface for the multiple threads you are distributing work among? Or would you be fine with assigning runtime/deadline for each one of those?
that's a tricky question and depends on the use-case: for me it's "good enough" to use something like:
main audio thread:
while running:
snd_pcm_read // blocks until data is delivered from ALSA wake helper(s)
do work()
sync_with_helper(s)
snd_pcm_write // blocks until data is delivered to ALSA
alsa is basically driven by the hardware, which delivers/requests data every ~1.3ms.
distributing the work is a little tricky: to avoid excessive scheduling overhead, the worker threads are typically woken up (after snd_pcm_read) and the result is collected where sync_with_helper() typically boils down to busy waiting. in this case, the workers as rate-monotonic in a similar manner as the main audio thread.
one could also use lock-free queues with semaphores to wake only as many threads as needed for the graph topology (which can depend on user input). in this case SCHED_FIFO sounds more suitable.
--
from a practical point of view: i'm not targeting a safety critical system. one advantage i'm seeing of DEADLINE over FIFO/RR is that it's easier to prevent lockups (e.g. when a user overloads the system). in the linux audio world this is typically done by a watchdog thread. the other part of the unix world (mach) is using time-constraint threads by default for audio use cases. so i'd assume that DEADLINE would free me from the need to spawn the watchdog thread ...
cheers,
tim
Quoted 16 lines · Tim Blechmann
Quoted 4 lines
In your case - assuming you could already use DEADLINE for !root - do you think you would need to use the same interface for the multiple threads you are distributing work among? Or would you be fine with assigning runtime/deadline for each one of those?
that's a tricky question and depends on the use-case: for me it's "good enough" to use something like:
main audio thread:
while running:
snd_pcm_read// blocks until data is delivered from ALSA wake helper(s)
dowork()
sync_with_helper(s)
snd_pcm_write// blocks until data is delivered to ALSA
Quoted 2 lines
alsa is basically driven by the hardware, which delivers/requests data every ~1.3ms.
Hi Tim!
Quoted 6 lines
distributing the work is a little tricky: to avoid excessive scheduling overhead, the worker threads are typically woken up (after
snd_pcm_read) and the result is collected wheresync_with_helper()typically boils down to busy waiting. in this case, the workers as rate-monotonic in a similar manner as the main audio thread.
When you say "rate-monitonic", do you mean that the worker threads with smaller period (high rate, so) have a higher fixed-priority? If so, what are the other periods?
as sync_with_helper() does a busy-wait, the "main audio thread" has a relative lower priority than the worker threads, right?
Quoted 5 lines
one could also use lock-free queues with semaphores to wake only as many threads as needed for the graph topology (which can depend on user input). in this case
SCHED_FIFOsounds more suitable.
--
Quoted 7 lines
from a practical point of view: i'm not targeting a safety critical system. one advantage i'm seeing of DEADLINE over FIFO/RR is that it's easier to prevent lockups (e.g. when a user overloads the system). in the linux audio world this is typically done by a watchdog thread. the other part of the unix world (mach) is using time-constraint threads by default for audio use cases. so i'd assume that DEADLINE would free me from the need to spawn the watchdog thread ...
So, your main wish here is to avoid having a watchdog thread to "throttle" misbehaving workload/setup?
For curiosity, what are the "time-constraints" setup used on mach?
-- Daniel
Quoted 17 lines
Quoted 8 lines
main audio thread:
while running:
snd_pcm_read// blocks until data is delivered from ALSA wake helper(s)
dowork()
sync_with_helper(s)
snd_pcm_write// blocks until data is delivered to ALSAQuoted 8 lines
alsa is basically driven by the hardware, which delivers/requests data every ~1.3ms.
distributing the work is a little tricky: to avoid excessive scheduling overhead, the worker threads are typically woken up (after
snd_pcm_read) and the result is collected wheresync_with_helper()typically boils down to busy waiting. in this case, the workers as rate-monotonic in a similar manner as the main audio thread.
Quoted 3 lines
When you say "rate-monitonic", do you mean that the worker threads with smaller period (high rate, so) have a higher fixed-priority? If so, what are the other periods?
the period will be the same period: i wake up the workers once per period of the main thread, distribute the work and collect it at the end. so "period" is basically the same, "runtime" a little lower.
Quoted 2 lines
as
sync_with_helper()does a busy-wait, the "main audio thread" has a relative lower priority than the worker threads, right?
typically all threads have the same priority and there will be at most N threads for N physical CPU cores. so each thread will have a CPU to run on and no thread will be preempted.
Quoted 2 lines
So, your main wish here is to avoid having a watchdog thread to "throttle" misbehaving workload/setup?
mainly ... and nerdy research if SCHED_DEADLINE is useful for audio applications ... ;)
Quoted 1 line
For curiosity, what are the "time-constraints" setup used on mach?
it's pretty similar to the SCHED_DEADLINE constraints:
from:
https://opensource.apple.com/source/xnu/xnu-124.7/osfmk/mach/thread_policy.h.auto.html
Quoted 18 lines
/*
*THREAD_TIME_CONSTRAINT_POLICY:
*
* This scheduling mode is for threads which have real time
* constraints on their execution.
*
* Parameters:
*
* period: This is the nominal amount of time between separate
* processing arrivals, specified in absolute time units. A
* value of 0 indicates that there is no inherent periodicity in
* the computation.
*
* computation: This is the nominal amount of computation
* time needed during a separate processing arrival, specified
* in absolute time units.
*
* constraint: This is the maximum amount of real time that
Quoted 8 lines
* may elapse from the start of a separate processing arrival
* to the end of computation for logically correct functioning,
* specified in absolute time units. Must be (>= computation).
* Note that latency = (constraint - computation).
*
* preemptible: This indicates that the computation may be
* interrupted, subject to the constraint specified above. */
cheers,
tim
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
On 21/08/18 00:45, Tim Blechmann wrote:
[...]
Quoted 4 lines
apart from that, i'm only missing two features:
* using
SCHED_DEADLINEfrom user applications is kind of a showstopper for me
Right. Would you be up for continuing experimenting with DEADLINE (while we get our heads around removing this constraint) by - ugly hack warning
- removing the check at
https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L4216
?
I guess it would be good if you could provide even more feedback than you already did, so that we can plan ahead what needs fixing/changing or implementing.
BTW, thanks a lot already for this discussion! :-)
Quoted 5 lines
* a nicer glibc wrapper, that allows me to use pthreads rather than using
syscall: it's all pretty easy to obtain thetidfor a calling thread (gettid), but it seems to be impossible to obtain thetidfrompthread_twithout some hacks which are way too dirty to share in
public ...
Indeed. We have been basically relying on
https://github.com/scheduler-tools/rt-app/tree/master/libdl
so far. But, I guess it's time - it has been time for a while - that we reach glibc guys and try to understand why this is not yet all in the standard libraries.
Quoted 29 lines · Juri Lelli
On 21/08/18 00:45, Tim Blechmann wrote:
[...]
Quoted 4 lines
apart from that, i'm only missing two features:
* using
SCHED_DEADLINEfrom user applications is kind of a showstopper for meRight. Would you be up for continuing experimenting with DEADLINE (while we get our heads around removing this constraint) by - ugly hack warning
- removing the check athttps://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L4216
?
I guess it would be good if you could provide even more feedback than you already did, so that we can plan ahead what needs fixing/changing or implementing.
BTW, thanks a lot already for this discussion! :-)
Quoted 5 lines
* a nicer glibc wrapper, that allows me to use pthreads rather than using
syscall: it's all pretty easy to obtain thetidfor a calling thread (gettid), but it seems to be impossible to obtain thetidfrompthread_twithout some hacks which are way too dirty to share in
public ...
Quoted 7 lines
Indeed. We have been basically relying on
https://github.com/scheduler-tools/rt-app/tree/master/libdl
so far. But, I guess it's time - it has been time for a while - that we reach glibc guys and try to understand why this is not yet all in the standard libraries.
FYI, I added both points to the list of known issues. Feel free to add details.
https://github.com/jlelli/linux/issues
Best,
- Juri
Hi all,
+cc-ing Alessio, who has been working on this just recently on Android (will tell more answering on a different msg).
Quoted 1 line · Tim Blechmann
the mach API also has this notion of "period == 0", but apart from that it's quite similar.
in terms of API, XNU time constrained threads look like DEADLINE, however in terms of kernel behavior/implementation behind that, it's something completely different: last time I had a glance at this, it was .... ehm .... probably just during that old Linux Audio Conference (LAC) back in 2011, where the take away was: 1) in the kernel there's nothing like the CBS or deadline-based scheduling, but just a heuristic boosting a bit more those tasks, accompanied by usual heuristics to de-boost in case the task becomes CPU-intensive (similar to CFS heuristics for interactive-vs-batch); 2) the only user of that API seemed to be some sound daemon on OS-X, that had an almost random value set as "runtime", accompanied by a comment "this just works" ;-P
But, it was long ago, if anyone has a pointer to what XNU does with time constrained threads in these days....
Quoted 4 lines
apart from that, i'm only missing two features:
* using
SCHED_DEADLINEfrom user applications is kind of a showstopper for me
unprivileged tasks using DEADLINE remained a request for quite a long time -- that old single-processor implementation (AQuoSA) we used long ago with JACK had such a feature, that was implementing this model:
http://retis.sssup.it/~tommaso/papers/rtas08.php
the criticality for DEADLINE is that those tasks preempt RT99 ones, so even with controlled runtime/deadline boundaries....
Quoted 5 lines
* a nicer glibc wrapper, that allows me to use pthreads rather than using
syscall: it's all pretty easy to obtain thetidfor a calling thread (gettid), but it seems to be impossible to obtain thetidfrompthread_twithout some hacks which are way too dirty to share in
public ...
well-known "pain" as well, thanks.
T.
Signature · 4 lines
Tommaso Cucinotta, Computer Engineering PhD Associate Professor at the Real-Time Systems Laboratory (ReTiS) Scuola Superiore Sant'Anna, Pisa, Italy http://retis.sssup.it/people/tommaso
[copying Alessio]
Hi,
Quoted 11 lines · Tim Blechmann
that's a tricky question and depends on the use-case: for me it's "good enough" to use something like:
main audio thread:
while running:
snd_pcm_read// blocks until data is delivered from ALSA wake helper(s)
dowork()
sync_with_helper(s)
snd_pcm_write// blocks until data is delivered to ALSA
Quoted 2 lines
alsa is basically driven by the hardware, which delivers/requests data every ~1.3ms.
this is exactly the use-case we considered in that old paper back in LAC 2011, and let me just add that there is more experimentation and details and results in Giacomo's MSc thesis:
https://retis.sssup.it/?q=node/77
FYI, we restarted activities on this use-case just recently, considering a couple of further dimensions:
1) support for heterogeneous platforms with the aim of achieving a sound support for Arm big.LITTLE on Android, and apply it to the Android video & audio processing pipelines;
2) proper consideration of power switching capabilities of the platform (assuming the user would block the CPU frequency to its maximum is not really multimedia-friendly);
3) support for multi-threaded processing workflows on multi-cores, as typically needed by (high-performance) audio applications, either with a single multi-threaded audio processing client, or, e.g., with JACK, with a DAG of computations that can take advantage of underlying multi-core processing.
In this context, we've been playing also with the hierarchical extension to SCHED_DEADLINE we sent last year on LKML
https://lkml.org/lkml/2017/3/31/658
I hope we can share some good write-up & experimental results pretty soon about some of the above...
Quoted 9 lines
distributing the work is a little tricky: to avoid excessive scheduling overhead, the worker threads are typically woken up (after
snd_pcm_read) and the result is collected wheresync_with_helper()typically boils down to busy waiting. in this case, the workers as rate-monotonic in a similar manner as the main audio thread.one could also use lock-free queues with semaphores to wake only as many threads as needed for the graph topology (which can depend on user input). in this case
SCHED_FIFOsounds more suitable.
proper handling of task dependencies & synchronization, and associated priority inversion issues, is also an aspect we've been and we're still working on, as when you mix DEADLINE tasks in the picture, it's a big mess.
Quoted 7 lines
from a practical point of view: i'm not targeting a safety critical system. one advantage i'm seeing of DEADLINE over FIFO/RR is that it's easier to prevent lockups (e.g. when a user overloads the system). in the linux audio world this is typically done by a watchdog thread. the other part of the unix world (mach) is using time-constraint threads by default for audio use cases. so i'd assume that DEADLINE would free me from the need to spawn the watchdog thread ...
I'm not sure I'm getting this: AFAIU, from this viewpoint, what you could get with DEADLINE, is very similar to what you can get already with RT throttling on RT (RR/FIFO), setting up properly CPU cgroups & limits. Why do you think DEADLINE would allow you to avoid a watchdog thread, that you need with RT instead ?
Thanks,
T.
Signature · 4 lines
Tommaso Cucinotta, Computer Engineering PhD Associate Professor at the Real-Time Systems Laboratory (ReTiS) Scuola Superiore Sant'Anna, Pisa, Italy http://retis.sssup.it/people/tommaso
Quoted 26 lines
Quoted 2 lines
alsa is basically driven by the hardware, which delivers/requests data every ~1.3ms.
this is exactly the use-case we considered in that old paper back in LAC 2011, and let me just add that there is more experimentation and details and results in Giacomo's MSc thesis:
https://retis.sssup.it/?q=node/77
FYI, we restarted activities on this use-case just recently, considering a couple of further dimensions:
1) support for heterogeneous platforms with the aim of achieving a sound support for Arm big.LITTLE on Android, and apply it to the Android video & audio processing pipelines;
2) proper consideration of power switching capabilities of the platform (assuming the user would block the CPU frequency to its maximum is not really multimedia-friendly);
3) support for multi-threaded processing workflows on multi-cores, as typically needed by (high-performance) audio applications, either with a single multi-threaded audio processing client, or, e.g., with JACK, with a DAG of computations that can take advantage of underlying multi-core processing.
Quoted 7 lines
In this context, we've been playing also with the hierarchical extension to
SCHED_DEADLINEwe sent last year on LKMLhttps://lkml.org/lkml/2017/3/31/658
I hope we can share some good write-up & experimental results pretty soon about some of the above...
very interesting! thanks for the links ...
Quoted 7 lines
I'm not sure I'm getting this: AFAIU, from this viewpoint, what you could get
with DEADLINE, is very similar to what you can get already with RT throttling
on RT (RR/FIFO), setting up properly CPU cgroups & limits. Why do you think DEADLINE would allow you to avoid a watchdog thread, that you need with RT instead ?
interesting, wasn't aware of rt throttling ... (haven't done much linux audio developments for almost 10 years). will give it another read ...
cheers,
tim
Hi Juri,
sorry for the late reply (I am just back from vacations), and thanks for cc-ing me.
I see you opened a bug on github about this, so I am going to add more details there, but basically I think that in order to use SCHED_DEADLINE as non-root we need to:
1) Disable the boosting mechanism (not the inheritance, just the "soft enforcement behaviour" of tasks holding mutexes)
2) Implement some mechanism to limit the amount of dl bandwidth a user can allocate to itself (I think the cgroup-based approach we discussed some time ago should be OK... If I remember well, you even had a patch implementing it :)
Luca
Quoted 48 lines · On Mon, 20 Aug 2018 10:54:20 +0200 Juri Lelli <[email protected]> wrote:
Quoted 25 lines · Tim Blechmann
Quoted 10 lines
Quoted 3 lines
audio applications typically use
SCHED_FIFO. however they are rate-monotonic by design, so i wanted to evaluate howSCHED_DEADLINEbehaves here.Right. Exactly what I started to think about lately as well.
Quoted 4 lines
Quoted 3 lines
Asking because we might want to have a look at how to relax the current privileges requirement.
Quoted 7 lines
Quoted 1 line
please :)
Anything to try attracting new users. :-P
More seriously, could you please describe in a bit more details your specific use case? Based on JACK I guess? Is it the application you are interested into public available somewhere?
it could be based on JACK or natively on ALSA. furthermore the fun part starts when trying to distribute work to multiple threads:
during the time slice of one audio callback (could be as low as 1.3ms), we need to wake helper threads and (busy?)wait to collect all the results. (while one of the applications is open source, it's not entirely straight-forward to use)Quoted 6 lines
for me it's more an investigation if
SCHED_DEADLINEis actually feasible in real-world audio application these days. tommaso used to have a research prototype [1], built directly into JACK. however currently i'd be more interested in using it in ALSA. but yeah, it's more an investigation asSCHED_FIFOdoes a decent job for my use case ...I (almost) see. Thanks for the details.
Quoted 1 line
[1] http://retis.sssup.it/~tommaso/publications/LAC-2011.pdf
So, this was IIRC using AQuoSA hierarchical RT on EDF scheduler implementation; which it was handy to easy the task of assigning CBS server parameters to JACK clients (graph).
In your case - assuming you could already use DEADLINE for !root - do you think you would need to use the same interface for the multiple threads you are distributing work among? Or would you be fine with assigning runtime/deadline for each one of those?
BTW, cc-ing Tommaso and Luca. Discussion started from
https://www.spinics.net/lists/linux-rt-users/msg19296.html
Hi,
Quoted 4 lines · luca abeni
Hi Juri,
sorry for the late reply (I am just back from vacations), and thanks for cc-ing me.
Welcome back! :-)
Quoted 2 lines
I see you opened a bug on github about this, so I am going to add more details there,
I use github issues to keep track of things, but I guess mailing list discussions are to be preferred though (so that more people can potentially follow).
Quoted 4 lines
but basically I think that in order to use
SCHED_DEADLINEas non-root we need to:
1) Disable the boosting mechanism (not the inheritance, just the "soft enforcement behaviour" of tasks holding mutexes)
But then what is a sane inheritance mechanism? Walk the chain and find the next potential deadline to inherit for the current boosted (still runtime enforced) task before throttling it? Not sure it's going to be any easier than the proxy solution. :-/
Quoted 4 lines
2) Implement some mechanism to limit the amount of dl bandwidth a user can allocate to itself (I think the cgroup-based approach we discussed some time ago should be OK... If I remember well, you even had a patch implementing it :)
I think most (all?) distributions today run with CONFIG_RT_GROUP_SCHED disabled, we should also think about a solution that doesn't rely on that interface. Maybe the existing system wide sched_rt_{period, runtime}_us are enough?
Best,
- Juri
Hi Juri,
Quoted 4 lines · On Thu, 23 Aug 2018 12:23:50 +0200 Juri Lelli <[email protected]> wrote:
Quoted 2 lines
sorry for the late reply (I am just back from vacations), and thanks for cc-ing me.
Welcome back! :-)
Thanks :)
Quoted 6 lines
Quoted 2 lines
I see you opened a bug on github about this, so I am going to add more details there,
I use github issues to keep track of things, but I guess mailing list discussions are to be preferred though (so that more people can potentially follow).
Ok
Quoted 6 lines
Quoted 4 lines
but basically I think that in order to use
SCHED_DEADLINEas non-root we need to:
1) Disable the boosting mechanism (not the inheritance, just the "soft enforcement behaviour" of tasks holding mutexes)But then what is a sane inheritance mechanism?
In my understanding (please correct me if I am wrong), this is an orthogonal issue: if I understand well, the issue preventing non-root usage of SCHED_DEADLINE is that a task inheriting a dl entity is not throttled (when the current runtime rrives to 0, the deadline is postponed, but the task stays schedulable). So, I think that removing this behaviour should allow to use SCHED_DEADLINE without starving other tasks...
Then, there is the issue about the deadline and runtime to inherit. And I agree that this is important (and the solution is not easy), but you have this issue even if you use the current "dl_boosted" behaviour... No?
Quoted 4 lines
Walk the chain and find
the next potential deadline to inherit for the current boosted (still runtime enforced) task before throttling it? Not sure it's going to be any easier than the proxy solution. :-/
Right; this is not easy... But I think it is not related with the issue we are discussing (if I understand this email thread well). Yes, it has to be fixed, but it does not prevent non-root usage. Or am I missing something?
Quoted 8 lines
Quoted 4 lines
2) Implement some mechanism to limit the amount of dl bandwidth a user can allocate to itself (I think the cgroup-based approach we discussed some time ago should be OK... If I remember well, you
even had a patch implementing it :)I think most (all?) distributions today run with
CONFIG_RT_GROUP_SCHEDdisabled, we should also think about a solution that doesn't rely on that interface.
I guess CONFIG_RT_GROUP_SCHED is often disabled because it ends up changing the "traditional" SCHED_{RR/FIFO} behaviour. So, maybe the solution is to have a different dl_{runtime,period} interface in control groups (enabled by CONFIG_DL_GROUP_SCHED :). CONFIG_DL_GROUP_SCHED does not change the scheduling behaviour, but only the admission test... So, enabling it could be safer than enabling CONFIG_RT_GROUP_SCHED.
Quoted 2 lines
Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?
I do not know... A cgroup-based interface looks more powerful (and not so difficult to implement... :), and would allow the sysadmin to decide which users can use SCHED_DEADLINE, how much SCHED_DEADLINE bandwidth can each user/group use, etc...
Luca
Quoted 2 lines · luca abeni
On Thu, 23 Aug 2018 12:23:50 +0200
Juri Lelli <[email protected]> wrote:
[...]
Quoted 9 lines
Quoted 1 line
But then what is a sane inheritance mechanism?
In my understanding (please correct me if I am wrong), this is an orthogonal issue: if I understand well, the issue preventing non-root usage of
SCHED_DEADLINEis that a task inheriting a dl entity is not throttled (when the current runtime rrives to 0, the deadline is postponed, but the task stays schedulable). So, I think that removing this behaviour should allow to useSCHED_DEADLINEwithout starving other tasks...
Right, potential starvation would be gone...
Quoted 4 lines
Then, there is the issue about the deadline and runtime to inherit. And I agree that this is important (and the solution is not easy), but you have this issue even if you use the current "
dl_boosted" behaviour... No?
... but, while it's true that current inheritance has problems w/ o w/o boosting behaviour, I wonder if the problem might be more painful for a normal user that it's still under runtime enforcement. Disabling enforcement seems to hide a bit the fact that we need proper inheritance. :-(
Quoted 9 lines
Quoted 4 lines
Walk the chain and find
the next potential deadline to inherit for the current boosted (still runtime enforced) task before throttling it? Not sure it's going to be any easier than the proxy solution. :-/Right; this is not easy... But I think it is not related with the issue we are discussing (if I understand this email thread well). Yes, it has to be fixed, but it does not prevent non-root usage. Or am I missing something?
It depends on how bad we think it's what I said above I guess.
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
2) Implement some mechanism to limit the amount of dl bandwidth a user can allocate to itself (I think the cgroup-based approach we discussed some time ago should be OK... If I remember well, you
even had a patch implementing it :)I think most (all?) distributions today run with
CONFIG_RT_GROUP_SCHEDdisabled, we should also think about a solution that doesn't rely on that interface.
Quoted 7 lines
I guess
CONFIG_RT_GROUP_SCHEDis often disabled because it ends up changing the "traditional" SCHED_{RR/FIFO} behaviour. So, maybe the solution is to have a different dl_{runtime,period} interface in control groups (enabled byCONFIG_DL_GROUP_SCHED:).CONFIG_DL_GROUP_SCHEDdoes not change the scheduling behaviour, but only the admission test... So, enabling it could be safer than enablingCONFIG_RT_GROUP_SCHED.
Not sure if adding yet another config switch is acceptable. FWIW, I'd prefer not to add it, also looking back at all the problems RT_GROUP_ SCHED seems to pose/have posed, but if turns to be the only option..
Quoted 7 lines
Quoted 2 lines
Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?I do not know... A cgroup-based interface looks more powerful (and not so difficult to implement... :), and would allow the sysadmin to decide which users can use
SCHED_DEADLINE, how muchSCHED_DEADLINEbandwidth can each user/group use, etc...
How about extending PAM limits instead? It looks like it's what (e.g.) audio users rely on already [1]. It is maybe possible to add dlruntime, dlperiod, dldeadline parameters in there?
1 - https://fedoraproject.org/wiki/JACK_Audio_Connection_Kit#Running_Jack_in_Realtime_mode
Quoted 18 lines · On Thu, 23 Aug 2018 14:37:39 +0200 Juri Lelli <[email protected]> wrote:
Quoted 2 lines · luca abeni
On Thu, 23 Aug 2018 12:23:50 +0200
Juri Lelli <[email protected]> wrote:[...]
Quoted 9 lines
Quoted 1 line
But then what is a sane inheritance mechanism?
In my understanding (please correct me if I am wrong), this is an orthogonal issue: if I understand well, the issue preventing non-root usage of
SCHED_DEADLINEis that a task inheriting a dl entity is not throttled (when the current runtime rrives to 0, the deadline is postponed, but the task stays schedulable). So, I think that removing this behaviour should allow to useSCHED_DEADLINEwithout starving other tasks...Right, potential starvation would be gone...
Quoted 8 lines
Quoted 4 lines
Then, there is the issue about the deadline and runtime to inherit. And I agree that this is important (and the solution is not easy), but you have this issue even if you use the current "
dl_boosted" behaviour... No?... but, while it's true that current inheritance has problems w/ o w/o boosting behaviour, I wonder if the problem might be more painful for a normal user that it's still under runtime enforcement.
I think that the only way to have an answer to this doubt is to try :)
Quoted 3 lines
Disabling
enforcement seems to hide a bit the fact that we need proper inheritance. :-(
My impression is that it hides the problem in some situations, but the Murphy law ensures that the issue will appear when the quality of the audio is more important :)
Luca
On Thu, 23 Aug 2018 14:37:39 +0200
Juri Lelli <[email protected]> wrote:
[...]
Quoted 11 lines
Quoted 7 lines
Quoted 2 lines
Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?I do not know... A cgroup-based interface looks more powerful (and not so difficult to implement... :), and would allow the sysadmin to decide which users can use
SCHED_DEADLINE, how muchSCHED_DEADLINEbandwidth can each user/group use, etc...How about extending PAM limits instead? It looks like it's what (e.g.) audio users rely on already [1]. It is maybe possible to add dlruntime, dlperiod, dldeadline parameters in there?
Ok, I do not know much about these PAM limits... Where can I find more information? What do we need to implement exactly, to add these dlruntime, dlperiod, dldeadline parameters?
Thanks,
Luca
Quoted 11 lines · luca abeni
On Thu, 23 Aug 2018 14:37:39 +0200
Juri Lelli <[email protected]> wrote:
[...]Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?I do not know... A cgroup-based interface looks more powerful (and not so difficult to implement... :), and would allow the sysadmin to decide which users can use
SCHED_DEADLINE, how muchSCHED_DEADLINEbandwidth can each user/group use, etc...
Quoted 6 lines
Quoted 3 lines
How about extending PAM limits instead? It looks like it's what (e.g.) audio users rely on already [1]. It is maybe possible to add dlruntime, dlperiod, dldeadline parameters in there?
Ok, I do not know much about these PAM limits... Where can I find more information?
The link I shared already gives some information, man pam_limits and limits.conf also.
Quoted 2 lines
What do we need to implement exactly, to add these dlruntime, dlperiod, dldeadline parameters?
Not sure yet "exactly" :-), but I'd say something along these lines
https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L4198
Quoted 22 lines · On Thu, 23 Aug 2018 15:09:57 +0200 Juri Lelli <[email protected]> wrote:
Quoted 11 lines · luca abeni
On Thu, 23 Aug 2018 14:37:39 +0200
Juri Lelli <[email protected]> wrote:
[...]Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?I do not know... A cgroup-based interface looks more powerful (and not so difficult to implement... :), and would allow the sysadmin to decide which users can use
SCHED_DEADLINE, how muchSCHED_DEADLINEbandwidth can each user/group use, etc...Quoted 6 lines
Quoted 3 lines
How about extending PAM limits instead? It looks like it's what (e.g.) audio users rely on already [1]. It is maybe possible to add dlruntime, dlperiod, dldeadline parameters in there?
Ok, I do not know much about these PAM limits... Where can I find more information?
The link I shared already gives some information, man
pam_limitsand limits.conf also.
Quoted 6 lines
Quoted 2 lines
What do we need to implement exactly, to add these dlruntime, dlperiod, dldeadline parameters?
Not sure yet "exactly" :-), but I'd say something along these lines
https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L4198
Ok, thanks; I think I got it: we need to extend the "rlim" array (in signal_struct) adding fields for "RLIMIT_DLRUNTIME" and friends (actually, I think a "RLIMIT_UTILIZATION" field is also needed). And then we need to add checks for these fields in __sched_setscheduler(), etc...
Maybe I am misunderstanding this stuff, but the cgroup-based approach looks simpler / more elegant :)
Thanks,
Luca
Quoted 16 lines · On Thursday 23 August 2018 09:00:40 luca abeni wrote:
On Thu, 23 Aug 2018 14:37:39 +0200
Juri Lelli <[email protected]> wrote:
[...]Quoted 11 lines
Quoted 7 lines
Quoted 2 lines
Maybe the existing system wide sched_rt_{period,
runtime}_us are enough?I do not know... A cgroup-based interface looks more powerful (and not so difficult to implement... :), and would allow the sysadmin to decide which users can use
SCHED_DEADLINE, how muchSCHED_DEADLINEbandwidth can each user/group use, etc...How about extending PAM limits instead? It looks like it's what (e.g.) audio users rely on already [1]. It is maybe possible to add dlruntime, dlperiod, dldeadline parameters in there?
Quoted 7 lines
Ok, I do not know much about these PAM limits... Where can I find more information? What do we need to implement exactly, to add these dlruntime, dlperiod, dldeadline parameters?
Thanks,
Luca
Are you folks interested in input from a real world user? On who at the moment is building about half these kernels as you announce them on a pi-3b, the same pi-3b running 1500 lbs of a Sheldon 11"x36" lathe.
I am running linuxcnc on both x86 and armhf machinery, two lathes and two milling machines, all being moved by stepper motors.
Steppers, if the steps are being software generated, are very sensitive to timing wibbles in a stream of steps, with a 5 microsecond wobble in the timing haveing serious effects on the available torque by causing a speedup and slowdown. This severely limits the speed at which the motor can be driven without stalling or losing steps, which in a stepper system is the equivalent of losing its home reference which=wrecked part. I have one machine still depending on software stepping. I could easily triple the speed of that machine by spending another 100 dollars to offload that to an fpga based interface card. As it is the base thread runs at about 28 microseconds. That also represents the granularity of a speed change. So it of late has been relegated to doing very precise work via EDM, which is slow anyway.
In short, linuxcnc, and its best rtai kernels can do pergaps 10"/minute whereas the fpga cards can move that same machine at 100 ipm. Any help in getting the latency down means we can do things faster.
LinuxCNC is also run as a common user, never as root. Thats my $0.02 from a users viewpoint.
Signature · 6 lines
Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Genes Web page <http://geneslinuxbox.net:6309/gene>
Quoted 2 lines · Gene Heskett
LinuxCNC is also run as a common user, never as root. Thats my $0.02 from a users viewpoint.
Given that these are special-purpose machines, can I ask why you wouldn't run as root if it gave better performance?
Chris
Quoted 8 lines · On Thursday 23 August 2018 12:01:41 Chris Friesen wrote:
Quoted 2 lines · Gene Heskett
LinuxCNC is also run as a common user, never as root. Thats my $0.02 from a users viewpoint.
Given that these are special-purpose machines, can I ask why you wouldn't run as root if it gave better performance?
Chris
Its a rather bad habit, and if I get used to doing it on this machine, I might do it on one of the metal carvers, which has a tendency to mess with the user perms. Root or known user is all the same to lcnc AFAIK. I have done it 3 or 4 times, but there is no detectable diff in performance. But sometime in the last few years it has now been trained to refuse to run as root.
The kernel argument isolcpus= comes in handy occasionally on really old but still multicore hdwe but has had little or no effect since about kernel 3.4.0 built for SMP and PREEMPT. We don't use more than 2GB of d-ram for the same reason, the patches to give a 4GB capable address are also a performance penalty.
Currently running
Linux GO704 3.4-9-rtai-686-pae #1 SMP PREEMPT Debian 3.4.55-4linuxcnc i686 GNU/Linux, a 32 bit build because its faster at a context switch than the larger stack image the 64 bit build uses.
This machine, my "doitall" machine doesn't drive any of that sort of hardware, and is currently running:3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.39-1+deb8u1~bpo70+1 (2017-02-24) x86_64 GNU/Linux which will run the simulator so I can write cgode from a comfy chair. Then export it with sshfs to the machine that will actually make the swarf, aka metallic sawdust.
Thanks Chris.
Signature · 6 lines
Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Genes Web page <http://geneslinuxbox.net:6309/gene>