Realtime LSM

The realtime LSM has been previously explained on this list. Its function is to allow selected nonroot users to run RT tasks. The most common application is low latency audio with JACK, http://jackit.sf.net.

Several people have reported that 2.6.10 is the best kernel yet for audio latency, see
http://ccrma-mail.stanford.edu/pipermail/planetccrma/2004-December/007341.html. If the realtime LSM were merged, then this would be the last step to making low latency audio work well with the stock kernel.

We (the authors and the Linux audio community) would like to request its inclusion in the next -mm release, with the eventual goal of having it in mainline.

This is identical to the last version Jack O'Quin posted (but didn't cc:
Andrew, or make clear that we would like this added to -mm), so I preserved his Signed-Off-By.

Quoted 12 lines · Lee Revell

The realtime LSM has been previously explained on this list. Its function is to allow selected nonroot users to run RT tasks. The most common application is low latency audio with JACK, http://jackit.sf.net.

Several people have reported that 2.6.10 is the best kernel yet for audio latency, see
http://ccrma-mail.stanford.edu/pipermail/planetccrma/2004-December/007341.html. If the realtime LSM were merged, then this would be the last step to making low latency audio work well with the stock kernel.

We (the authors and the Linux audio community) would like to request its inclusion in the next -mm release, with the eventual goal of having it in mainline.

Quoted 3 lines

This is identical to the last version Jack O'Quin posted (but didn't cc:
Andrew, or make clear that we would like this added to -mm), so I preserved his Signed-Off-By.

This is far too specialized. And option to the capability LSM to grant capabilities to certain uids/gids sounds like the better choise - and would also allow to get rid of the magic hugetlb uid horrors.

Quoted 6 lines · Christoph Hellwig

Quoted 4 lines · Lee Revell

The realtime LSM has been previously explained on this list. Its function is to allow selected nonroot users to run RT tasks. The most common application is low latency audio with JACK, http://jackit.sf.net.

Quoted 3 lines

This is far too specialized. And option to the capability LSM to grant capabilities to certain uids/gids sounds like the better choise - and would also allow to get rid of the magic hugetlb uid horrors.

those can go away anyway now that there is an rlimit to achieve the exact same thing.....

I can see the point of making an rlimit like thing instead for both the nice levels allowed and maybe the "can do rt" bit

Quoted 12 lines · Lee Revell

The realtime LSM has been previously explained on this list. Its function is to allow selected nonroot users to run RT tasks. The most common application is low latency audio with JACK, http://jackit.sf.net.

Several people have reported that 2.6.10 is the best kernel yet for audio latency, see
http://ccrma-mail.stanford.edu/pipermail/planetccrma/2004-December/007341.html. If the realtime LSM were merged, then this would be the last step to making low latency audio work well with the stock kernel.

We (the authors and the Linux audio community) would like to request its inclusion in the next -mm release, with the eventual goal of having it in mainline.

Quoted 3 lines

This is identical to the last version Jack O'Quin posted (but didn't cc:
Andrew, or make clear that we would like this added to -mm), so I preserved his Signed-Off-By.

This is far too specialized. And option to the capability LSM to grant capabilities to certain uids/gids sounds like the better choise - and would also allow to get rid of the magic hugetlb uid horrors.

Got a patch? Code talks, BS walks. This is working perfectly, right now, and is being used by thousands of Linux ausio users.

Which still doesn't mean it's the right design. And no, I don't need the feature so I won't write it. If you want a certain feature it's up to you to implement it in a way that's considered mergeable.

But, the lack of this feature has been a continual impediment for years now. It affects not just me, but most other serious Linux audio developers and many of our users. We need a simple way for users to configure a Digital Audio Workstation without having to run large, complex, insecure audio applications as `root'. Our competition runs on Windows and Mac systems where no such configuration is needed.

We could do it the was OSX (our real competition) does if that would make people happy. They just let any user run RT tasks. Oh wait, but that's a "broken design", everyone knows that OSX is a joke, no one would use *that* OS to mix a CD or score a movie. :-)

The problem with uid/gid based hacks is that they get really ugly to administer really fast. Especially once you have users who need realtime and hugetlb, and users who need one only.

It would be far cleaner to split CAP_SYS_NICE capability down - which should cover the real time OS functions nicely. Right now it gives a few too many rights but that could be fixed easily.

The problem with uid/gid based hacks is that they get really ugly to administer really fast. Especially once you have users who need realtime and hugetlb, and users who need one only.

Quoted 4 lines · Lee Revell

We could do it the was OSX (our real competition) does if that would make people happy. They just let any user run RT tasks. Oh wait, but that's a "broken design", everyone knows that OSX is a joke, no one would use *that* OS to mix a CD or score a movie. :-)

You can do that already, just make everyone root

The problem with uid/gid based hacks is that they get really ugly to administer really fast. Especially once you have users who need realtime and hugetlb, and users who need one only.

It would be far cleaner to split CAP_SYS_NICE capability down - which should cover the real time OS functions nicely. Right now it gives a few too many rights but that could be fixed easily.

Quoted 4 lines · * Alan Cox ([email protected]) wrote:

The problem with uid/gid based hacks is that they get really ugly to administer really fast. Especially once you have users who need realtime and hugetlb, and users who need one only.

I don't believe the hugetlb gid stuff is useful anymore. It should be handled nicely via rlimits.

The last time I checked users could belong to more than one group. Am I missing something?

No, you're not. I think Alan's just saying the gid based checks are suboptimal if there's a cleaner way to do it (to which I agree). Personally, I don't have a big problem with the Realtime LSM. I've helped you with it, and suggested a few times that I'd prefer it to be generic; but never stepped up to deliver code of that sort. Since it's your itch, you've scratched it, and it's quite simple and contained, I consider it acceptable.

Here's a relatively simple idea: Why not make the "Realtime LSM" just check for a certain "Realtime" credential in the new credential store (Patch is in 2.6.10, see [1] for control program). You would mark it as a system credential and give access to that credential via the appropriate capability with a small utility program.

Quoted 2 lines · * Lee Revell ([email protected]) wrote:

The last time I checked users could belong to more than one group. Am I missing something?

No, you're not. I think Alan's just saying the gid based checks are suboptimal if there's a cleaner way to do it (to which I agree). Personally, I don't have a big problem with the Realtime LSM. I've helped you with it, and suggested a few times that I'd prefer it to be generic; but never stepped up to deliver code of that sort. Since it's your itch, you've scratched it, and it's quite simple and contained, I consider it acceptable.

the RT-LSM thing is a bit dangerous because it doesnt really protect against a runaway, buggy app. So i think the right way to approach this problem is to not apply RT-LSM for the time being, but to provide an 'advanced latency needs' scheduling class that is _still_ safe even if the task is runaway, but behaves with near-RT priorities if the task is 'nice' (i.e. doesnt use up large amount of CPU time.)

incidentally, there is such a scheduling class already: negative nice levels. Please skip any preconceptions you might have about nice levels, nice levels have been improved in 2.6.10, the timeslices are now given out exponentially, giving nice -20 tasks far more weight and priority than they used to have. (They are obviously still preemptable if they keep looping burning CPU - but that we can consider a feature.) (Also, in 2.6 the negative nice levels have a much more agressive interactivity setting, allowing them to preempt everything lower-prio.)

so, could you try vanilla 2.6.10 (without LSM and without jackd running with RT priorities), with jackd set to nice -20? Make sure the jack-client process gets this priority too. Best to achieve this is to renice a shell to -20 and start up everything from there - the nice settings will be inherited. How does such an audio test compare to a test done with jackd running at SCHED_FIFO with RT priority 1?

if this works out well then we could achieve something comparable to RT-LSM, via nice levels alone.

2) this is *not* only about scheduling. Realtime tasks need mlockall() and/or mlock as well. even the man page for mlock recognizes this, yet almost all the discussion here has focused on scheduling.

3) christoph claims that using uid/gid to define priviledge scope is a bad idea. but that is the *desired* method. uid/gid corresponds exactly to what the users of these systems want. they don't want priviledge accorded to specific applications - its the *users* not the applications that have the right to get RT scheduling, lock down memory and so on. these applications will run without RT priviledges, just not very well (in general, so badly that they are unusable for their intended purpose).

4) christoph's claims about OS X are nothing but ridiculous. whatever the internals of Darwin may or may not be (and they certainly include some of the best ideas about media-friendly kernels from the last 20 years, unlike our favorite OS), professional people are using OS X

5) setuid wrappers don't work for this, because even though you can change the scheduling class of another process, you cannot "grant" it the ability to use mlock. at least not without capabilities, so back to (1) above ...

I've spent probably burnt through to $250,000 supporting myself and my family over the last 5 years while I develop pro-level audio software for Linux. I don't expect to see any of that back. So when Christoph chimes in with the "I'm not paid, I don't have to tell you why I don't like it, I just don't" ... that really, really, really irritates me in a way that few other comments do.

We (Jack, Lee and now myself) have tried to explain what the problem with the kernel is, how LSM makes a solution possible, acknowledged issues and attempted to address them, and finally have offered up a working patch that makes life easier for a bunch of people who don't want to run webservers or compile kernels all day.

Quoted 4 lines

2) this is *not* only about scheduling. Realtime tasks need mlockall() and/or mlock as well. even the man page for mlock recognizes this, yet almost all the discussion here has focused on scheduling.

RLIMIT_MEMLOCK is your friend.

it doesn't really matter what you want, but how we can implement something that fits in the kernel design.

Quoted 4 lines

4) christoph's claims about OS X are nothing but ridiculous. whatever the internals of Darwin may or may not be (and they certainly include some of the best ideas about media-friendly kernels from the last 20 years, unlike our favorite OS), professional people are using OS X

professional people are also using Windows or Solaris. That doesn't mean we have to copy every bad idea from them.

And we have told you that this solution is not okay. You can spend

more time whining which won't do anything or you could help brainstorming how to implement a workable solution.

Quoted 6 lines · Paul Davis

Quoted 4 lines

2) this is *not* only about scheduling. Realtime tasks need mlockall() and/or mlock as well. even the man page for mlock recognizes this, yet almost all the discussion here has focused on scheduling.

Quoted 1 line

RLIMIT_MEMLOCK is your friend.

rlimit_memlock limits the *amount* of memory that mlock() can be used on, not whether mlock can be used. at least, thats my understanding of the POSIX design for this. the man page and the source code for mlock support make that reasonably clear.

Fine, we'll continue to tell people to use "realtime" LSM for audio work. The people this really affects probably won't use vanilla kernels anyway.

Quoted 4 lines

rlimit_memlock limits the *amount* of memory that mlock() can be used on, not whether mlock can be used. at least, thats my understanding of the POSIX design for this. the man page and the source code for mlock support make that reasonably clear.

eh no. It defaults to zero, but if you increase it for a specific user, that user is allowed to mlock more.

much). If you are unwilling to even discuss fixing the underlying design issues then I'm scared that this issue will never come to any workable solution.

Quoted 5 lines

Quoted 4 lines

rlimit_memlock limits the *amount* of memory that mlock() can be used on, not whether mlock can be used. at least, thats my understanding of the POSIX design for this. the man page and the source code for mlock support make that reasonably clear.

Quoted 2 lines

eh no. It defaults to zero, but if you increase it for a specific user, that user is allowed to mlock more.

from mm/mlock.c:do_mlock() in 2.6.8:

	if (on && !capable(CAP_IPC_LOCK))
		return -EPERM;

now try 2.6.9 ;)
this deficiency got already fixed

Quoted 2 lines

now try 2.6.9 ;)
this deficiency got already fixed

well thats good, i hope someone updated the man page too :)

but is there actually any way to grant specific users a reasonable rlimit,

Quoted 4 lines

well thats good, i hope someone updated the man page too :)

but is there actually any way to grant specific users a reasonable rlimit,

yes; most distributions will use pam for this, you can set per user or per gorup limits there.

Quoted 2 lines

yes; most distributions will use pam for this, you can set per user or per gorup limits there.

isn't that a uid/gid based system? ok, i'm being a little snide :)

It's quite probable that the current system of capabilities is not well suited for this, but I think that although it's tempting to work around it by introducing a new security module, in the long term it's much better to extend and/or fix the capabilities -- I don't see any fundamental reason for capabilities being unusable for this goal, it's much more likely to be just minor details in the implementation.

capabilities work - we use them in 2.4 where a helper suid application gets the ball rolling, and then its child grants capabilities to new clients.

fine, so the mlock situation may have improved enough post-2.6.9 that it can be considered fixed. that leaves the scheduler issue. but apparently, a uid/gid solution is OK for mlock, and not for the scheduler. am i missing something?

Quoted 5 lines · Paul Davis

fine, so the mlock situation may have improved enough post-2.6.9 that it can be considered fixed. that leaves the scheduler issue. but apparently, a uid/gid solution is OK for mlock, and not for the scheduler. am i missing something?

I think you skipped a step. You don't have a scheduler requirement, you have a latency requirement. You currently *solve* that latency requirement via a scheduler "hack", yet is quite clear that the "hard" realtime solution is most likely not the right approach. Note that I'm not saying that you shouldn't get the latency that that currently provides, but the downsides (can hang the machine) are bad; a solution that solves that would be far preferable
something like a soft realtime flag that acts as if it's the hard realtime one unless the app shows "misbehavior" (eg eats its timeslice for X times in a row) might for example be such a solution. And with the anti abuse protection it can run with far lighter privilegs.

At Fri, 7 Jan 2005 17:03:51 +0100,

Quoted 4 lines · Arjan van de Ven

something like a soft realtime flag that acts as if it's the hard realtime one unless the app shows "misbehavior" (eg eats its timeslice for X times in a row) might for example be such a solution. And with the anti abuse protection it can run with far lighter privilegs.

This reminds me about the soft-RT patch posted quite sometime ago. I feel such a handy psuedo-RT scheduler class would be useful for other systems than JACK, too...

Quoted 10 lines · Takashi Iwai

At Fri, 7 Jan 2005 17:03:51 +0100,

Quoted 4 lines · Arjan van de Ven

something like a soft realtime flag that acts as if it's the hard realtime one unless the app shows "misbehavior" (eg eats its timeslice for X times in a row) might for example be such a solution. And with the anti abuse protection it can run with far lighter privilegs.

This reminds me about the soft-RT patch posted quite sometime ago. I feel such a handy psuedo-RT scheduler class would be useful for other systems than JACK, too...

You've already proven that soft RT does not suit your requirements. The current scheduler running a task at nice -20 has extremely long periods of cpu availability at the expense of lower priority tasks and is close to the behaviour you would get with a soft RT patch. Your concern is exactly the scenario where nice -20 fails, and would be the same scenario where a soft RT policy would fail. Doing this with a scheduling policy, you want cpu time long after there is any hope for fairness or safety of hanging. From experimentation with such soft RT policies, we find average latencies can be reduced but the maximum ones, which are the ones that concern professional audio, remain the same.

Quoted 5 lines

fine, so the mlock situation may have improved enough post-2.6.9 that it can be considered fixed. that leaves the scheduler issue. but apparently, a uid/gid solution is OK for mlock, and not for the scheduler. am i missing something?

I think you skipped a step. You don't have a scheduler requirement, you have a latency requirement. You currently *solve* that latency requirement via a scheduler "hack", yet is quite clear that the "hard" realtime solution is most likely not the right approach. Note that I'm not saying that you

shouldn't get the latency that that currently provides, but the downsides (can hang the machine) are bad; a solution that solves that would be far preferable

something like a soft realtime flag that acts as if it's the hard realtime one unless the app shows "misbehavior" (eg eats its timeslice for X times in a row) might for example be such a solution. And with the anti abuse protection it can run with far lighter privilegs.

Quoted 11 lines · Paul Davis

Quoted 5 lines

fine, so the mlock situation may have improved enough post-2.6.9 that it can be considered fixed. that leaves the scheduler issue. but apparently, a uid/gid solution is OK for mlock, and not for the scheduler. am i missing something?

I think you skipped a step. You don't have a scheduler requirement, you have a latency requirement. You currently *solve* that latency requirement via a scheduler "hack", yet is quite clear that the "hard" realtime solution is most likely not the right approach. Note that I'm not saying that you

Why is that clear? In just about every respect, realtime audio has the same characteristics as hard realtime, except that nobody gets hurt when a deadline is missed :) We have an IRQ source, and a deadline (sometimes on the sub-msec range, but more typically 1-5msec) for the work that has to be done. This deadline is tight enough that the task essentially *has* to run with SCHED_FIFO scheduling, because doing almost anything else instead will cause the deadline to be missed.

Really, I think Linux has owned the server space for so long that some folks on this list are getting hubristic. Just because you have the best server OS does not mean it's the best at everything.

...
Last I checked they could be controlled separately in that module. It has been suggested (by me and others) that one possible solution would be to expand it to be generic for all caps.

Quoted 3 lines · Chris Wright

Last I checked they could be controlled separately in that module. It has been suggested (by me and others) that one possible solution would be to expand it to be generic for all caps.

Maybe this is the way?

Quoted 6 lines · Andrew Morton

Quoted 3 lines · Chris Wright

Last I checked they could be controlled separately in that module. It has been suggested (by me and others) that one possible solution would be to expand it to be generic for all caps.

Quoted 1 line

Maybe this is the way?

We already *know* how to (in principle) fix the capabilities system to make it useful. We should probably investigate doing that and at the same time fixing the current CAP_SYS_ADMIN mess (which we also have at least some ideas on fixing). The remaining problem is possible breakage of software that's doing capability things The Old Way (as the inheritance rules are incompatible).

Linus at one time said that a 2.7 might open if there was some issue that caused enough disruption to require a fork - could this be it, or does somebody have a better way to address the backward-combatability problem?

Quoted 5 lines

We already *know* how to (in principle) fix the capabilities system to make it useful. We should probably investigate doing that and at the same time fixing the current CAP_SYS_ADMIN mess (which we also have at least some ideas on fixing). The remaining problem is possible breakage of software that's doing capability things The Old Way (as the inheritance rules are incompatible).

Fixing CAP_SYS_ADMIN whole other can o' worms. No point in tangling the two.

Quoted 3 lines

Linus at one time said that a 2.7 might open if there was some issue that caused enough disruption to require a fork - could this be it, or does somebody have a better way to address the backward-combatability problem?

There's at least two ways. Introduce a new capability module or introduce a PF flag to opt in. Neither are great

fix the inheritance problems

Does anyone actually have a handle on what's involved in fixing the inheritance problem?

Quoted 2 lines · [email protected] wrote:

fix the inheritance problems

Does anyone actually have a handle on what's involved in fixing the inheritance problem?

It's risky, but it is something which we should do.

<grumpytroll> We really shouldn't have merged all that new fancy security stuff when the existing security framework was known-badly-broken. Especially as the new stuff seems incapable of doing simple things which unbroken inherited caps would do perfectly.</grumpytroll>

Quoted 5 lines · Chris Wright

...
Last I checked they could be controlled separately in that module. It has been suggested (by me and others) that one possible solution would be to expand it to be generic for all caps.

Maybe this is the way?

Quoted 8 lines · Andrew Morton

Quoted 5 lines · Chris Wright

...
Last I checked they could be controlled separately in that module. It has been suggested (by me and others) that one possible solution would be to expand it to be generic for all caps.

Maybe this is the way?

It's at least not as bad as the current hack (when properly done in the capabilities modules instead of adding one ontop).

I must say I'm not exactly happy with that idea still. It ties the privilegues we have been separating from a special uid (0) to filesystem permissions again. It's not nessecarily a bad idea per, but it doesn't really fit into the model we've been working to. I'd expect quite a few unpleasant devices when a user detects that the distibution had been binding various capabilities to uids/gids behinds his back.

So to make forward progress I'd like the audio people to confirm whether the mlock bits in 2.6.9+ do help that half of their requirement first

(and if not find a way to fix it) and then tackle the scheduling part. For that one I really wonder whether the combination of the now actually working nicelevels (see Mingo's post) and a simple wrapper for the really high requirements cases doesn't work.

Quoted 2 lines · * Christoph Hellwig ([email protected]) wrote:

So to make forward progress I'd like the audio people to confirm whether the mlock bits in 2.6.9+ do help that half of their requirement first

It sure should, but I guess they can reply on that.

Quoted 4 lines

(and if not find a way to fix it) and then tackle the scheduling part. For that one I really wonder whether the combination of the now actually working nicelevels (see Mingo's post) and a simple wrapper for the really high requirements cases doesn't work.

I saw Jack (I think) post some numbers showing that it wasn't enough. What about making priority level protected via rlimit?

Here's an uncompiled, untested patch doing that (probably has some math error or logic hole in it, but idea seems sound enough). I think it has at least one problem, where nice 19 process, could renice itself back to
0. And it doesn't really handle the different scheduling policies, other than implicit 40 - 139 being used for SCHED_FIFO/SCHED_RR.

It takes the 140 priority levels (0-139), inverts their priority order, and then uses that number as the basis for the rlimit (so that a larger rlimit means higher priority, to fall inline with normal rlimit semantics). Defaults to 19 (which should be niceval of 0). And allows CAP_SYS_NICE to continue to override if the rlimit is too low.

I find it hard to understand why some of you think PAM is an adequate solution. As currently deployed, it is poorly documented and nearly impossible for non-experts to administer securely. On my Debian woody system, when I login from the console I get one fairly sensible set of ulimit values, but from gdm I get a much more permissive set (with ulimited mlocking, BTW). Apparently, this is because the gdm' PAM config includes session required pam_limits.so' but the system comes with an empty /etc/security/limits.conf. I'm just guessing about that because I can't find any decent documentation for any of this crap.

Remember, if something is difficult to administer, it's *not* secure.

Quoted 10 lines · Jack O'Quin

I find it hard to understand why some of you think PAM is an adequate solution. As currently deployed, it is poorly documented and nearly impossible for non-experts to administer securely. On my Debian woody system, when I login from the console I get one fairly sensible set of ulimit values, but from gdm I get a much more permissive set (with ulimited mlocking, BTW). Apparently, this is because the gdm' PAM config includes session required pam_limits.so' but the system comes with an empty /etc/security/limits.conf. I'm just guessing about that because I can't find any decent documentation for any of this crap.

Quoted 1 line

Remember, if something is difficult to administer, it's *not* secure.

Not to mention that not everyone chooses to use PAM for precisely this reason. Slackware has never included PAM and probably never will. My audio workstation has worked swell with the 2.4+caps solution and the 2.6+LSM solution. PAM would break me ::-(

Not to mention that not everyone chooses to use PAM for precisely this reason. Slackware has never included PAM and probably never will. My audio workstation has worked swell with the 2.4+caps solution and the 2.6+LSM solution. PAM would break me ::-(

Quoted 4 lines · On Sat, 2005-01-08 at 11:56 -0500, [email protected] wrote:

Not to mention that not everyone chooses to use PAM for precisely this reason. Slackware has never included PAM and probably never will. My audio workstation has worked swell with the 2.4+caps solution and the 2.6+LSM solution. PAM would break me ::-(

Hmm. How could you (for example) configure all your machines to authenticate against an LDAP server without PAM?

I find it hard to understand why some of you think PAM is an adequate solution. As currently deployed, it is poorly documented and nearly impossible for non-experts to administer securely. On my Debian woody system, when I login from the console I get one fairly sensible set of ulimit values, but from gdm I get a much more permissive set (with ulimited mlocking, BTW). Apparently, this is because the gdm' PAM config includes session required pam_limits.so' but the system comes with an empty /etc/security/limits.conf. I'm just guessing about that because I can't find any decent documentation for any of this crap.

Quoted 4 lines · Chris Wright

Quoted 2 lines · * Christoph Hellwig ([email protected]) wrote:

So to make forward progress I'd like the audio people to confirm whether the mlock bits in 2.6.9+ do help that half of their requirement first

Quoted 1 line

It sure should, but I guess they can reply on that.

That does seem to work now (finally). It looks like that longstanding CAP_IPC_LOCK bug is finally fixed, too.

I find it hard to understand why some of you think PAM is an adequate solution.

Running `nice --20' is still significantly worse than SCHED_FIFO, but not the unmitigated disaster shown in the middle column. But, this improved performance is still not adequate for audio work. The worst delay was absurdly long (~1/2 sec).

Rlimits are neither UID/GID or PAM-specific. They fit well within the general model of UNIX security, extending an existing mechanism rather than adding a completely new one. That PAM happens to be the way rlimits are usually administered may be unfortunate, yes, but it doesn't mean that rlimits is the wrong way.

Quoted 4 lines

Running `nice --20' is still significantly worse than SCHED_FIFO, but not the unmitigated disaster shown in the middle column. But, this improved performance is still not adequate for audio work. The worst delay was absurdly long (~1/2 sec).

Let's work on that. It'd be _far_ better to have unprivileged near-RT capability everywhere without potential scheduling DoS.

Quoted 5 lines

Rlimits are neither UID/GID or PAM-specific. They fit well within the general model of UNIX security, extending an existing mechanism rather than adding a completely new one. That PAM happens to be the way rlimits are usually administered may be unfortunate, yes, but it doesn't mean that rlimits is the wrong way.

agreed, although i note with interest the flap over RLIMIT_MEMLOCK being made accessible to unprivileged users by people working on grsecurity.

Quoted 2 lines

Let's work on that. It'd be _far_ better to have unprivileged near-RT capability everywhere without potential scheduling DoS.

I am not sure what you mean here. I think we've established that SCHED_OTHER cannot be made adequate for realtime audio work. Its intended purpose (timesharing the machine in ways that should generally benefit tasks that don't do a lot and/or are dominated by user interaction, thus rendering the machine apparently responsive) is really at odds with what we need.

Con has discussed the idea of a new scheduling class, one that has no internal priority, runs like SCHED_RR but is subject to cpu utilization limits, and is accessible to unprivileged users. I think this makes a lot of sense. It can be controlled using sysctl's and/or rlimit.

But please note: in any sane world, adding stuff like this could only take place in an unstable tree. It seems really odd to me that anyone can be talking about adding any of these *mechanisms* to 2.6. That was the whole reason we (well, Jack, Torben and others) worked with LSM:
LSM appeared to be the "blessed" method in 2.6 of allowing changes to security policy to be made. We are now finding out that even if Linus "blessed" it by inclusion, there is enough vocal opposition to actually using it for something useful that something else has to be done. I wouldn't want to run an important machine on 2.6 if adding, say SCHED_ISO or even RLIMIT_RT_CPU is part of 2.6's "maintainance".

Meanwhile, as I mentioned before, every realtime audio user of 2.6 is *still* going to use "realtime" LSM because its really the only effective way to get the privilege needed to do what they want to get done.

Quoted 5 lines · Paul Davis

Quoted 5 lines

Rlimits are neither UID/GID or PAM-specific. They fit well within the general model of UNIX security, extending an existing mechanism rather than adding a completely new one. That PAM happens to be the way rlimits are usually administered may be unfortunate, yes, but it doesn't mean that rlimits is the wrong way.

PAM is how most GNU/Linux systems manage rlimits. It is very UID/GID oriented. So from the sysadmin perspective, claiming that rlimits is "better" or "easier to manage" than "GID hacks" is bogus.

Quoted 5 lines · Paul Davis

Quoted 5 lines

Rlimits are neither UID/GID or PAM-specific. They fit well within the general model of UNIX security, extending an existing mechanism rather than adding a completely new one. That PAM happens to be the way rlimits are usually administered may be unfortunate, yes, but it doesn't mean that rlimits is the wrong way.

PAM is how most GNU/Linux systems manage rlimits. It is very UID/GID oriented. So from the sysadmin perspective, claiming that rlimits is "better" or "easier to manage" than "GID hacks" is bogus.

Quoted 5 lines · Paul Davis

Quoted 5 lines

Rlimits are neither UID/GID or PAM-specific. They fit well within the general model of UNIX security, extending an existing mechanism rather than adding a completely new one. That PAM happens to be the way rlimits are usually administered may be unfortunate, yes, but it doesn't mean that rlimits is the wrong way.

PAM is how most GNU/Linux systems manage rlimits. It is very UID/GID oriented. So from the sysadmin perspective, claiming that rlimits is "better" or "easier to manage" than "GID hacks" is bogus.

Quoted 3 lines

agreed, although i note with interest the flap over RLIMIT_MEMLOCK being made accessible to unprivileged users by people working on grsecurity.

:-)

Quoted 7 lines

Quoted 4 lines

Running `nice --20' is still significantly worse than SCHED_FIFO, but not the unmitigated disaster shown in the middle column. But, this improved performance is still not adequate for audio work. The worst delay was absurdly long (~1/2 sec).

Let's work on that. It'd be _far_ better to have unprivileged near-RT capability everywhere without potential scheduling DoS.

I am not sure what you mean here. I think we've established that SCHED_OTHER cannot be made adequate for realtime audio work. Its intended purpose (timesharing the machine in ways that should generally benefit tasks that don't do a lot and/or are dominated by user interaction, thus rendering the machine apparently responsive) is really at odds with what we need.

Quoted 6 lines

I am not sure what you mean here. I think we've established that SCHED_OTHER cannot be made adequate for realtime audio work. Its intended purpose (timesharing the machine in ways that should generally benefit tasks that don't do a lot and/or are dominated by user interaction, thus rendering the machine apparently responsive) is really at odds with what we need.

We have not established that at all. In principle, because SCHED_OTHER tasks running at full priority lie on the boundary between SCHED_OTHER and SCHED_FIFO, they can be made to run arbitrarily close to the performance of tasks in SCHED_FIFO. With the upside that they won't be able to deadlock the machine.

And I mean arbitrarily close in the strict delta-epsilon sense. It's not perfect, but neither is SCHED_FIFO, in principle or in practice.

Quoted 6 lines · Paul Davis

I am not sure what you mean here. I think we've established that SCHED_OTHER cannot be made adequate for realtime audio work. Its intended purpose (timesharing the machine in ways that should generally benefit tasks that don't do a lot and/or are dominated by user interaction, thus rendering the machine apparently responsive) is really at odds with what we need.

We have not established that at all. In principle, because SCHED_OTHER tasks running at full priority lie on the boundary between SCHED_OTHER and SCHED_FIFO, they can be made to run arbitrarily close to the performance of tasks in SCHED_FIFO. With the upside that they won't be able to deadlock the machine.

Quoted 5 lines

We have not established that at all. In principle, because SCHED_OTHER tasks running at full priority lie on the boundary between SCHED_OTHER and SCHED_FIFO, they can be made to run arbitrarily close to the performance of tasks in SCHED_FIFO. With the upside that they won't be able to deadlock the machine.

I don't think they lie quite so neatly on this boundary. There's one fundamental difference which is how the dynamic priority is adjusted which alters the basic preemptibility rules.

Quoted 3 lines · Chris Wright

I don't think they lie quite so neatly on this boundary. There's one fundamental difference which is how the dynamic priority is adjusted which alters the basic preemptibility rules.

but at nice level -20 this adjustment is at most +5 priority levels - i.e. down to an equivalent of nice -15. Consider that a nice 0 task can at most get a -5 priority boost gives a nice -5 task worst-case - so the nice -20 task still preempts the lower prio task.

Hmm, I wonder if this could have anything to do with it. These are within striking range:

PID COMMAND NI PRI

    9 events/1        -10  34
  931 kcryptd/1       -10  33
  930 kcryptd/0       -10  34
    8 events/0        -10  34
  892 ata/1           -10  34
  891 ata/0           -10  34
 3747 udevd           -10  33
   26 kacpid          -10  31
  238 aio/1           -10  34
  237 aio/0           -10  31
  117 kblockd/1       -10  34
  116 kblockd/0       -10  34
   10 khelper         -10  34

you are right, i forgot about kernel threads. If they are nice -10 on Jack's system too then they are within striking range indeed, especially since they are typically idle and if then they are active for short bursts of time and get the maximum boost. Jack, could you renice these to -5, to make sure they dont interfere?

btw., why are these at nice -10? workqueue.c sets nice value to -5 normally.

Quoted 5 lines · * Ingo Molnar ([email protected]) wrote:

you are right, i forgot about kernel threads. If they are nice -10 on Jack's system too then they are within striking range indeed, especially since they are typically idle and if then they are active for short bursts of time and get the maximum boost. Jack, could you renice these to -5, to make sure they dont interfere?

Yup, their bursty nature makes them seem a likely culprit.

Quoted 2 lines

btw., why are these at nice -10? workqueue.c sets nice value to -5 normally.

Heh, I was just wondering the same thing.

BTW, grepping set_user_nice shows a few more possible culprits. One more reason that there may be value in promoting the audio app to rt scheduling.

Quoted 4 lines · Chris Wright

Hmm, I wonder if this could have anything to do with it. These are within striking range:

PID COMMAND NI PRI

Quoted 13 lines

    9 events/1        -10  34
  931 kcryptd/1       -10  33
  930 kcryptd/0       -10  34
    8 events/0        -10  34
  892 ata/1           -10  34
  891 ata/0           -10  34
 3747 udevd           -10  33
   26 kacpid          -10  31
  238 aio/1           -10  34
  237 aio/0           -10  31
  117 kblockd/1       -10  34
  116 kblockd/0       -10  34
   10 khelper         -10  34

you are right, i forgot about kernel threads. If they are nice -10 on Jack's system too then they are within striking range indeed, especially since they are typically idle and if then they are active for short bursts of time and get the maximum boost. Jack, could you renice these to -5, to make sure they dont interfere?

Quoted 5 lines · Ingo Molnar

you are right, i forgot about kernel threads. If they are nice -10 on Jack's system too then they are within striking range indeed, especially since they are typically idle and if then they are active for short bursts of time and get the maximum boost. Jack, could you renice these to -5, to make sure they dont interfere?

Sure. My system does have some of these running at nice -10. Where (how) do I change them?

BTW, let's not lose sight of the fact that `nice --20 foo' requires CAP_SYS_NICE just like SCHED_FIFO does. From a privilege perspective, this recurses to the same (still unsolved) problem.

Chris's rlimits proposal was the only workable suggestion I've seen for that. Is there any hope of doing something like that in the 2.6.x timeframe?

Quoted 4 lines · Chris Wright

Hmm, I wonder if this could have anything to do with it. These are within striking range:

PID COMMAND NI PRI

Quoted 13 lines

    9 events/1        -10  34
  931 kcryptd/1       -10  33
  930 kcryptd/0       -10  34
    8 events/0        -10  34
  892 ata/1           -10  34
  891 ata/0           -10  34
 3747 udevd           -10  33
   26 kacpid          -10  31
  238 aio/1           -10  34
  237 aio/0           -10  31
  117 kblockd/1       -10  34
  116 kblockd/0       -10  34
   10 khelper         -10  34

you are right, i forgot about kernel threads. If they are nice -10 on Jack's system too then they are within striking range indeed, especially since they are typically idle and if then they are active for short bursts of time and get the maximum boost. Jack, could you renice these to -5, to make sure they dont interfere?

One major problem: this `nice --20' hack affects every thread, not just the critical realtime ones. That's not what we want. Audio applications make very conscious choices which threads run with high priority and which do not.

Plus, we maintain JACK for several platforms including GNU/Linux, FreeBSD, and Mac OS X. IRIX support is planned soon, possibly Solaris some day. I would really prefer for Linux to support genuine POSIX realtime with SCHED_FIFO scheduling. Since that is our primary development platform, it makes our code a lot more portable.

And, this is not just about JACK. We could change to call nice() instead of pthread_setschedparam() on Linux, but that about all the other audio applications? I don't think this is a reasonable thing to ask of people. It would take a year just to get them all changed, like herding cats.

This whole approach seems like a "dry well" to me.

Tomorrow, I'll try the test again after making a new kernel with STARVATION_LIMIT set to zero.

Anything else I should try?

If we can get high priority SCHED_OTHER working sufficiently well, that will be preferable in the long run as the security implications are slightly less dire. It's already been noted that it doesn't solve your privilege problem, but it's still interesting to us because it has potential to address the deadlock issue.

Doesn't mean you have to use it (though you'll probably want to give your users the option).

Quoted 1 line

This whole approach seems like a "dry well" to me.

It may turn out to be. Please continue testing it though - you've got a good test case handy.

Quoted 4 lines

Tomorrow, I'll try the test again after making a new kernel with STARVATION_LIMIT set to zero.

Anything else I should try?

Testing feedback on the bits from Ingo that have gone to -mm will probably help speed their acceptance in mainline.

But there may be other, better solutions to the deadlock problem. Several years ago, Roger Larsson wrote a completely user-space realtime monitor program that works perfectly well for revoking realtime privileges when it detects CPU starvation. I still use it occasionally to help debug problems if the built-in JACK watchdog timer doesn't catch them.

Quoted 6 lines

But there may be other, better solutions to the deadlock problem. Several years ago, Roger Larsson wrote a completely user-space realtime monitor program that works perfectly well for revoking realtime privileges when it detects CPU starvation. I still use it occasionally to help debug problems if the built-in JACK watchdog timer doesn't catch them.

Do you have a link to Roger Larsson's RT watchdog?

Quoted 8 lines · Jack O'Quin

Quoted 7 lines

But there may be other, better solutions to the deadlock problem. Several years ago, Roger Larsson wrote a completely user-space realtime monitor program that works perfectly well for revoking realtime privileges when it detects CPU starvation. I still use it occasionally to help debug problems if the built-in JACK watchdog timer doesn't catch them.

Lee Revell <[email protected]> writes:

Quoted 1 line

Do you have a link to Roger Larsson's RT watchdog?

No official, supported version. With his permission, I posted a copy on my home system a year ago for some audio users who had inquired about it. That copy is here...

http://www.joq.us/joq/rt_monitor.tgz

Quoted 2 lines

Quoted 1 line · Lee Revell

Do you have a link to Roger Larsson's RT watchdog?

Quoted 4 lines · Jack O'Quin

No official, supported version. With his permission, I posted a copy on my home system a year ago for some audio users who had inquired about it. That copy is here...
http://www.joq.us/joq/rt_monitor.tgz

Bad URL, not found....

OK, I reran with just 5 processes reniced from -10 to -5. On my system they were: events, khelper, kblockd, aio and reiserfs. In addition, I reniced loop0 from -20 to -5.

One major problem: this `nice --20' hack affects every thread, not just the critical realtime ones. That's not what we want. Audio applications make very conscious choices which threads run with high priority and which do not.

Quoted 3 lines · Jack O'Quin

OK, I reran with just 5 processes reniced from -10 to -5. On my system they were: events, khelper, kblockd, aio and reiserfs. In addition, I reniced loop0 from -20 to -5.

Quoted 4 lines

One major problem: this `nice --20' hack affects every thread, not just the critical realtime ones. That's not what we want. Audio applications make very conscious choices which threads run with high priority and which do not.

how much did this problem affect your test? Could the source of the 500 msec delays be the non-highprio components of the test that somehow became nice --20?

Quoted 7 lines · Jack O'Quin

Quoted 4 lines

One major problem: this `nice --20' hack affects every thread, not just the critical realtime ones. That's not what we want. Audio applications make very conscious choices which threads run with high priority and which do not.

Quoted 3 lines · Ingo Molnar

how much did this problem affect your test? Could the source of the 500 msec delays be the non-highprio components of the test that somehow became nice --20?

Probably, the best way to tell would be patching JACK so it uses nice(-20) instead of pthread_setschedparam() for the realtime threads. As a hack, that looks easy. I'll build a working directory with just that change, so we can experiment with it better.

According to the manpage, nice(2) is per-process not per-thread. That does not give the granularity we need.

Quoted 2 lines

According to the manpage, nice(2) is per-process not per-thread. That does not give the granularity we need.

the manpage is incorrect - sys_nice() is per-thread. (Btw., you could use setpriority() too.)

Quoted 3 lines · Jack O'Quin

Quoted 2 lines

According to the manpage, nice(2) is per-process not per-thread. That does not give the granularity we need.

Quoted 2 lines · Ingo Molnar

the manpage is incorrect - sys_nice() is per-thread. (Btw., you could use setpriority() too.)

OK. Where is this stuff documented?

BTW, I think this violates POSIX, which states...

The nice value set with nice() shall be applied to the process. If the process is multi-threaded, the nice value shall affect all system scope threads in the process.

We are talking about two different things here. POSIX is just about API and has, correct me if I'm wrong, nothing to do with system calls whatsoever. The manpage nice(2) is about the libc library call nice(), which is per-process, which it should be according to POSIX. The system call, called sys_nice() in C, is per-thread. Apparently glibc or some thread library contains some magic to make the translation.

Is it possible to call sched_setscheduler() with a thread ID instead of a pid? That's what I really need. JACK sets and resets the thread priorities from a different thread.

Quoted 3 lines · Jack O'Quin

Is it possible to call sched_setscheduler() with a thread ID instead of a pid? That's what I really need. JACK sets and resets the thread priorities from a different thread.

yes. The PID arguments in these APIs are all treated as 'TIDs'. One day the APIs themselves might switch over to what POSIX specifies, and there will be new, thread-specific APIs - but at the moment they are all thread-granular.

In the absence of any documentation, I'm guessing about storing the nice value in the priority field of the sched_param struct. But, I have not been able to figure out how to make that work.

Quoted 3 lines · Jack O'Quin

In the absence of any documentation, I'm guessing about storing the nice value in the priority field of the sched_param struct. But, I have not been able to figure out how to make that work.

the call you need is:

       setpriority(PRIO_PROCESS, tid, -20);

where 'tid' is the TID (pid) of the thread in question. There's no way i know of to utilize the pthread_t ID to do this, so you'll have to figure the TID out via gettid() - which needs to happen in the child context - how hard would it be to attach the TID field to some per-thread Jack structure? [while the purpose is still a quick hack.]

Adding a tid field is relatively easy. Fixing the race condition between setting it in the new thread and using it in the creating thread is harder, but not impossible. But, even setting it in the new thread would create an incompatible interface. With hundreds of JACK client applications, binary compatibility is a serious consideration.

i'm not suggesting that this is the way to go, it's just to test how nice--20 tasks would perform (on the hacked kernel). We still dont have this data, because in the other tests you tried, some non-highprio threads got nice--20 priority as well, which can (and apparently do) interfere with the highprio threads.

Quoted 5 lines · Ingo Molnar

i'm not suggesting that this is the way to go, it's just to test how nice--20 tasks would perform (on the hacked kernel). We still dont have this data, because in the other tests you tried, some non-highprio threads got nice--20 priority as well, which can (and apparently do) interfere with the highprio threads.

('highprio threads' are the ones that normally get SCHED_FIFO priority with -R, 'lowprio threads' are the other client-side threads, if any.)

to make it easier to test, i've written an API hack: with the kernel patch below setscheduler() will set the task to nice --20 if you use SCHED_FIFO and sched_priority of 1. I.e. all you need to do is to run Jack with -R and use an RT priority of 1 - all the highprio threads should then become nice --20. If you use RT prio 2 (or higher) it should be SCHED_FIFO again. Just apply the patch to 2.6.11-rc1 (2.6.10 might work too) and it will work automatically. (the hack also includes the earlier 'no starvation for nice--20 tasks' hack.)

kernel/sched.c renamed from kernel/sched.c.orig +0 -0

@@ -3211,6 +3211,12 @@ static inline task_t *find_process_by_pi
 static void __setscheduler(struct task_struct *p, int policy, int prio)
 {

Quoted 4 lines

kernel/sched.c renamed from kernel/sched.c.orig +0 -0

@@ -3211,6 +3211,12 @@ static inline task_t *find_process_by_pi
 static void __setscheduler(struct task_struct *p, int policy, int prio)
 {

kernel/sched.c static inline task_t *find_process_by_pi · line 3222

Quoted 10 lines

kernel/sched.c renamed from kernel/sched.c.orig +6 -0

 	BUG_ON(p->array);
+	if (prio == 1 && policy != SCHED_NORMAL) {
+		p->policy = SCHED_NORMAL;
+		p->static_prio = NICE_TO_PRIO(-20);
+		p->prio = p->static_prio;
+		return;
+	}
 	p->policy = policy;
 	p->rt_priority = prio;
 	if (policy != SCHED_NORMAL)

JACK actually uses three different priorities, the defaults are 9, 10 and 20. How about if I change this test?

	if (prio <= 20 && policy != SCHED_NORMAL) {

Or, should that be?

	if (prio > 0 && prio <= 20 && policy != SCHED_NORMAL) {

Quoted 3 lines

Or, should that be?

	if (prio > 0 && prio <= 20 && policy != SCHED_NORMAL) {

Or you can just drop the 'prio == 1 &&' part for this test. Ingo was trying to be clever to allow some RT bits, but that's not really necessary.

JACK actually uses three different priorities, the defaults are 9, 10 and 20. How about if I change this test?

	if (prio <= 20 && policy != SCHED_NORMAL) {

Or, should that be?

	if (prio > 0 && prio <= 20 && policy != SCHED_NORMAL) {

Quoted 2 lines

I find it hard to understand why some of you think PAM is an adequate solution.

The best we can do _here_ is present something that userspace can use sensibly. We can't make userspace actually use it that way though.

Rlimits are neither UID/GID or PAM-specific. They fit well within the general model of UNIX security, extending an existing mechanism rather than adding a completely new one. That PAM happens to be the way rlimits are usually administered may be unfortunate, yes, but it doesn't mean that rlimits is the wrong way.

Quoted 4 lines

Running `nice --20' is still significantly worse than SCHED_FIFO, but not the unmitigated disaster shown in the middle column. But, this improved performance is still not adequate for audio work. The worst delay was absurdly long (~1/2 sec).

Let's work on that. It'd be _far_ better to have unprivileged near-RT capability everywhere without potential scheduling DoS.

Quoted 2 lines

Let's work on that. It'd be _far_ better to have unprivileged near-RT capability everywhere without potential scheduling DoS.

"Near-RT" is about the most useless concept I've heard of in a long time. It sounds like the answer to a question nobody asked. ;-)

Quoted 2 lines · Jack O'Quin

"Near-RT" is about the most useless concept I've heard of in a long time. It sounds like the answer to a question nobody asked. ;-)

To my way of thinking, it's a pretty good description of Ingo's work or anything you're ever going to see on a PC. If you think you're going to get real hard RT performance on your off-the-shelf x86 box running a conventional OS, you are fooling yourself.

Thankfully a buffer underrun is no more fatal for pro audio than a broken guitar string. CDs skip, DATs glitch, XLR cables flake out, circuit breakers trip, amps clip, Powerbooks crash, and the show goes on. I've done more than enough stage tech to know it's a huge pain in the ass, but let's stop pretending we require absolute perfection, please.

Quoted 9 lines · Matt Mackall

Quoted 2 lines · Jack O'Quin

"Near-RT" is about the most useless concept I've heard of in a long time. It sounds like the answer to a question nobody asked. ;-)

To my way of thinking, it's a pretty good description of Ingo's work or anything you're ever going to see on a PC. If you think you're going to get real hard RT performance on your off-the-shelf x86 box running a conventional OS, you are fooling yourself.

Quoted 6 lines

Thankfully a buffer underrun is no more fatal for pro audio than a broken guitar string. CDs skip, DATs glitch, XLR cables flake out, circuit breakers trip, amps clip, Powerbooks crash, and the show goes on. I've done more than enough stage tech to know it's a huge pain in the ass, but let's stop pretending we require absolute perfection, please.

In _practice_, Ingo's patches are considerably better than what you seem to consider "good enough for mere audio work".

Anyway, *plonk*.

Quoted 1 line · Matt Mackall

Anyway, *plonk*.

Plonk? WTF? Jack comes up with what many people think is a reasonable solution to a real problem, that affects thousands of users, and in the middle of what seems to me a civilized discussion, you killfile him because he disagrees with you?

Plonk to you too, asshole.

Quoted 8 lines · * Lee Revell ([email protected]) wrote:

Quoted 1 line · Matt Mackall

Anyway, *plonk*.

Plonk? WTF? Jack comes up with what many people think is a reasonable solution to a real problem, that affects thousands of users, and in the middle of what seems to me a civilized discussion, you killfile him because he disagrees with you?

Quoted 1 line

Plonk to you too, asshole.

Guys, could we please bring this back to a useful discussion. None of you have commented on whether the rlimits for priority are useful. As I said before, I've no real problem with the module as it stands since it's tiny, quite contained, and does something people need. But I agree it'd be better to find something that's workable as long term solution.

Quoted 5 lines · Chris Wright

Guys, could we please bring this back to a useful discussion. None of you have commented on whether the rlimits for priority are useful. As I said before, I've no real problem with the module as it stands since it's tiny, quite contained, and does something people need. But I agree it'd be better to find something that's workable as long term solution.

Chris, I did comment on it, see [email protected] from around 5:15 on Saturday.

Guys, could we please bring this back to a useful discussion. None of you have commented on whether the rlimits for priority are useful. As I said before, I've no real problem with the module as it stands since it's tiny, quite contained, and does something people need. But I agree it'd be better to find something that's workable as long term solution.

20: privilege to set any RT priority

But I'm also still not convinced this policy can't be most flexibly handled by a setuid helper together with the mlock rlimit.

Why can't this be done with a simple SUID helper to promote given tasks to RT with sched_setschedule, doing essentially all the checks this LSM is doing?

Objections of "because it requires dangerous root or suid" don't fly, an RT app under user control can DoS the box trivially. Never mind you need root to configure the LSM anyway..

Yes but a bug in an app running as root can trash the filesystem. The worst you can do with RT privileges is lock up the machine.

Quoted 2 lines

Yes but a bug in an app running as root can trash the filesystem. The worst you can do with RT privileges is lock up the machine.

several filesystem and IO threads run at prio -10 but not RT. That makes me a bit less sure of your statement....

Quoted 4 lines · Lee Revell

Quoted 2 lines

Yes but a bug in an app running as root can trash the filesystem. The worst you can do with RT privileges is lock up the machine.

Quoted 2 lines

several filesystem and IO threads run at prio -10 but not RT. That makes me a bit less sure of your statement....

Its completely orthogonal. Lee didn't say "tasks running without RT can't mess up filesystems". He said "tasks running as root can trash the filesystem" and "tasks running as RT can lock up the machine". obviously, the intersection point (a root, RT task) is double trouble.

Quoted 4 lines · * Paul Davis ([email protected]) wrote:

Quoted 4 lines · Lee Revell

Quoted 2 lines

Yes but a bug in an app running as root can trash the filesystem. The worst you can do with RT privileges is lock up the machine.

Quoted 8 lines

Quoted 2 lines

several filesystem and IO threads run at prio -10 but not RT. That makes me a bit less sure of your statement....

Its completely orthogonal. Lee didn't say "tasks running without RT can't mess up filesystems". He said "tasks running as root can trash the filesystem" and "tasks running as RT can lock up the machine". obviously, the intersection point (a root, RT task) is double trouble.

This is straying from the core issue... But, Arjan's saying that an RT (non-root) task could trash the filesystem if it deadlocks the machine (because those important fs and IO threads don't run).

Quoted 3 lines

This is straying from the core issue... But, Arjan's saying that an RT (non-root) task could trash the filesystem if it deadlocks the machine (because those important fs and IO threads don't run).

Lexicographic ambiguity: Lee and Paul are using "trash" for things like installing a hidden suid root shell or co-opting sendmail into an open spam relay. Arjan just means crashing the system which forces reboot to run fsck.

Quoted 8 lines · Jack O'Quin

Quoted 3 lines

This is straying from the core issue... But, Arjan's saying that an RT (non-root) task could trash the filesystem if it deadlocks the machine (because those important fs and IO threads don't run).

Lexicographic ambiguity: Lee and Paul are using "trash" for things like installing a hidden suid root shell or co-opting sendmail into an open spam relay. Arjan just means crashing the system which forces reboot to run fsck.

I actually meant data corruption.

Quoted 4 lines · Jack O'Quin

Lexicographic ambiguity: Lee and Paul are using "trash" for things like installing a hidden suid root shell or co-opting sendmail into an open spam relay. Arjan just means crashing the system which forces reboot to run fsck.

I actually meant data corruption.

Quoted 6 lines · Arjan van de Ven

Quoted 4 lines · Jack O'Quin

Lexicographic ambiguity: Lee and Paul are using "trash" for things like installing a hidden suid root shell or co-opting sendmail into an open spam relay. Arjan just means crashing the system which forces reboot to run fsck.

Quoted 1 line

I actually meant data corruption.

Are you concerned about something different from the "normal" risk of data corruption when the kernel panics or someone trips over the power cord?

Quoted 7 lines

Quoted 4 lines · Jack O'Quin

Quoted 2 lines

Arjan just means crashing the system which forces reboot to run fsck.

Quoted 1 line · Arjan van de Ven

I actually meant data corruption.

Quoted 3 lines · Jack O'Quin

Are you concerned about something different from the "normal" risk of data corruption when the kernel panics or someone trips over the power cord?

yes; the "normal" risk is time limited, eg the kernel will wait at most 30 seconds before writing back your dirty data, 5 seconds for ext3 actually. With the "RT-abuse" hang, this 30 second thing goes on hold (because it's done from those kernel threads that cause you those hickups in sound :-) and you can starve a far longer period of time.. which may well mean a far larger dataset not hitting the disk.

(Probably, this simplistic analysis misses some other, more subtle, factors.)

RT threads should not do FS writes of their own. But, a badly broken or malicious one could, I suppose. So, that might provide a mechanism for losing more data than usual. Is that what you had in mind?

Quoted 3 lines · Jack O'Quin

(Probably, this simplistic analysis misses some other, more subtle, factors.)

I think you can do nasty things to the locks held by those threads too

Quoted 1 line

Quoted 3 lines

RT threads should not do FS writes of their own. But, a badly broken or malicious one could, I suppose. So, that might provide a mechanism for losing more data than usual. Is that what you had in mind?

basically yes.
note that "FS writes" can come from various things, including library calls made and such. But I think you got my point; even though it might seem a bit theoretical it sure is unpleasant.

The basic issue is that the current semantics of SCHED_FIFO seem make the deadlock/data corruption due to runaway RT thread issue difficult. The obvious solution is a new scheduling class equivalent to SCHED_FIFO but with a mechanism for the kernel to demote the offending thread to SCHED_OTHER in an emergency.

Quoted 5 lines · Lee Revell

The basic issue is that the current semantics of SCHED_FIFO seem make the deadlock/data corruption due to runaway RT thread issue difficult. The obvious solution is a new scheduling class equivalent to SCHED_FIFO but with a mechanism for the kernel to demote the offending thread to SCHED_OTHER in an emergency.

and this is getting really close to the original "counter proposal" to the LSM module that was basically "lets make lower nice limit an rlimit, and have -20 mean "basically FIFO" *if* the task behaves itself".

Otherwise, your rlimits proposal is fine. I still think it puts more of a burden on the sysadmin, but nobody else seems to care about that.

Quoted 8 lines · Arjan van de Ven

Quoted 5 lines · Lee Revell

The basic issue is that the current semantics of SCHED_FIFO seem make the deadlock/data corruption due to runaway RT thread issue difficult. The obvious solution is a new scheduling class equivalent to SCHED_FIFO but with a mechanism for the kernel to demote the offending thread to SCHED_OTHER in an emergency.

Quoted 3 lines

and this is getting really close to the original "counter proposal" to the LSM module that was basically "lets make lower nice limit an rlimit, and have -20 mean "basically FIFO" *if* the task behaves itself".

Yes. However, my tests have so far shown a need for "actual FIFO as long as the task behaves itself."

Quoted 11 lines · Jack O'Quin

Quoted 8 lines · Arjan van de Ven

Quoted 5 lines · Lee Revell

The basic issue is that the current semantics of SCHED_FIFO seem make the deadlock/data corruption due to runaway RT thread issue difficult. The obvious solution is a new scheduling class equivalent to SCHED_FIFO but with a mechanism for the kernel to demote the offending thread to SCHED_OTHER in an emergency.

Quoted 7 lines

Quoted 3 lines

and this is getting really close to the original "counter proposal" to the LSM module that was basically "lets make lower nice limit an rlimit, and have -20 mean "basically FIFO" *if* the task behaves itself".

Yes. However, my tests have so far shown a need for "actual FIFO as long as the task behaves itself."

I should comment on this thread on lkml. After some investigation/discussion and testing I came up with a proposal for this problem. Since we are a general purpose operating system and not a hard rt system (although addon patches are clearly making that a future possibility) we need a solution that is satisfactory to a general...

There are two ways I suggested for this.

First, (and I am increasingly believing in the second) is to implement a new scheduling class for isochronous scheduling. This would be a class for unprivileged users, and behave like SCHED_RR (to avoid complications of QoS features we dont have infrastrucutre for) at a priority just above SCHED_NORMAL, but below all privileged SCHED_RR and SCHED_FIFO. Importantly, a soft cpu limit and rate period can be set by default for this scheduling class that provides good true SCHED_RR performance, and is configurable. Literature suggests that 70% is adequate cpu for good real time performance and would be starvation free. I believe setting 70% with 10% hysteresis (dropping to say 63% on hitting limit) would be a good start. Beyond this, however, to satisfy the needs of those with

more demanding setups, a simple configurable runtime setting to set both the cpu% and the rate period could be available to something as simple as proc
/proc/sys/kernel/iso_cpu
/proc/sys/kernel/iso_cpu_period
where iso_cpu is set to 70, and period to maybe 1 second. The actual mode of setting this tunable is not important, and could be in /sys or whatever

The second option is to not implement a new scheduling class at all, and allow unprivileged users to use either SCHED_FIFO or SCHED_RR, but to make the cpu constraints described for SCHED_ISO above apply to their use of those classes. Supporting priority settings for these could be possible, but in my opinion, it would work as a better class if they only had one priority level, as for the SCHED_ISO implementation above (better than any SCHED_NORMAL, but lower than privileged SCHED_RR/FIFO).

This latter approach to me seems the least invasive and most user and sysadmin friendly method.

What was amusing to me was that after I suggested the latter option, I discovered that was basically what OSX does, however being not a real multi-user operating system they had absurd limits for cpu at 90% by default. Theory suggests 70% should be a good default limit.

Cheers,
Con

Quoted 23 lines · Con Kolivas

Quoted 13 lines · Jack O'Quin

Quoted 9 lines · Arjan van de Ven

Quoted 5 lines · Lee Revell

The basic issue is that the current semantics of SCHED_FIFO seem make the deadlock/data corruption due to runaway RT thread issue difficult. The obvious solution is a new scheduling class equivalent to SCHED_FIFO but with a mechanism for the kernel to demote the offending thread to SCHED_OTHER in an emergency.

Quoted 7 lines

Quoted 3 lines

and this is getting really close to the original "counter proposal" to the LSM module that was basically "lets make lower nice limit an rlimit, and have -20 mean "basically FIFO" *if* the task behaves itself".

Yes. However, my tests have so far shown a need for "actual FIFO as long as the task behaves itself."

Quoted 8 lines

I should comment on this thread on lkml. After some investigation/discussion and testing I came up with a proposal for this problem. Since we are a general purpose operating system and not a hard rt system (although addon patches are clearly making that a future possibility) we need a solution that is satisfactory to a general...

There are two ways I suggested for this.

First, (and I am increasingly believing in the second) is to implement a new scheduling class for isochronous scheduling. This would be a class for unprivileged users, and behave like SCHED_RR (to avoid complications of QoS features we dont have infrastrucutre for) at a priority just above SCHED_NORMAL, but below all privileged SCHED_RR and SCHED_FIFO. Importantly, a soft cpu limit and rate period can be set by default for this scheduling class that provides good true SCHED_RR performance, and is configurable. Literature suggests that 70% is adequate cpu for good real time performance and would be starvation free. I believe setting 70% with 10% hysteresis (dropping to say 63% on hitting limit) would be a good start. Beyond this, however, to satisfy the needs of those with more demanding setups, a simple configurable runtime setting to set both the cpu% and the rate period could be available to something as simple as proc /proc/sys/kernel/iso_cpu
/proc/sys/kernel/iso_cpu_period
where iso_cpu is set to 70, and period to maybe 1 second. The actual mode of setting this tunable is not important, and could be in /sys or whatever

This sounds promising, but I think it still needs to be privileged. See a) below.

Quoted 1 line

The second option is to not implement a new scheduling class at all, and allow unprivileged users to use either SCHED_FIFO or SCHED_RR, but to make the cpu constraints described for SCHED_ISO above apply to their use of those classes. Supporting priority settings for these could be possible, but in my opinion, it would work as a better class if they only had one priority level, as for the SCHED_ISO implementation above (better than any SCHED_NORMAL, but lower than privileged SCHED_RR/FIFO).

a) How to arbitrate between competing unprivileged users that want pseudo-SCHED_FIFO? Do they all lose?
b) Priority levels are important here, we want to be able to do things like have audio run at higher priority than video.

Quoted 7 lines · Arjan van de Ven

Quoted 5 lines · Lee Revell

The basic issue is that the current semantics of SCHED_FIFO seem make the deadlock/data corruption due to runaway RT thread issue difficult. The obvious solution is a new scheduling class equivalent to SCHED_FIFO but with a mechanism for the kernel to demote the offending thread to SCHED_OTHER in an emergency.

Quoted 3 lines

and this is getting really close to the original "counter proposal" to the LSM module that was basically "lets make lower nice limit an rlimit, and have -20 mean "basically FIFO" *if* the task behaves itself".

Yes. However, my tests have so far shown a need for "actual FIFO as long as the task behaves itself."

Quoted 2 lines · Jack O'Quin

Yes. However, my tests have so far shown a need for "actual FIFO as long as the task behaves itself."

I for one wonder why that appears to be so. What happens if you use SCHED_RR instead of SCHED_FIFO?

(ie is the problem just one of running out of slice at a bad time, or is it the dynamic priority adjustment)

Quoted 7 lines · Mike Galbraith

Quoted 2 lines · Jack O'Quin

Yes. However, my tests have so far shown a need for "actual FIFO as long as the task behaves itself."

I for one wonder why that appears to be so. What happens if you use SCHED_RR instead of SCHED_FIFO?

Quoted 2 lines

(ie is the problem just one of running out of slice at a bad time, or is it the dynamic priority adjustment)

I have no quick and easy test for that.

If it's important, I can modify a version of JACK to use SCHED_RR, instead.

I very much doubt it would make any difference, since we normally only run one realtime thread at a time. Each client taps the next on the shoulder when it is time for it to run, so there is essentially no concurrency among them.

Quoted 12 lines · Jack O'Quin

Quoted 9 lines · Mike Galbraith

Quoted 2 lines · Jack O'Quin

Yes. However, my tests have so far shown a need for "actual FIFO as long as the task behaves itself."

I for one wonder why that appears to be so. What happens if you use SCHED_RR instead of SCHED_FIFO?

(ie is the problem just one of running out of slice at a bad time, or is it the dynamic priority adjustment)

Quoted 4 lines

I have no quick and easy test for that.

If it's important, I can modify a version of JACK to use SCHED_RR, instead.

I think the problem you're seeing is strange enough to consider trying the (possibly odd sounding) test. I haven't seen an explanation of why nice -20 doesn't work for you.

Quoted 4 lines

I very much doubt it would make any difference, since we normally only run one realtime thread at a time. Each client taps the next on the shoulder when it is time for it to run, so there is essentially no concurrency among them.

It may not make any difference. Seeing that would at least be an additional datapoint. The only significant difference I see between a gaggle of SCHED_FIFO tasks and one of nice -20 tasks, who are alone in their top-of-the-heap queue, and who are not cpu hogs, is the timeslice. I don't recall there being any wakeup/preempt logic differences, ergo the SCHED_RR suggestion.

Quoted 8 lines · Arjan van de Ven

Quoted 3 lines · Jack O'Quin

(Probably, this simplistic analysis misses some other, more subtle, factors.)

I think you can do nasty things to the locks held by those threads too

Quoted 1 line

Quoted 9 lines

Quoted 3 lines

RT threads should not do FS writes of their own. But, a badly broken or malicious one could, I suppose. So, that might provide a mechanism for losing more data than usual. Is that what you had in mind?

basically yes.
note that "FS writes" can come from various things, including library calls made and such. But I think you got my point; even though it might seem a bit theoretical it sure is unpleasant.

I added Con to the cc: because this thread is starting to converge with an email discussion we've been having.

The basic issue is that the current semantics of SCHED_FIFO seem make the deadlock/data corruption due to runaway RT thread issue difficult. The obvious solution is a new scheduling class equivalent to SCHED_FIFO but with a mechanism for the kernel to demote the offending thread to SCHED_OTHER in an emergency. The problem can be solved in userspace with a SCHED_FIFO watchdog thread that runs at a higher RT priority than all other RT processes.

This all seems to imply that introducing an rlimit for MAX_RT_PRIO is an excellent solution. The RT watchdog thread could run as root, and the rlimit would be used to ensure than even nonroot users in the RT group could never preempt the watchdog thread.

Quoted 31 lines · Lee Revell

Quoted 8 lines · Arjan van de Ven

Quoted 2 lines · Jack O'Quin

(Probably, this simplistic analysis misses some other, more subtle, factors.)

I think you can do nasty things to the locks held by those threads too

Quoted 9 lines

Quoted 3 lines

RT threads should not do FS writes of their own. But, a badly broken or malicious one could, I suppose. So, that might provide a mechanism for losing more data than usual. Is that what you had in mind?

basically yes.
note that "FS writes" can come from various things, including library calls made and such. But I think you got my point; even though it might seem a bit theoretical it sure is unpleasant.

I added Con to the cc: because this thread is starting to converge with an email discussion we've been having.

The basic issue is that the current semantics of SCHED_FIFO seem make the deadlock/data corruption due to runaway RT thread issue difficult. The obvious solution is a new scheduling class equivalent to SCHED_FIFO but with a mechanism for the kernel to demote the offending thread to SCHED_OTHER in an emergency. The problem can be solved in userspace with a SCHED_FIFO watchdog thread that runs at a higher RT priority than all other RT processes.

Quoted 4 lines

This all seems to imply that introducing an rlimit for MAX_RT_PRIO is an excellent solution. The RT watchdog thread could run as root, and the rlimit would be used to ensure than even nonroot users in the RT group could never preempt the watchdog thread.

Just an idea. What about throttling runaway RT tasks?
If the system spend more than 98% in RT tasks for 5s consider this as a _fatal error_. Print an error message and throttle RT tasks by inserting ticks where only SCHED_OTHER tasks allowed. For a limit of 98% this means one SCHED_OTHER only tick all 50 ticks.

The limit and timeout should be configurable and of course it can be disabled.

I know this is against RT task preempt all SCHED_OTHER but this is only for a fatal system state to be able to recover sanely. A locked up machine is is the worse alternative.

There is a patch in -mm currently designed to use a sysrq key
combination which converts all real time tasks to sched normal to save you if you desire in a lockup situation.

Quoted 48 lines · utz lehmann

Quoted 31 lines · Lee Revell

Quoted 8 lines · Arjan van de Ven

Quoted 2 lines · Jack O'Quin

(Probably, this simplistic analysis misses some other, more subtle, factors.)

I think you can do nasty things to the locks held by those threads too

Quoted 9 lines

Quoted 3 lines

RT threads should not do FS writes of their own. But, a badly broken or malicious one could, I suppose. So, that might provide a mechanism for losing more data than usual. Is that what you had in mind?

basically yes.
note that "FS writes" can come from various things, including library calls made and such. But I think you got my point; even though it might seem a bit theoretical it sure is unpleasant.

I added Con to the cc: because this thread is starting to converge with an email discussion we've been having.

The basic issue is that the current semantics of SCHED_FIFO seem make the deadlock/data corruption due to runaway RT thread issue difficult. The obvious solution is a new scheduling class equivalent to SCHED_FIFO but with a mechanism for the kernel to demote the offending thread to SCHED_OTHER in an emergency. The problem can be solved in userspace with a SCHED_FIFO watchdog thread that runs at a higher RT priority than all other RT processes.

Quoted 4 lines

This all seems to imply that introducing an rlimit for MAX_RT_PRIO is an excellent solution. The RT watchdog thread could run as root, and the rlimit would be used to ensure than even nonroot users in the RT group could never preempt the watchdog thread.

Just an idea. What about throttling runaway RT tasks?
If the system spend more than 98% in RT tasks for 5s consider this as a _fatal error_. Print an error message and throttle RT tasks by inserting ticks where only SCHED_OTHER tasks allowed. For a limit of 98% this means one SCHED_OTHER only tick all 50 ticks.

The limit and timeout should be configurable and of course it can be disabled.

Quoted 3 lines

I know this is against RT task preempt all SCHED_OTHER but this is only for a fatal system state to be able to recover sanely. A locked up machine is is the worse alternative.

There is a patch in -mm currently designed to use a sysrq key combination which converts all real time tasks to sched normal to save you if you desire in a lockup situation. We do want to preserve RT scheduling behaviour at all times without caveats for privileged users.

Quoted 7 lines · Con Kolivas

Quoted 6 lines · utz lehmann

Just an idea. What about throttling runaway RT tasks?
If the system spend more than 98% in RT tasks for 5s consider this as a _fatal error_. Print an error message and throttle RT tasks by inserting ticks where only SCHED_OTHER tasks allowed. For a limit of 98% this means one SCHED_OTHER only tick all 50 ticks.

Quoted 8 lines

Quoted 6 lines

The limit and timeout should be configurable and of course it can be disabled.

I know this is against RT task preempt all SCHED_OTHER but this is only for a fatal system state to be able to recover sanely. A locked up machine is is the worse alternative.

There is a patch in -mm currently designed to use a sysrq key combination which converts all real time tasks to sched normal to save you if you desire in a lockup situation. We do want to preserve RT scheduling behaviour at all times without caveats for privileged users.

The sysrq is already in 2.6.10. I had to use it the last days a few times. But it does help if you have no access to the console.

The RT throttling idea is not to change the behavior in normal conditions. It's only for a fatal system state. If you have a runaway RT task you can't guarantee the system is work properly anyway. It's blocking vital kernel threads, filesystems, swap, keyboard, ...

It's a bit like out of memory. You can do nothing and panic. Or trying something bad (killing processes) which is hopefully better as the former.
btw: Are RT tasks excluded by the oom killer?

Quoted 2 lines

btw: Are RT tasks excluded by the oom killer?

I haven't looked. VM hackers?

Quoted 4 lines · Con Kolivas

Quoted 2 lines

btw: Are RT tasks excluded by the oom killer?

I haven't looked. VM hackers?

Nope. We're nastier to tasks which have been niced down, but we're not nicer to tasks which have been given elevated priority/policy.

Quoted 22 lines · utz lehmann

Quoted 10 lines · Con Kolivas

Quoted 9 lines · utz lehmann

Just an idea. What about throttling runaway RT tasks?
If the system spend more than 98% in RT tasks for 5s consider this as a _fatal error_. Print an error message and throttle RT tasks by inserting ticks where only SCHED_OTHER tasks allowed. For a limit of 98% this means one SCHED_OTHER only tick all 50 ticks.

The limit and timeout should be configurable and of course it can be disabled.

Quoted 8 lines

Quoted 3 lines

I know this is against RT task preempt all SCHED_OTHER but this is only for a fatal system state to be able to recover sanely. A locked up machine is is the worse alternative.

There is a patch in -mm currently designed to use a sysrq key combination which converts all real time tasks to sched normal to save you if you desire in a lockup situation. We do want to preserve RT scheduling behaviour at all times without caveats for privileged users.

Quoted 7 lines

The sysrq is already in 2.6.10. I had to use it the last days a few times. But it does help if you have no access to the console.

The RT throttling idea is not to change the behavior in normal conditions. It's only for a fatal system state. If you have a runaway RT task you can't guarantee the system is work properly anyway. It's blocking vital kernel threads, filesystems, swap, keyboard, ...

I understand fully your concern. If such a thing were to be introduced it would have to be disabled by default. Since I'm looking at implementing such throttling for user RT tasks, it should be trivial to add it to other RT tasks, and have 100% as the default cpu limit. How does that sound?

Quoted 4 lines · Lee Revell

This all seems to imply that introducing an rlimit for MAX_RT_PRIO is an excellent solution. The RT watchdog thread could run as root, and the rlimit would be used to ensure than even nonroot users in the RT group could never preempt the watchdog thread.

Just an idea. What about throttling runaway RT tasks?
If the system spend more than 98% in RT tasks for 5s consider this as a _fatal error_. Print an error message and throttle RT tasks by inserting ticks where only SCHED_OTHER tasks allowed. For a limit of 98% this means one SCHED_OTHER only tick all 50 ticks.

The limit and timeout should be configurable and of course it can be disabled.

SCHED_FIFO and SCHED_RR are definitely privileged operations and you

Quoted 1 line

SCHED_FIFO and SCHED_RR are definitely privileged operations and you

this is the crux of what this whole debate is about. for all of you people who think about linux on multi-user systems with network connectivity, running servers and so forth, this is clearly a given.

but there is large and growing body of machines that run linux where the sole human user of the machine has a strong and overwhelming desire to have tasks run with the characteristics offered by SCHED_FIFO and/or SCHED_RR. are they still "privileged" operations on this class of linux system? what about linux installed on an embedded system, with a small LCD screen and the sole purpose of running audio apps live? are they still privileged then?

i think there is room for debate, but its clear that in general, SCHED_FIFO/SCHED_RR's "definite" status as privileged operations is not clear. we are trying to find ways to provide access to it in ways that don't conflict with the other categories of linux systems where it clearly needs to be off-limits to unprivileged users.

Quoted 2 lines

SCHED_FIFO and SCHED_RR are definitely privileged operations and you

this is the crux of what this whole debate is about. for all of you people who think about linux on multi-user systems with network connectivity, running servers and so forth, this is clearly a given.

but there is large and growing body of machines that run linux where the sole human user of the machine has a strong and overwhelming desire to have tasks run with the characteristics offered by SCHED_FIFO and/or SCHED_RR. are they still "privileged" operations on this class of linux system? what about linux installed on an embedded system, with a small LCD screen and the sole purpose of running audio apps live? are they still privileged then?

Quoted 6 lines · Paul Davis

Quoted 1 line

SCHED_FIFO and SCHED_RR are definitely privileged operations and you

this is the crux of what this whole debate is about. for all of you people who think about linux on multi-user systems with network connectivity, running servers and so forth, this is clearly a given.

Quoted 8 lines

but there is large and growing body of machines that run linux where the sole human user of the machine has a strong and overwhelming desire to have tasks run with the characteristics offered by SCHED_FIFO and/or SCHED_RR. are they still "privileged" operations on this class of linux system? what about linux installed on an embedded system, with a small LCD screen and the sole purpose of running audio apps live? are they still privileged then?

Paul. Everyone agrees with you. I think. We just need to work out the best way of doing it.

Would I be right in suspecting that we know what to do, but nobody has stepped up to write the code? It's kinda looking like that?

Quoted 18 lines · Andrew Morton

Quoted 14 lines · Paul Davis

Quoted 1 line

SCHED_FIFO and SCHED_RR are definitely privileged operations and you

this is the crux of what this whole debate is about. for all of you people who think about linux on multi-user systems with network connectivity, running servers and so forth, this is clearly a given.

but there is large and growing body of machines that run linux where the sole human user of the machine has a strong and overwhelming desire to have tasks run with the characteristics offered by SCHED_FIFO and/or SCHED_RR. are they still "privileged" operations on this class of linux system? what about linux installed on an embedded system, with a small LCD screen and the sole purpose of running audio apps live? are they still privileged then?

Quoted 5 lines

Paul. Everyone agrees with you. I think. We just need to work out the best way of doing it.

Would I be right in suspecting that we know what to do, but nobody has stepped up to write the code? It's kinda looking like that?

I thought I made it clear i had already volunteered. I was after a response to my proposal for how to do it.

I wouldn't have thought it is so much a matter of having real-time-ish scheduling available that tries to play nicely in a multi user machine. That must still imply that either the user is able to unduly tie up resources (and thus it has to be a privileged operation), or that it sometimes can't meet its "guarantees" (in which case, is it useful?).

It sounds to me like both your proposals may be too complex and not sufficiently deterministic (I don't know for sure, maybe that's exactly what the RT people want).

I could be completely off the rails though. I haven't really been following this thread so please shoot me in my foot if I have put it in my mouth.

Quoted 3 lines · Nick Piggin

It sounds to me like both your proposals may be too complex and not sufficiently deterministic (I don't know for sure, maybe that's exactly what the RT people want).

This is the solution already employed in the real world by OSX. It works well, and the audio people have told me they are happy with it.

Quoted 3 lines

I could be completely off the rails though. I haven't really been following this thread so please shoot me in my foot if I have put it in my mouth.

If your foot is in your mouth and you ask me to shoot you in the foot it would blow your head off... Hmm it's tempting...

Alternatively, could you grant the required capabilities to use real RT scheduling and not foul up the scheduler?

Quoted 2 lines

Alternatively, could you grant the required capabilities to use real RT scheduling and not foul up the scheduler?

this is precisely the point i was making. either you agree that unprivileged users can get easy access to a scheduling class that can reliably DOS the system, or they can't. if they can't, what kind of scheduling class can they access easily?

according to andrew, and i agree with his conclusion, many people agree that its OK for them to get access to the DOS class, but there's little agreement on the security model to allow this. Con is suggesting that they are not, but instead get a different scheduling class that is functionally equivalent except that it can't (theoretically) be used to DOS the system.

It sounds to me like both your proposals may be too complex and not sufficiently deterministic (I don't know for sure, maybe that's exactly what the RT people want).

I wouldn't have thought it is so much a matter of having real-time-ish scheduling available that tries to play nicely in a multi user machine. That must still imply that either the user is able to unduly tie up resources (and thus it has to be a privileged operation), or that it sometimes can't meet its "guarantees" (in which case, is it useful?).

I was thinking that the solution might be more along the lines of a nice way to handle privileges for these guys.

Quoted 9 lines · Nick Piggin

It sounds to me like both your proposals may be too complex and not sufficiently deterministic (I don't know for sure, maybe that's exactly what the RT people want).

I wouldn't have thought it is so much a matter of having real-time-ish scheduling available that tries to play nicely in a multi user machine. That must still imply that either the user is able to unduly tie up resources (and thus it has to be a privileged operation), or that it sometimes can't meet its "guarantees" (in which case, is it useful?).

The VM system with overcommit is in a similar pickle. It can't honor the "guarantees" it makes. Yet, I think it is in wide use. Overcommit is a useful behavior for many people, despite the fact that it allows any user to turn loose the oom_killer on the system.

So I think many people would also find a best-effort-at-realtime SCHED_ISO type thing pretty useful, even if it allowed unprivileged users to tie up resources (while protecting the system from DOS). Heck, we don't have to allow unprivileged users to tie up resources. SCHED_ISO use could be limited to members of a certain group, possibly implemented using some sort of LSM module... :)

Of course, suggesting that access to SCHED_ISO be limited pretty much admits that running processes as SCHED_ISO should be a privileged operation, like accessing /dev/dsp (a privilege that is granted through group membership on most desktops).

Quoted 3 lines

I was thinking that the solution might be more along the lines of a nice way to handle privileges for these guys.

A nice, flexible way to hand out scheduler (and perhaps other) privileges would be... nice. Are you thinking of something more fine-grained than per-user?

Quoted 5 lines

Paul. Everyone agrees with you. I think. We just need to work out the best way of doing it.

Would I be right in suspecting that we know what to do, but nobody has stepped up to write the code? It's kinda looking like that?

I thought I made it clear i had already volunteered. I was after a response to my proposal for how to do it.

Quoted 7 lines

Quoted 5 lines

Paul. Everyone agrees with you. I think. We just need to work out the best way of doing it.

Would I be right in suspecting that we know what to do, but nobody has stepped up to write the code? It's kinda looking like that?

I thought I made it clear i had already volunteered. I was after a response to my proposal for how to do it.

I think your proposal is a good (maybe even excellent) one, but it somewhat sidesteps the issue (which may be the best thing to do). Rather than answering the question "how best to allow regular users access to SCHED_FIFO", it says "lets offer regular users SCHED_ISO which is essentially identical to SCHED_FIFO unless tasks running SCHED_ISO use too much cpu time".

its a fine answer, but its the answer to a slightly different question. if anyone (maybe us audio freaks, maybe someone else) comes up with a reason to want "The Real SCHED_FIFO", the original question will have gone unanswered.

Quoted 4 lines

its a fine answer, but its the answer to a slightly different question. if anyone (maybe us audio freaks, maybe someone else) comes up with a reason to want "The Real SCHED_FIFO", the original question will have gone unanswered.

Ah then you missed something. You can set the max cpu of SCHED_ISO to 100% and then you have it.

Quoted 6 lines

Quoted 4 lines

its a fine answer, but its the answer to a slightly different question. if anyone (maybe us audio freaks, maybe someone else) comes up with a reason to want "The Real SCHED_FIFO", the original question will have gone unanswered.

Ah then you missed something. You can set the max cpu of SCHED_ISO to 100% and then you have it.

true, i missed that :) but i also recall you saying you were thinking of having no prioritization within SCHED_ISO ... or am i remembering wrong?

also, is it just me, or having to ways to achieve the exact same result seems very un-linux-like ... and if they are not exact same results, how does a regular user get the SCHED_FIFO ones? is the answer just "they don't" ?

Quoted 9 lines · Paul Davis

Quoted 7 lines

Quoted 4 lines

its a fine answer, but its the answer to a slightly different question. if anyone (maybe us audio freaks, maybe someone else) comes up with a reason to want "The Real SCHED_FIFO", the original question will have gone unanswered.

Ah then you missed something. You can set the max cpu of SCHED_ISO to 100% and then you have it.

Quoted 3 lines

true, i missed that :) but i also recall you saying you were thinking of having no prioritization within SCHED_ISO ... or am i remembering wrong?

Nothing is set in stone. I wont even look at code until Ingo or Linus rules on this. Ingo has expressed interest in SCHED_ISO on a previous thread with me.

Quoted 4 lines

also, is it just me, or having to ways to achieve the exact same result seems very un-linux-like ... and if they are not exact same results, how does a regular user get the SCHED_FIFO ones? is the answer just "they don't" ?

To answer your question, the second of my proposals was to not have a separate scheduling class at all. To let normal users set SCHED_FIFO and SCHED_RR, possibly with all their priorities intact, but for there to be limits placed on their usage of these classes. The reason I suggested not supporting priorities is that proper real time scheduling would entail being able to say "I need x cycles, to complete by y time and I can or cannot be preempted". With these QoS requirements, a whole new scheduling style (EDF) would need to be implemented. Without actually

Quoted 4 lines · Paul Davis

its a fine answer, but its the answer to a slightly different question. if anyone (maybe us audio freaks, maybe someone else) comes up with a reason to want "The Real SCHED_FIFO", the original question will have gone unanswered.

Ah then you missed something. You can set the max cpu of SCHED_ISO to 100% and then you have it.

Quoted 7 lines · Paul Davis

Quoted 2 lines

SCHED_FIFO and SCHED_RR are definitely privileged operations and you

this is the crux of what this whole debate is about. for all of you people who think about linux on multi-user systems with network connectivity, running servers and so forth, this is clearly a given.

Quoted 8 lines

but there is large and growing body of machines that run linux where the sole human user of the machine has a strong and overwhelming desire to have tasks run with the characteristics offered by SCHED_FIFO and/or SCHED_RR. are they still "privileged" operations on this class of linux system? what about linux installed on an embedded system, with a small LCD screen and the sole purpose of running audio apps live? are they still privileged then?

Paul. Everyone agrees with you. I think. We just need to work out the best way of doing it.

Would I be right in suspecting that we know what to do, but nobody has stepped up to write the code? It's kinda looking like that?

The closest thing to concensus I've seen yet was a new rlimit for
scheduling with code from Chris Wright.

We really ought not get in
the habit of adding new rlimits though.

Quoted 3 lines · Matt Mackall

The closest thing to concensus I've seen yet was a new rlimit for
scheduling with code from Chris Wright.

hmm, yes. It doesn't feel like an rlimity thing to me, unless the rlimit actually _limits_ something. Say, minimum permissible nice level. But scheduling policy sounds more like a capability than an rlimit.

Quoted 2 lines

We really ought not get in
the habit of adding new rlimits though.

How come? It's a real pita that the standard shells don't appear to have a way of setting an unknown rlimit. But what else?

The closest thing to concensus I've seen yet was a new rlimit for scheduling with code from Chris Wright. The version I last saw had some rough edges on the API (exposing the internal scheduler priority levels) but wasn't too bad in principle. We really ought not get in the habit of adding new rlimits though.

Perhaps he can post whatever he has again, I'm not sure what the current state is.

Quoted 8 lines · * Matt Mackall ([email protected]) wrote:

The closest thing to concensus I've seen yet was a new rlimit for scheduling with code from Chris Wright. The version I last saw had some rough edges on the API (exposing the internal scheduler priority levels) but wasn't too bad in principle. We really ought not get in the habit of adding new rlimits though.

Perhaps he can post whatever he has again, I'm not sure what the current state is.

This is the latest version, with the idea from Utz to break nice and rtprio apart.

The basic issue on the rlimit value is how to sanely encode nice values, realtime prioroties and scheduler policies into a number. The first incarnation was the clumsiest, and tried to pack it all into a number in range of [0,139]. This, as many agree, too closely reflects kernel internal values. This one gives 0-39 (nice values 19,-20) to RLIMIT_NICE, and 0-99 (rt priorities) to RLIMIT_RTPRIO. There's no distinction in rt policy, and the traditional override (CAP_SYS_NICE) is still in place. The defaults for both rlimits are 0, and behaviour should be backwards compatible. I tested this one a bit, and it worked as expected. I've got a patch to pam_limits as well, although it's untested.

+#define NICE_TO_RLIMIT_NICE(nice) (19 - nice)

Quoted 10 lines · Chris Wright

The basic issue on the rlimit value is how to sanely encode nice values, realtime prioroties and scheduler policies into a number. The first incarnation was the clumsiest, and tried to pack it all into a number in range of [0,139]. This, as many agree, too closely reflects kernel internal values. This one gives 0-39 (nice values 19,-20) to RLIMIT_NICE, and 0-99 (rt priorities) to RLIMIT_RTPRIO. There's no distinction in rt policy, and the traditional override (CAP_SYS_NICE) is still in place. The defaults for both rlimits are 0, and behaviour should be backwards compatible. I tested this one a bit, and it worked as expected. I've got a patch to pam_limits as well, although it's untested.

This is looking pretty good.

Quoted 1 line

+#define NICE_TO_RLIMIT_NICE(nice) (19 - nice)

...

Review

Quoted 4 lines

(quoted diff) +4 -0

+unsigned long nice_to_rlimit_nice(const int nice)
+{
+	return NICE_TO_RLIMIT_NICE(nice);
+}

This is a bit silly.

Review

Quoted 5 lines

(quoted diff) +4 -1

-	if (niceval < task_nice(p) && !capable(CAP_SYS_NICE)) {
+	if (niceval < task_nice(p) &&
+		nice_to_rlimit_nice(niceval) >
+		p->signal->rlim[RLIMIT_NICE].rlim_cur &&
+		!capable(CAP_SYS_NICE)) {

Perhaps we want another helper function to do the rlim and CAP_SYS_NICE check together.

Quoted 2 lines

Perhaps we want another helper function to do the rlim and CAP_SYS_NICE check together.

Sure.
-chris

Quoted 2 lines · Matt Mackall

But I'm also still not convinced this policy can't be most flexibly handled by a setuid helper together with the mlock rlimit.

Quoting my message from a few days ago:

Quoted 4 lines · Matt Mackall

Why can't this be done with a simple SUID helper to promote given tasks to RT with sched_setschedule, doing essentially all the checks this LSM is doing?

Quoted 3 lines

Objections of "because it requires dangerous root or suid" don't fly, an RT app under user control can DoS the box trivially. Never mind you need root to configure the LSM anyway..

Yes but a bug in an app running as root can trash the filesystem. The worst you can do with RT privileges is lock up the machine.

Quoted 5 lines · Chris Wright

Guys, could we please bring this back to a useful discussion. None of you have commented on whether the rlimits for priority are useful. As I said before, I've no real problem with the module as it stands since it's tiny, quite contained, and does something people need. But I agree it'd be better to find something that's workable as long term solution.

I almost like it. I don't like that it exposes the internal scheduler priorities directly (-tiny in fact has options to change these!). So perhaps some thought could be given to either stratifying it a bit more (>2000 for SCHED_FIFO, >1000 for SCHED_RR, then SCHED_OTHER) or separate limits for the different scheduling disciplines.

But I'm also still not convinced this policy can't be most flexibly handled by a setuid helper together with the mlock rlimit.

Quoted 2 lines

But I'm also still not convinced this policy can't be most flexibly handled by a setuid helper together with the mlock rlimit.

Wait, why can't it be done with (to date fictitious) pam_prio, which simply calls sched_setscheduler? It's already privileged while it's doing these things...

Quoted 6 lines · Chris Wright

Quoted 2 lines

But I'm also still not convinced this policy can't be most flexibly handled by a setuid helper together with the mlock rlimit.

Wait, why can't it be done with (to date fictitious) pam_prio, which simply calls sched_setscheduler? It's already privileged while it's doing these things...

You certainly do not want to run everything at RT from login on. That'd be bad.

Also, tying to UIDs rather than (UID, executable) is worrisome as random_game_with_audio in Gnome might decide it needs RT, much to the admin's surprise.

Quoted 5 lines · * Matt Mackall ([email protected]) wrote:

On Tue, Jan 11, 2005 at 01:42:51PM -0800, Chris Wright wrote:

Also, tying to UIDs rather than (UID, executable) is worrisome as random_game_with_audio in Gnome might decide it needs RT, much to the admin's surprise.

Hmm, well, the pam_limit approach has that problem.

Quoted 8 lines · Chris Wright

Quoted 5 lines · * Matt Mackall ([email protected]) wrote:

On Tue, Jan 11, 2005 at 01:42:51PM -0800, Chris Wright wrote:

Also, tying to UIDs rather than (UID, executable) is worrisome as random_game_with_audio in Gnome might decide it needs RT, much to the admin's surprise.

Hmm, well, the pam_limit approach has that problem.

selinux?

Quoted 5 lines · Chris Wright

Guys, could we please bring this back to a useful discussion. None of you have commented on whether the rlimits for priority are useful. As I said before, I've no real problem with the module as it stands since it's tiny, quite contained, and does something people need. But I agree it'd be better to find something that's workable as long term solution.

I almost like it. I don't like that it exposes the internal scheduler priorities directly (-tiny in fact has options to change these!). So perhaps some thought could be given to either stratifying it a bit more (>2000 for SCHED_FIFO, >1000 for SCHED_RR, then SCHED_OTHER) or separate limits for the different scheduling disciplines.

Right now, you can make a good argument that SCHED_FIFO > SCHED_RR > SCHED_OTHER from a privilege point of view, but that could change if we add a pseudo-RT scheduling class of some sort. Similarly, adding a discipline means adding an rlimit with the split approach, so that's not very friendly either.

Another way:

0-20: normal nice values (inverted)

Quoted 1 line

20: privilege to set any RT priority

Limiting to below normal nice is a little weird and the offset to make everything positive is weird as well. Above 20, any RT app can starve SCHED_OTHER and it's less important to dole out fine-grained levels here as these apps must be engineered to cooperate to some degree anyway.

But I'm also still not convinced this policy can't be most flexibly handled by a setuid helper together with the mlock rlimit.

Quoted 2 lines

But I'm also still not convinced this policy can't be most flexibly handled by a setuid helper together with the mlock rlimit.

This has been explained several times already.

When you run a JACK client, the user should not be required to use a different command sequence depending on whether or not JACK is running with RT scheduling or not. That's almost more arcane than the current situation and is a step backwards from even 2.4, where we use capabilities to allow JACK itself to pass on the ability to use RT scheduling and memlock to its clients.

And that is a failure of imagination on the part of the JACK

developers. Simply add a library function to libjack or whatever:

jack_make_me_important(...); /* pretty please */

A client starts at normal priority, asks jack nicely to promote it to RT, then jackd, if so configured/enabled, calls the wrapper with a PID

Quoted 1 line

And that is a failure of imagination on the part of the JACK

Please be careful with your words. Based on your comments below, it appears that you've never read any of the technical docs on it, and almost certainly never read the source code.

Quoted 2 lines

A client starts at normal priority, asks jack nicely to promote it to RT, then jackd, if so configured/enabled, calls the wrapper with a PID

a PID? clients are multithreaded, and only specific threads run with RT scheduling (normally just the one created for them by libjack). So you presumably mean a TID, which in turn creates a problem for any system (e.g. 2.4) where all threads share the PID, and sched_setscheduler() really does use the PID as a PID, not a TID.

but its gets worse. JACK clients need to drop RT scheduling under certain, well-defined circumstances. how do they get it back under this scheme?

Quoted 3 lines · Paul Davis

Quoted 2 lines

A client starts at normal priority, asks jack nicely to promote it to RT, then jackd, if so configured/enabled, calls the wrapper with a PID

Quoted 5 lines

a PID? clients are multithreaded, and only specific threads run with RT scheduling (normally just the one created for them by libjack). So you presumably mean a TID, which in turn creates a problem for any system (e.g. 2.4) where all threads share the PID, and sched_setscheduler() really does use the PID as a PID, not a TID.

That actually sounds like an independent API problem.

Quoted 3 lines

but its gets worse. JACK clients need to drop RT scheduling under certain, well-defined circumstances. how do they get it back under this scheme?

Assuming a more thread-aware API, they just ask for privileges again. But with the non-thread-aware API, my first reaction would be the thread in question clones, and the clone drops privileges.

Quoted 6 lines · Matt Mackall

Thankfully a buffer underrun is no more fatal for pro audio than a broken guitar string. CDs skip, DATs glitch, XLR cables flake out, circuit breakers trip, amps clip, Powerbooks crash, and the show goes on. I've done more than enough stage tech to know it's a huge pain in the ass, but let's stop pretending we require absolute perfection, please.

In _practice_, Ingo's patches are considerably better than what you seem to consider "good enough for mere audio work".

Thankfully a buffer underrun is no more fatal for pro audio than a broken guitar string. CDs skip, DATs glitch, XLR cables flake out, circuit breakers trip, amps clip, Powerbooks crash, and the show goes on. I've done more than enough stage tech to know it's a huge pain in the ass, but let's stop pretending we require absolute perfection, please.

The numbers I reported yesterday were so bad I couldn't figure out why anyone even thought it was worth trying. Now I realize why.

When Ingo said to try "nice -20", I took him literally, forgetting that the stupid command to achieve a nice value of -20 is `nice --20'. So I was actually testing with a nice value of 19. Bah! No wonder it sucked.

Running `nice --20' is still significantly worse than SCHED_FIFO, but not the unmitigated disaster shown in the middle column. But, this improved performance is still not adequate for audio work. The worst delay was absurdly long (~1/2 sec).

Here are the corrected results...

                                 With -R        Without -R      Without -R
                               (SCHED_FIFO)     (nice -20)      (nice --20)

************* SUMMARY RESULT ****************

Total seconds ran . . . . . . :   300
Number of clients . . . . . . :    20
Ports per client  . . . . . . :     4
Frames per buffer . . . . . . :    64
*********************************************
Timeout Count . . . . . . . . :(    1)          (    1)          (    1)
XRUN Count  . . . . . . . . . :     2             2837               43
Delay Count (>spare time) . . :     0                0                0
Delay Count (>1000 usecs) . . :     0                0                0
Delay Maximum . . . . . . . . :  3130 usecs    5038044 usecs   501374 usecs
Cycle Maximum . . . . . . . . :   960 usecs      18802 usecs     1036 usecs
Average DSP Load. . . . . . . :    34.3 %           44.1 %         34.3 %    

Quoted 5 lines · Jack O'Quin

Here are the corrected results...

                                 With -R        Without -R      Without -R
                               (SCHED_FIFO)     (nice -20)      (nice --20)

Quoted 3 lines

XRUN Count  . . . . . . . . . :     2             2837               43
Delay Maximum . . . . . . . . :  3130 usecs    5038044 usecs   501374 usecs
Cycle Maximum . . . . . . . . :   960 usecs      18802 usecs     1036 usecs

what kind of non-audio workload was there during this test? 43 xruns arent nice but arent that bad either.

plus, is it 100% sure that all audio threads inherited the nice --20 priority - including the client threads? Nornally jackd does a setscheduler for the client threads so that they get boosted to SCHED_FIFO, but there is no parallel to that in the nice --20 case, did you do that manually (or did you start the clients up from the nice --20 shell too?))

If the nice --20 priority setup is perfect and there are still xruns then could you try the following hack, change this line in kernel/sched.c:

#define STARVATION_LIMIT (MAX_SLEEP_AVG)

to:

#define STARVATION_LIMIT 0

this will turn off starvation checking, for testing purposes. (to see whether there's anything else but anti-starvation causing xruns.)

what kind of non-audio workload was there during this test? 43 xruns arent nice but arent that bad either.

this will turn off starvation checking, for testing purposes. (to see whether there's anything else but anti-starvation causing xruns.)

Quoted 11 lines

sched.c renamed from sched.c~ +1 -1

@@ -95,7 +95,7 @@
 #define MAX_BONUS		(MAX_USER_PRIO * PRIO_BONUS_RATIO / 100)
 #define INTERACTIVE_DELTA	  2
 #define MAX_SLEEP_AVG		(DEF_TIMESLICE * MAX_BONUS)
-#define STARVATION_LIMIT	(MAX_SLEEP_AVG)
+#define STARVATION_LIMIT	0
 #define NS_MAX_SLEEP_AVG	(JIFFIES_TO_NS(MAX_SLEEP_AVG))
 #define CREDIT_LIMIT		100

could you try the patch below? The above patch wasnt enough. With the patch below we turn off the starvation limits for nice --20 tasks only. This is still a hack only. If we cannot make nice --20 perform like RT-prio-1 then there's some problem with SCHED_OTHER scheduling.

Quoted 4 lines · Ingo Molnar

could you try the patch below? The above patch wasnt enough. With the patch below we turn off the starvation limits for nice --20 tasks only. This is still a hack only. If we cannot make nice --20 perform like RT-prio-1 then there's some problem with SCHED_OTHER scheduling.

*** Terminated Sat Jan 15 18:15:13 CST 2005 ***
************* SUMMARY RESULT ****************

Total seconds ran . . . . . . :   300
Number of clients . . . . . . :    20
Ports per client  . . . . . . :     4
Frames per buffer . . . . . . :    64

*********************************************
Timeout Count . . . . . . . . :( 1)
XRUN Count . . . . . . . . . : 47
Delay Count (>spare time) . . : 0
Delay Count (>1000 usecs) . . : 0

Delay Maximum . . . . . . . . : 500544   usecs
Cycle Maximum . . . . . . . . :  1086   usecs
Average DSP Load. . . . . . . :    36.1 %
Average CPU System Load . . . :     8.2 %
Average CPU User Load . . . . :    26.3 %
Average CPU Nice Load . . . . :     0.0 %
Average CPU I/O Wait Load . . :     0.4 %
Average CPU IRQ Load  . . . . :     0.7 %
Average CPU Soft-IRQ Load . . :     0.0 %
Average Interrupt Rate  . . . :  1703.3 /sec
Average Context-Switch Rate . : 11600.6 /sec

*********************************************

I think this means the starvation test was not the problem. So far, I've seen no proof that there is any problem with the 2.6.10 scheduler, just some evidence that nice --20 does not work for multi-threaded realtime audio.

If someone can suggest a way to run certain threads of a process with a different nice value than the others, I can probably hack that into JACK in some crude way. That should tell us whether my intuition is right about the source of scheduling interference.

Otherwise, I'm out of ideas at the moment. I don't think SCHED_RR will be any different from SCHED_FIFO in this test. Even if it were, I'm not sure what that would prove.

Studying the test script, I discovered that it starts a separate program running in the background. So, I hacked the script to run it with nice -15 in order not to interfere with the realtime threads. The XRUNS didn't get much better, but the maximum delay went way down, from 1/2 sec to a much more believable (but still too high) 32.5 msec. I ran this with the same patched scheduler.

This supports my intuition that lack of per-thread granularity is the main problem. Where I was able to isolate some non-realtime code and run it at lower priority, it helped quite a bit.

So to make forward progress I'd like the audio people to confirm whether the mlock bits in 2.6.9+ do help that half of their requirement first

(and if not find a way to fix it) and then tackle the scheduling part. For that one I really wonder whether the combination of the now actually working nicelevels (see Mingo's post) and a simple wrapper for the really high requirements cases doesn't work.

It's not nessecarily a bad idea per, but it doesn't
really fit into the model we've been working to. I'd expect quite a few unpleasant devices when a user detects that the distibution had been binding various capabilities to uids/gids behinds his back.

For that one I really wonder whether the combination of the now actually working nicelevels (see Mingo's post)

Quoted 3 lines

Last I checked they could be controlled separately in that module. It has been suggested (by me and others) that one possible solution would be to expand it to be generic for all caps.

Maybe this is the way?

Quoted 3 lines · Lee Revell

Really, I think Linux has owned the server space for so long that some folks on this list are getting hubristic. Just because you have the best server OS does not mean it's the best at everything.

nah, the requirement is clearly valid, and longstanding. We need to satisfy it. It's just a matter of working out the best way.

Chris Wright <[email protected]> wrote:

Quoted 4 lines

...
Last I checked they could be controlled separately in that module. It has been suggested (by me and others) that one possible solution would be to expand it to be generic for all caps.

Maybe this is the way?

Lee, Jack and I have been very willing to discuss the issue. Christoph isn't willing to discuss it, he's just told us "its the wrong design, and I'm not telling you why or what's better". If there is a better design that will end up in the mainstream kernel, we'd love to see it implemented, and will likely be involved in doing it, because its really important to us.

Quoted 6 lines

Lee, Jack and I have been very willing to discuss the issue. Christoph isn't willing to discuss it, he's just told us "its the wrong design, and I'm not telling you why or what's better". If there is a better design that will end up in the mainstream kernel, we'd love to see it implemented, and will likely be involved in doing it, because its really important to us.

Calm down and read through the thread again.

Olaf:
-----
Capabilities don't work, because of missing filesystem capabilities. If you have them, it's a question of setting the appropriate permitted, inheritable and effective capability sets.

Sure, filesystem capabilities would be nice, but for the stuff Paul mentions they aren't needed -- what you need is to grant capabilities to the user's session, which can be easily done by a PAM module.

i think this is true only if the kernel comes with capabilities enabled.

various media-centric distributions (CCRMA, demudi, dyne:bolic and others) enabled them for their 2.4 kernels, but not the major desktop-centric ones. then the impression began to be received that in 2.6, capabilities were even more questionable of a mechanism to use. In addition, the LSM system appeared, and seemed to offer a much better solution entirely: no need to patch the kernel at all, or at least it appeared to be so in the beginning. Hence the "realtime" LSM.

Yes, but is there really some difference between people having to enable LSM and add a new LSM module, and people recompiling the kernel to include capabilities?

Also, is somebody really shipping 2.4 kernels without capabilities? I'm unable to find any such config switch in 2.4.28 -- maybe it's because I'm almost sleeping now, but it doesn't seem to be there.

They are present but disabled by default. You have to hack the initial values of CAP_INIT_EFF_SET and CAP_INIT_IHN_SET.

Hello!

Quoted 2 lines

They are present but disabled by default. You have to hack the initial values of CAP_INIT_EFF_SET and CAP_INIT_IHN_SET.

Oops. Does anybody know why this has been done?

Also, it seems that it has a relatively easy work-around: boot with init=/sbin/simple-wrapper and let the wrapper set the cap_bset and exec real init. (I agree that it's a hack, but a temporarily usable one.)

Yes, SETPCAP became a gaping security hole. Recall the sendmail hole.

This won't work, you can't increase the bset, which is hardcoded to leave out SETPCAP. Also, init is hard coded to start without SETPCAP.

Hello!

Quoted 1 line

Yes, SETPCAP became a gaping security hole. Recall the sendmail hole.

Hmmm, I don't remember now, could you give me some pointer, please?

Quoted 2 lines

This won't work, you can't increase the bset, which is hardcoded to leave out SETPCAP. Also, init is hard coded to start without SETPCAP.

If I read the source correctly, init is allowed to increase the bset, the other processes aren't.

Quoted 1 line

Yes, SETPCAP became a gaping security hole. Recall the sendmail hole.

Hmmm, I don't remember now, could you give me some pointer, please?

Quoted 4 lines · Jack O'Quin

The biggest problem was CAP_SETPCAP, which for good reasons[1] is disabled in distributed kernels. This forced every user to patch and build a custom kernel. Worse, it opened all our systems up to the problems reported by this sendmail security advisory.

Hello!

Quoted 10 lines

i think this is true only if the kernel comes with capabilities enabled.

various media-centric distributions (CCRMA, demudi, dyne:bolic and others) enabled them for their 2.4 kernels, but not the major desktop-centric ones. then the impression began to be received that in 2.6, capabilities were even more questionable of a mechanism to use. In addition, the LSM system appeared, and seemed to offer a much better solution entirely: no need to patch the kernel at all, or at least it appeared to be so in the beginning. Hence the "realtime" LSM.

Yes, but is there really some difference between people having to enable LSM and add a new LSM module, and people recompiling the kernel to include capabilities?

Quoted 3 lines

Yes, but is there really some difference between people having to enable LSM and add a new LSM module, and people recompiling the kernel to include capabilities?

For distributors, it's much easier to provide an additional module than to let people recompile kernels.

So, we have a few responses, some references to various potential solutions all of which have problems just as deep if not deeper than the uid/gid-based model that this particular LSM adopts. No proposal for any system that would actually work and address anyone's real needs in a useful way.