Where is RLIMIT_RT_CPU?
Hi,
I saw many references to RLIMIT_RT_CPU (e.g. in
http://lwn.net/Articles/120797/ ) as a way of limiting the amount of CPU an unprivileged task can use at real-time priority. My understand was that the feature had been merged into 2.6.12 as part of Ingo's rt-limit patches. Unfortunately, I can't find any reference to that on my system (latest Ubuntu), so I'm wondering where it's gone. Has it been removed, renamed, ...? Considering that Ubuntu Dapper currently allows any user to make unlimited use of realtime scheduling, this feature would be really useful to prevent user apps from accidently crashing the system.
Thanks,
Jean-Marc
P.S. Please CC me since I'm not subscribed to the list.
Quoted 11 lines · Jean-Marc Valin
Hi,
I saw many references to
RLIMIT_RT_CPU(e.g. in
http://lwn.net/Articles/120797/ ) as a way of limiting the amount of CPU an unprivileged task can use at real-time priority. My understand was that the feature had been merged into 2.6.12 as part of Ingo's rt-limit patches. Unfortunately, I can't find any reference to that on my system (latest Ubuntu), so I'm wondering where it's gone. Has it been removed, renamed, ...? Considering that Ubuntu Dapper currently allows any user to make unlimited use of realtime scheduling, this feature would be really useful to prevent user apps from accidently crashing the system.
It was not merged.
This problem should be addressed by a userspace RT watchdog. Ubuntu should not have shipped their system with unlimited non-root realtime enabled and no watchdog.
Lee
Quoted 5 lines
It was not merged.
This problem should be addressed by a userspace RT watchdog. Ubuntu should not have shipped their system with unlimited non-root realtime enabled and no watchdog.
Does such a daemon currently exist? Even then, is there any way to police all unprivileged RT apps without introducing an O(N) operation running continuously at max rt priority? Sort of defeats the purpose of having an O(1) scheduler, no? I assume the RT check can be some in O(1) time in the kernel, right?
About Ubuntu, I agree it wasn't a smart thing to do (I'm not an Ubuntu devel, so I don't know what the reason was), but it would be nice if this could be fixed without having to entirely remove the unprivileged real-time feature. In the end, I'm not sure what the best solution is (just an RT audio developer).
Jean-Marc
Quoted 6 lines · Jean-Marc Valin
Quoted 5 lines
It was not merged.
This problem should be addressed by a userspace RT watchdog. Ubuntu should not have shipped their system with unlimited non-root realtime enabled and no watchdog.
Quoted 6 lines
Does such a daemon currently exist? Even then, is there any way to police all unprivileged RT apps without introducing an O(N) operation running continuously at max rt priority? Sort of defeats the purpose of having an O(1) scheduler, no? I assume the RT check can be some in O(1) time in the kernel, right?
Can't you just make a prio 1 task which signals a prio 99 once say every second. If the priority 99 task doesn't get the signal after say 2 seconds, it will look for a rt task running wild. At worst it will have to do an O(n) algorith when things have gone wrong, not when everything is working.
Esben
Quoted 13 lines
About Ubuntu, I agree it wasn't a smart thing to do (I'm not an Ubuntu devel, so I don't know what the reason was), but it would be nice if this could be fixed without having to entirely remove the unprivileged real-time feature. In the end, I'm not sure what the best solution is (just an RT audio developer).
Jean-Marc
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines
Can't you just make a prio 1 task which signals a prio 99 once say every second. If the priority 99 task doesn't get the signal after say 2 seconds, it will look for a rt task running wild. At worst it will have to do an O(n) algorith when things have gone wrong, not when everything is working.
Well, that would work in sort of preventing a complete lockup, but the watchdog wouldn't even know if the task eating lots of CPU is privileged (OK) or unprivileged (not OK). Also, the original RLIMIT_RT_CPU feature allowed you to really control how much CPU is available to unprivileged users, not just prevent them from getting 100% CPU.
Jean-Marc
Quoted 11 lines · Jean-Marc Valin
Quoted 5 lines
Can't you just make a prio 1 task which signals a prio 99 once say every second. If the priority 99 task doesn't get the signal after say 2 seconds, it will look for a rt task running wild. At worst it will have to do an O(n) algorith when things have gone wrong, not when everything is working.
Well, that would work in sort of preventing a complete lockup, but the watchdog wouldn't even know if the task eating lots of CPU is privileged (OK) or unprivileged (not OK). Also, the original
RLIMIT_RT_CPUfeature allowed you to really control how much CPU is available to unprivileged users, not just prevent them from getting 100% CPU.
Non-root RT tasks are not "unprivileged" - they have a level of privileges between a normal user and root. Really I think it's OK for these tasks to consume 100% CPU, as the admin has explicitly allowed it.
The only problem is that Ubuntu shipped with this enabled for everyone.
Lee
Quoted 3 lines
Non-root RT tasks are not "unprivileged" - they have a level of privileges between a normal user and root. Really I think it's OK for these tasks to consume 100% CPU, as the admin has explicitly allowed it.
Sure, if the admin has allowed it, but I think it's also OK for desktop users to have access to rt scheduling by default, as long as their tasks are "well-behaved" (i.e. only use a few percent CPU).
Quoted 1 line
The only problem is that Ubuntu shipped with this enabled for everyone.
The problem is not the fact that it's enabled, but the fact that the amount of CPU allowed isn't restricted.
Jean-Marc
Quoted 4 lines · Jean-Marc Valin
Quoted 3 lines
Non-root RT tasks are not "unprivileged" - they have a level of privileges between a normal user and root. Really I think it's OK for these tasks to consume 100% CPU, as the admin has explicitly allowed it.
Quoted 8 lines
Sure, if the admin has allowed it, but I think it's also OK for desktop users to have access to rt scheduling by default, as long as their tasks are "well-behaved" (i.e. only use a few percent CPU).
Quoted 1 line
The only problem is that Ubuntu shipped with this enabled for everyone.
The problem is not the fact that it's enabled, but the fact that the amount of CPU allowed isn't restricted.
as long as you can fork and exec as many of those processes as you want a per process rlimit is useless security wise... an evil user just fires off a second process just before the first one gets killed and a non-RT root still is starved out.
Quoted 4 lines
as long as you can fork and exec as many of those processes as you want a per process rlimit is useless security wise... an evil user just fires off a second process just before the first one gets killed and a non-RT root still is starved out.
Of course, which is why the idea is for the limit to be global, across all non-root users. AFAIK, that's what Ingo's original (pre-2.6.12) patch did and also what Con Kolivas' SCHED_ISO patch does. That's also why I think it would be very hard (if possible at all) to do this in user space.
Jean-Marc
Quoted 10 lines · Jean-Marc Valin
Quoted 4 lines
as long as you can fork and exec as many of those processes as you want a per process rlimit is useless security wise... an evil user just fires off a second process just before the first one gets killed and a non-RT root still is starved out.
Of course, which is why the idea is for the limit to be global, across all non-root users. AFAIK, that's what Ingo's original (pre-2.6.12) patch did and also what Con Kolivas'
SCHED_ISOpatch does. That's also why I think it would be very hard (if possible at all) to do this in user space.
I don't think it's a problem. If the admin does not want non-root users to be able to lock up the machine, just don't put them in the realtime group.
Lee
Quoted 3 lines
I don't think it's a problem. If the admin does not want non-root users to be able to lock up the machine, just don't put them in the realtime group.
What if the admin *wants* non-root users to have good quality audio, and just doesn't want them to crash the system (voluntarily and especially accidentally). Enforcing CPU limits *is* possible and it has already been done independently by both Ingo and Con. I'm just waiting for the feature to be available out-of-the box, which is not for today if kernel space keeps pointing at userspace and vice versa. :-(
Jean-Marc
Quoted 4 lines · Jean-Marc Valin
Quoted 3 lines
I don't think it's a problem. If the admin does not want non-root users to be able to lock up the machine, just don't put them in the realtime group.
Quoted 9 lines
What if the admin *wants* non-root users to have good quality audio, and just doesn't want them to crash the system (voluntarily and especially accidentally). Enforcing CPU limits *is* possible and it has already been done independently by both Ingo and Con. I'm just waiting for the feature to be available out-of-the box, which is not for today if kernel space keeps pointing at userspace and vice versa. :-(
Jean-Marc
You can't have "random" users scheduling thing at real-time priorities.
A real-time system can only work if it is set up as whole and all real-time tasks are taken into consideration. If you allow a user to start another real-time task, that task might destroy the real-time properties of all the rest by taking too much cpu.
As I see it the only thing you can do is to use sudo to run anything,
which needs real-time priority, with higher priviliges, than what a normal user have. Then he can only start specific audio programs and can't crash the system (unless those programs have a bug).
Esben
Quoted 10 lines
You can't have "random" users scheduling thing at real-time priorities. A real-time system can only work if it is set up as whole and all real-time tasks are taken into consideration. If you allow a user to start another real-time task, that task might destroy the real-time properties of all the rest by taking too much cpu.
As I see it the only thing you can do is to use sudo to run anything, which needs real-time priority, with higher priviliges, than what a normal user have. Then he can only start specific audio programs and can't crash the system (unless those programs have a bug).
I though we were past that point a long time ago. Of course you don't want *any* random user to have access to rt scheduling. However, you *do* want the user logged-in on the console be allowed to run tasks with rt priority (within some limits). Why? Because 1) you don't want to give root access to any user who needs RT for some apps and 2) you don't want to make all these apps suid root either.
Think about it, would you really feel safe with your sound daemon, Ekiga/Gnomemeeting, jackd, Amarok, mplayer, ... all being run setuid root. Yet all these apps (and many others) can benefit from being allowed a few percent CPU running at rt priority. Not to mention the fact that you may actually want to do rt *development* as a regular user. That's why people have come up with realtime-lsm, and more recently, with SCHED_ISO and rt-limit. What's currently missing is just the tiny bit that prevents the user from locking up the machine. Even that one was done by Ingo, but unfortunately not merged in the mainline kernel. Controlled properly, access to rt scheduling is no more dangerous (and probably less) than fork(), malloc() or mlock(), all of which can lock up a machine efficiently if unlimited use is allowed.
Jean-Marc
Quoted 11 lines · Jean-Marc Valin
Quoted 10 lines
You can't have "random" users scheduling thing at real-time priorities. A real-time system can only work if it is set up as whole and all real-time tasks are taken into consideration. If you allow a user to start another real-time task, that task might destroy the real-time properties of all the rest by taking too much cpu.
As I see it the only thing you can do is to use sudo to run anything, which needs real-time priority, with higher priviliges, than what a normal user have. Then he can only start specific audio programs and can't crash the system (unless those programs have a bug).
Quoted 7 lines
I though we were past that point a long time ago. Of course you don't want *any* random user to have access to rt scheduling. However, you *do* want the user logged-in on the console be allowed to run tasks with rt priority (within some limits). Why? Because 1) you don't want to give root access to any user who needs RT for some apps and 2) you don't want to make all these apps suid root either.
Who said suid root? What about suid realtime or suid audio?
Quoted 4 lines
Think about it, would you really feel safe with your sound daemon, Ekiga/Gnomemeeting, jackd, Amarok, mplayer, ... all being run setuid root. Yet all these apps (and many others) can benefit from being allowed a few percent CPU running at rt priority. Not to mention the
Quoted 8 lines
fact that you may actually want to do rt *development* as a regular user. That's why people have come up with realtime-lsm, and more recently, with
SCHED_ISOand rt-limit. What's currently missing is just the tiny bit that prevents the user from locking up the machine. Even that one was done by Ingo, but unfortunately not merged in the mainline kernel. Controlled properly, access to rt scheduling is no more dangerous (and probably less) thanfork(),malloc()ormlock(), all of which can lock up a machine efficiently if unlimited use is allowed.
My point is not that they can lock up the machine. My point is that you just can't add a rt task to a rt system! A rt-system consists of a fixed set of threads, which in worst case can meet it's deadlines. If you add just one task you might break the whole system. Only someone with overview of the whole system can add those tasks.
It is very simple if you have only a audio application or only a driver needing low latencies. What if you have both? You have to make sure that the higher priority one leaves enough cpu for the lower priority one. And it is not a question of using a low percentage of cpu. It is a question of how long the cpu is used in one go and how often it can happen within the critical timeframe of the lower priority application.
You can make a system which checks that but it is much harder to do than a moving average. The only thing which makes sense is (a) square filter(s) with a width equal to the required latency of the lower priority task(s).
So the sys-admin should somehow be able to give the right either to specific (audio) applications with specific priorities or a developer whom he trusts (and it does not make sense to give it to both as they would then mess it up for each other!)
We have discussed that total lock-up can be prevented with a simple watchdog. That solution doesn't need anything added into the scheduler.
Esben
Quoted 3 lines
Jean-Marc
Quoted 1 line
Who said suid root? What about suid realtime or suid audio?
Still, I don't see what you gain by making them setuid $group vs allowing member(s) of $group to use rt scheduling.
Quoted 5 lines
My point is not that they can lock up the machine. My point is that you just can't add a rt task to a rt system! A rt-system consists of a fixed set of threads, which in worst case can meet it's deadlines. If you add just one task you might break the whole system. Only someone with overview of the whole system can add those tasks.
This issue can also happen if you start 10 suid rt apps. It's the responsibility of the user to make sure there's no bad interaction. The reason we want a limit is to make sure the system remains relatively responsive (not just up). In the ideal case (not sure it's possible), the scheduler would make sure that non-root rt apps wouldn't get (on average) more CPU than they would get running at normal priority. i.e. if there are 3 tasks competing and you start a rt task, then that task couldn't get more than 25% CPU. Of course, it may not be possible to do that perfectly, but anything remotely close would be pretty good.
In any case, most audio (and probably other) applications tend to only require a very small amount (<10%) of CPU to run properly. I'd be quite happy is my system was configured to allow me to run rt tasks as long as the total doesn't exceed 30% CPU.
Quoted 6 lines
It is very simple if you have only a audio application or only a driver needing low latencies. What if you have both? You have to make sure that the higher priority one leaves enough cpu for the lower priority one. And it is not a question of using a low percentage of cpu. It is a question of how long the cpu is used in one go and how often it can happen within the critical timeframe of the lower priority application.
I understand that, but adding artificial restrictions (to setuid audio apps) isn't going to help.
Quoted 3 lines
You can make a system which checks that but it is much harder to do than a moving average. The only thing which makes sense is (a) square filter(s) with a width equal to the required latency of the lower priority task(s).
Why not? It could be nice as well if someone wants to implement that. I'd already be quite happy to just have basic control on the CPU time.
Quoted 4 lines
So the sys-admin should somehow be able to give the right either to specific (audio) applications with specific priorities or a developer whom he trusts (and it does not make sense to give it to both as they would then mess it up for each other!)
How about simply giving rt rights to whomever is logged on the console? That's the user that really counts since he's (usually) next to the audio device.
Quoted 2 lines
We have discussed that total lock-up can be prevented with a simple watchdog. That solution doesn't need anything added into the scheduler.
As I mentioned earlier, it's not about total lock-up, but having things run relatively smoothly and (if possible?) even fairly.
Jean-Marc
Quoted 4 lines · Jean-Marc Valin
Why not? It could be nice as well if someone wants to implement that. I'd already be quite happy to just have basic control on the CPU time.
Have you thought about using something like Xen? Have a virtual machine that you can even give root access to users, and still have control of the actual physical machine.
Quoted 2 lines
As I mentioned earlier, it's not about total lock-up, but having things run relatively smoothly and (if possible?) even fairly.
One issue I think you might have is what exactly is a CPU limit? If the system is idle, and you have an app that goes into a busy loop, do you kill it after it hits the limit, even if it isn't RT? Or do you just force it to schedule? Or do you consider idle a special case? Do you want just the apps to be limited, or all the apps that belong to a specific user.
Quoted 1 line
From this thread, it seems your goal is to have a single console that
users can log into and run a RT thread for audio but still not be able to lock up the entire system. Right? So having an RT limit for this use might actually be beneficial. But this is a very rare case, and if you are the only one needing this type of feature, then it will likely not make it into the kernel. But it if turns out that lots of people like this feature, and want it, then it might have a chance, if there is no other way to accomplish it.
Currently, it looks like you can use either Xen or just stick to one of the patches you mentioned earlier.
-- Steve
Quoted 3 lines
Have you thought about using something like Xen? Have a virtual machine that you can even give root access to users, and still have control of the actual physical machine.
What does Xen have to do with that? I don't want to run stuff in another virtual machine, just on my desktop. Also, no matter how good the virtualization is, if the host eats CPU, the guest doesn't have it.
Quoted 6 lines
One issue I think you might have is what exactly is a CPU limit? If the system is idle, and you have an app that goes into a busy loop, do you kill it after it hits the limit, even if it isn't RT? Or do you just force it to schedule? Or do you consider idle a special case? Do you want just the apps to be limited, or all the apps that belong to a specific user.
The standard was is to simply demote ill-behaved tasks to SCHED_OTHER or suspend them temporarily, i.e. limit the percentage of CPU they can get over a certain period of time.
Quoted 6 lines
Quoted 1 line
From this thread, it seems your goal is to have a single console that
users can log into and run a RT thread for audio but still not be able to lock up the entire system. Right? So having an RT limit for this use might actually be beneficial. But this is a very rare case, and if you are the only one needing this type of feature, then it will likely not make it into the kernel.
Rare case? So you never use a VoIP app (sure they can work as non-rt, but you get much better quality as latency goes down)? How about media players that don't skip? There are lots of uses for low-latency (i.e. rt scheduling) on a desktop. Pro audio people are the first to benefit, but all users can see an improvement.
Quoted 3 lines
But it if turns out that lots of people like
this feature, and want it, then it might have a chance, if there is no other way to accomplish it.
There *are* lots of people who want it and who have been complaining for quite a while (see threads on linux-audio-dev). All that's missing is a tiny bit, which makes it even more frustrating.
Quoted 2 lines
Currently, it looks like you can use either Xen or just stick to one of the patches you mentioned earlier.
Sure, I can patch my kernel, but then the distros will just keep on ignoring the problem, which is bad.
Jean-Marc
Quoted 7 lines · Jean-Marc Valin
Quoted 3 lines
Have you thought about using something like Xen? Have a virtual machine that you can even give root access to users, and still have control of the actual physical machine.
What does Xen have to do with that? I don't want to run stuff in another virtual machine, just on my desktop. Also, no matter how good the virtualization is, if the host eats CPU, the guest doesn't have it.
What exactly is it that you want? Limit an RT process, but not limit it???
Quoted 19 lines
Quoted 7 lines
One issue I think you might have is what exactly is a CPU limit? If the system is idle, and you have an app that goes into a busy loop, do you kill it after it hits the limit, even if it isn't RT? Or do you just force it to schedule? Or do you consider idle a special case? Do you want just the apps to be limited, or all the apps that belong to a specific user.
The standard was is to simply demote ill-behaved tasks to
SCHED_OTHERor suspend them temporarily, i.e. limit the percentage of CPU they can get over a certain period of time.Quoted 6 lines
Quoted 1 line
From this thread, it seems your goal is to have a single console that
users can log into and run a RT thread for audio but still not be able to lock up the entire system. Right? So having an RT limit for this use might actually be beneficial. But this is a very rare case, and if you are the only one needing this type of feature, then it will likely not make it into the kernel.
Quoted 5 lines
Rare case? So you never use a VoIP app (sure they can work as non-rt, but you get much better quality as latency goes down)? How about media players that don't skip? There are lots of uses for low-latency (i.e. rt scheduling) on a desktop. Pro audio people are the first to benefit, but all users can see an improvement.
If I need something that requires RT I simply set it to be RT. That is not what you are asking. You want something to be RT but limited. This _is_ a rare case AFAIC.
Quoted 8 lines
Quoted 4 lines
But it if turns out that lots of people like
this feature, and want it, then it might have a chance, if there is no other way to accomplish it.There *are* lots of people who want it and who have been complaining for quite a while (see threads on linux-audio-dev). All that's missing is a tiny bit, which makes it even more frustrating.
People want their RT apps to have a CPU limit?
Quoted 6 lines
Quoted 3 lines
Currently, it looks like you can use either Xen or just stick to one of the patches you mentioned earlier.
Sure, I can patch my kernel, but then the distros will just keep on ignoring the problem, which is bad.
It could just be me. I'm not into audio, and the RT apps that I use are in a controlled environment. So please educate me and tell me what exactly is it that you want and why? And not just a buggy RT app that takes down the system. That is expected behavior. And it is possible to control that in user space.
-- Steve
Quoted 52 lines · Jean-Marc Valin
Quoted 1 line
Who said suid root? What about suid realtime or suid audio?
Still, I don't see what you gain by making them setuid $group vs allowing member(s) of $group to use rt scheduling.
Quoted 5 lines
My point is not that they can lock up the machine. My point is that you just can't add a rt task to a rt system! A rt-system consists of a fixed set of threads, which in worst case can meet it's deadlines. If you add just one task you might break the whole system. Only someone with overview of the whole system can add those tasks.
This issue can also happen if you start 10 suid rt apps. It's the responsibility of the user to make sure there's no bad interaction. The reason we want a limit is to make sure the system remains relatively responsive (not just up). In the ideal case (not sure it's possible), the scheduler would make sure that non-root rt apps wouldn't get (on average) more CPU than they would get running at normal priority. i.e. if there are 3 tasks competing and you start a rt task, then that task couldn't get more than 25% CPU. Of course, it may not be possible to do that perfectly, but anything remotely close would be pretty good.
In any case, most audio (and probably other) applications tend to only require a very small amount (<10%) of CPU to run properly. I'd be quite happy is my system was configured to allow me to run rt tasks as long as the total doesn't exceed 30% CPU.
Quoted 6 lines
It is very simple if you have only a audio application or only a driver needing low latencies. What if you have both? You have to make sure that the higher priority one leaves enough cpu for the lower priority one. And it is not a question of using a low percentage of cpu. It is a question of how long the cpu is used in one go and how often it can happen within the critical timeframe of the lower priority application.
I understand that, but adding artificial restrictions (to setuid audio apps) isn't going to help.
Quoted 3 lines
You can make a system which checks that but it is much harder to do than a moving average. The only thing which makes sense is (a) square filter(s) with a width equal to the required latency of the lower priority task(s).
Why not? It could be nice as well if someone wants to implement that. I'd already be quite happy to just have basic control on the CPU time.
Quoted 4 lines
So the sys-admin should somehow be able to give the right either to specific (audio) applications with specific priorities or a developer whom he trusts (and it does not make sense to give it to both as they would then mess it up for each other!)
How about simply giving rt rights to whomever is logged on the console? That's the user that really counts since he's (usually) next to the audio device.
Quoted 8 lines
Quoted 2 lines
We have discussed that total lock-up can be prevented with a simple watchdog. That solution doesn't need anything added into the scheduler.
As I mentioned earlier, it's not about total lock-up, but having things run relatively smoothly and (if possible?) even fairly.
Jean-Marc
The more I think about it the more it looks like you want a nice -40 task, not a real-time task. The basic behaviour you want is a task which can wake up and get the cpu almost immediately, but if it starts to use too much CPU it looses that privilige. That is you want moving priorities just as a normal SCHED_OTHER tasks. The only difference is that you want it to start further up the scale than usual so it takes much longer before the scheduler has moved it down to a priority, where it has to compete with other tasks. Maybe it is already enough to run your audio application at nice -20 ?
When you are not developing for a specific CPU and/or the your application isn't the only RT application on the system, you can't use real-time
scheduling anyway, it simply doesn't make much sense. Using a very highSCHED_OTHER priority would probably work much better. Leave the RT talk for the controlled embedded applications, where the developers (are supposed to) know and test the whole system.
I also think that Joe user who is locked into the console should only be able to start specific applications with extraordinary priorities. That is, it should not be enough to look at his permissions, but look at the permissions of the program he wants to run. Therefore a suid system is the right thing to make.
Esben