Reserve N process to root
Quoted 8 lines · David Newall
Quoted 6 lines · [email protected] wrote:
What David meant was that "root will always have a slot" doesn't *actually* help unless you *also* have a way to actually *spawn* such a process. In order to do the ps, kill, and so on that you need to recover, you need to already have either a root shell available, or a way to *get* a root shell that doesn't rely on a non-root process (so
/bin/sudoesn't help here).
Quoted 5 lines
That's right, although it's worse than that. You need to have a process with
CAP_SYS_ADMIN. If root processes normally have that capability then the reserved slots may well disappear before you notice a problem. If root processes normally don't have it, then you need to guarantee that one is already running.
I once posted a patch to handle this DoS, but, as usual, it wasn't accepted. Go figure...
Here is an excerpt:
Re: [PATCH 1/1] threads_max: Simple lockout prevention patch
From: Al Boldi <[email protected]>
To: Andrew Morton <[email protected]>
CC: [email protected]
Date: 04/24/06 02:12 pm
Quoted 5 lines · Andrew Morton
Quoted 2 lines · Al Boldi
This is a another resend, which was ignored before w/o comment. Andrew, can you at least comment on it? Thanks!
I don't have a clue what it's for.
Quoting from the 'Resource limits' thread on lkml on 27/09/05:
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 8 lines
Consider this dilemma:
Runaway proc/s hit the limit.
Try to kill some and you are denied due to the resource limit. Use some previously running app like top, hope it hasn't been killed by some OOM situation, try killing some procs and another one takes it's place because of the runaway situation.
Raise the limit, and it gets filled by the runaways.
You are pretty much stuck.Not really, this is the sort of thing ulimit is meant for. To keep processes from any one user from running away. It lets you limit the damage it can do, until such time as you can control it and fix the runaway application.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
threads-max = 1024
ulimit = 100 forks
11 runaway procs hitting the threads-max limitThis is incorrect. If you ulimit a user to 100 forks, and 11 processes running with that uid
Different uid.
Then yes, if you set a system-wide limit that is less than the sum of the limits imposed on each accountable part of the system you can have lock out. But thats your fault for misconfiguring the system. Don't do that.
-- end of quote
Thanks!
--
Al
Please don't trim CC lists
Quoted 7 lines · Al Boldi
Quoted 4 lines · David Newall
Quoted 1 line · [email protected] wrote:
What David meant was that "root will always have a slot" doesn't *actually* help unless you *also* have a way to actually *spawn* such a process. In order to do the ps, kill, and so on that you need to recover, you need to already have either a root shell available, or a way to *get* a root shell that doesn't rely on a non-root process (so
/bin/sudoesn't help here).That's right, although it's worse than that. You need to have a process with
CAP_SYS_ADMIN. If root processes normally have that capability then the reserved slots may well disappear before you notice a problem. If root processes normally don't have it, then you need to guarantee that one is already running.I once posted a patch to handle this DoS, but, as usual, it wasn't accepted. Go figure...
This isn't really necessary any more with the new CFS scheduler. If you want to prevent excess memory usage then you limit memory usage, not process count, so just set the system max process count to something absurdly high and leave the user counts down at the maximum a user might run. Then as long as the sum of the user processes is less than the max number of processes (which you just set absurdly high or unlimited), you may still log in. With the per-user scheduling enabled CFS allows you to run an optimistically-real-time game as one user and several thousand busy-loops as another user and get almost picture perfect 50% CPU distribution between the users. To me that seems a much better DoS-prevention system than limits which don't scale based on how many people are requesting resources.
Cheers,
Kyle Moffett
Quoted 21 lines · Kyle Moffett
Please don't trim CC lists
Quoted 9 lines · Al Boldi
Quoted 8 lines · David Newall
Quoted 6 lines · [email protected] wrote:
What David meant was that "root will always have a slot" doesn't *actually* help unless you *also* have a way to actually *spawn* such a process. In order to do the ps, kill, and so on that you need to recover, you need to already have either a root shell available, or a way to *get* a root shell that doesn't rely on a non-root process (so
/bin/sudoesn't help here).Quoted 8 lines
Quoted 5 lines
That's right, although it's worse than that. You need to have a process with
CAP_SYS_ADMIN. If root processes normally have that capability then the reserved slots may well disappear before you notice a problem. If root processes normally don't have it, then you need to guarantee that one is already running.I once posted a patch to handle this DoS, but, as usual, it wasn't accepted. Go figure...
Quoted 12 lines
This isn't really necessary any more with the new CFS scheduler. If you want to prevent excess memory usage then you limit memory usage, not process count, so just set the system max process count to something absurdly high and leave the user counts down at the maximum a user might run. Then as long as the sum of the user processes is less than the max number of processes (which you just set absurdly high or unlimited), you may still log in. With the per-user scheduling enabled CFS allows you to run an optimistically-real-time game as one user and several thousand busy-loops as another user and get almost picture perfect 50% CPU distribution between the users. To me that seems a much better DoS-prevention system than limits which don't scale based on how many people are requesting resources.
You have a point, and resource-controllers can probably control DoS a lot better, but the they also incur more overhead. Think of this "lockout prevention" patch as a near zero overhead safety valve.
Thanks!
--
Al
Quoted 4 lines · Al Boldi
Quoted 1 line · Kyle Moffett
This isn't really necessary any more with the new CFS scheduler. If you want to prevent excess memory usage then you limit memory usage, not process count, so just set the system max process count to something absurdly high and leave the user counts down at the maximum a user might run. Then as long as the sum of the user processes is less than the max number of processes (which you just set absurdly high or unlimited), you may still log in. With the per-user scheduling enabled CFS allows you to run an optimistically-real-time game as one user and several thousand busy-loops as another user and get almost picture perfect 50% CPU distribution between the users. To me that seems a much better DoS-prevention system than limits which don't scale based on how many people are requesting resources.
You have a point, and resource-controllers can probably control DoS a lot better, but the they also incur more overhead. Think of this "lockout prevention" patch as a near zero overhead safety valve.
But why do you need to add "lockout prevention" if it already exists? With CFS' extremely efficient per-user-scheduling (hopefully soon to be the default) there are only two forms of lockout by non-root processes: (1) Running out of PIDs in the box's PID-space (think tens or hundreds of thousands of processes), or (2) Swap-storming the box to death. To put it bluntly trying to reserve free PID slots is attacking the wrong end of the problem and your so called "lockout prevention" could very easily ensure that 10 PIDs are available even if the user has swapstormed the box with the PIDs he does have.
Cheers,
Kyle Moffett
Quoted 7 lines · Kyle Moffett
Quoted 3 lines · Al Boldi
You have a point, and resource-controllers can probably control DoS a lot better, but the they also incur more overhead. Think of this "lockout prevention" patch as a near zero overhead safety valve.
But why do you need to add "lockout prevention" if it already exists?
I said this before, but I'll say it again: it's about overhead!
Quoted 9 lines
With CFS' extremely efficient per-user-scheduling (hopefully soon to be the default) there are only two forms of lockout by non- root processes: (1) Running out of PIDs in the box's PID-space (think tens or hundreds of thousands of processes), or (2) Swap- storming the box to death. To put it bluntly trying to reserve free PID slots is attacking the wrong end of the problem and your so called "lockout prevention" could very easily ensure that 10 PIDs are available even if the user has swapstormed the box with the PIDs he does have.
I think you are reading this wrong. It's not about reserving PIDs, it's about exceeding the max-threads limit. This limit is global and affects every user including root, which is good, as this allows the sysadmin to fence the system into a controllable state. So once the system reaches the fence, sysadmin-intervention allows root to exceed the fence.
Again, this is much nicer with real resource-controllers, but again it's also more overhead.
Thanks!
--
Al
Quoted 29 lines · El Fri, 12 Oct 2007 09:29:10 +0300 Al Boldi <[email protected]> escribió:
Quoted 8 lines · Kyle Moffett
Quoted 4 lines · Al Boldi
You have a point, and resource-controllers can probably control DoS a lot better, but the they also incur more overhead. Think of this "lockout prevention" patch as a near zero overhead safety valve.
But why do you need to add "lockout prevention" if it already exists?
I said this before, but I'll say it again: it's about overhead!
Quoted 9 lines
With CFS' extremely efficient per-user-scheduling (hopefully soon to be the default) there are only two forms of lockout by non- root processes: (1) Running out of PIDs in the box's PID-space (think tens or hundreds of thousands of processes), or (2) Swap- storming the box to death. To put it bluntly trying to reserve free PID slots is attacking the wrong end of the problem and your so called "lockout prevention" could very easily ensure that 10 PIDs are available even if the user has swapstormed the box with the PIDs he does have.
I think you are reading this wrong. It's not about reserving PIDs, it's about exceeding the max-threads limit. This limit is global and affects every user including root, which is good, as this allows the sysadmin to fence the system into a controllable state. So once the system reaches the fence, sysadmin-intervention allows root to exceed the fence.
Quoted 2 lines
Again, this is much nicer with real resource-controllers, but again it's also more overhead.
Just an _if()_ ?
may be enable it as an option in kernel config ?
Quoted 11 lines
Thanks!
--
Al-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 4 lines
Gustavo Chaín Dumit Alumno de Ingeniería de Ejecución Informática Pontificia Universidad Católica de Valparaiso http://aleph.homeunix.com/~gchain
Quoted 30 lines · Gustavo Chain
Quoted 29 lines · Al Boldi
Quoted 8 lines · Kyle Moffett
Quoted 4 lines · Al Boldi
You have a point, and resource-controllers can probably control DoS a lot better, but the they also incur more overhead. Think of this "lockout prevention" patch as a near zero overhead safety valve.
But why do you need to add "lockout prevention" if it already exists?
I said this before, but I'll say it again: it's about overhead!
Quoted 9 lines
With CFS' extremely efficient per-user-scheduling (hopefully soon to be the default) there are only two forms of lockout by non- root processes: (1) Running out of PIDs in the box's PID-space (think tens or hundreds of thousands of processes), or (2) Swap- storming the box to death. To put it bluntly trying to reserve free PID slots is attacking the wrong end of the problem and your so called "lockout prevention" could very easily ensure that 10 PIDs are available even if the user has swapstormed the box with the PIDs he does have.
I think you are reading this wrong. It's not about reserving PIDs, it's about exceeding the max-threads limit. This limit is global and affects every user including root, which is good, as this allows the sysadmin to fence the system into a controllable state. So once the system reaches the fence, sysadmin-intervention allows root to exceed the fence.
Quoted 6 lines
Quoted 2 lines
Again, this is much nicer with real resource-controllers, but again it's also more overhead.
Just an _if()_ ?
may be enable it as an option in kernel config ?
Here is the patch again:
[PATCH 1/1] threads_max: Simple lockout prevention patch
Simple attempt to provide a backdoor in a process lockout situation.
echo $$ > /proc/sys/kernel/su-pid allows pid to exceed the threads_max limit.
Note that this patch incurs zero runtime-overhead.
---
(patch against 2.6.14)
fork.c 2005-11-14 20:58:25.000000000 +0300 renamed from fork.c.orig 2005-11-14 20:55:33.000000000 +0300 +2 -0
@@ -57,6 +57,7 @@ int nr_threads; /* The idle threads do not count.. */ int max_threads; /* tunable limit on nr_threads */ +int su_pid; /* BackDoor pid to exceed limit on nr_threads */ DEFINE_PER_CPU(unsigned long, process_counts) = 0;
@@ -926,6 +927,7 @@ * to stop root fork bombs. */ if (nr_threads >= max_threads) + if (p->pid != su_pid) goto bad_fork_cleanup_count; if (!try_module_get(p->thread_info->exec_domain->module))
sysctl.c 2005-11-14 21:01:20.000000000 +0300 renamed from sysctl.c.orig 2005-11-14 20:58:45.000000000 +0300 +9 -0
@@ -57,6 +57,7 @@ extern int sysctl_overcommit_memory; extern int sysctl_overcommit_ratio; extern int max_threads; +extern int su_pid; extern int sysrq_enabled; extern int core_uses_pid; extern int suid_dumpable;
@@ -509,6 +510,14 @@ .proc_handler = &proc_dointvec, }, { + .ctl_name = KERN_SU_PID, + .procname = "su-pid", + .data = &su_pid, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec, + }, + { .ctl_name = KERN_RANDOM, .procname = "random", .mode = 0555,
linux/sysctl.h 2005-11-14 20:55:15.000000000 +0300 renamed from linux/sysctl.h.orig 2005-11-14 20:54:55.000000000 +0300 +0 -0
@@ -146,6 +146,7 @@ KERN_RANDOMIZE=68, /* int: randomize virtual address space */ KERN_SETUID_DUMPABLE=69, /* int: behaviour of dumps for setuid core
*/
KERN_SPIN_RETRY=70, /* int: number of spinlock retries */ + KERN_SU_PID=71, /* int: BackDoor pid to exceed Maximum + /* nr of threads in the system */
};