Skip to contentSkip to conversation

Resource limits

DISCUSSION Al Boldi 8 participants 22 messages Sep 25, 2005 – Sep 27, 2005

lore
Share & tools
lore

Al Boldi

author
#0 lore raw

Resource limits in Linux, when available, are currently very limited.

i.e.:
Too many process forks and your system may crash.
This can be capped with threads-max, but may lead you into a lock-out.

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

Would this be difficult to implement?

Thanks!

--
Al

#1 lore raw
Quoted 10 lines · Al Boldi

Resource limits in Linux, when available, are currently very limited.

i.e.:
Too many process forks and your system may crash.
This can be capped with threads-max, but may lead you into a lock-out.

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

Would this be difficult to implement?

How would you reclaim the resource after that limited time is over ? Kill processes?

Signature · 1 line
All Rights Reversed
author
#2 lore raw
Quoted 7 lines · Rik van Riel
Quoted 6 lines · Al Boldi

Resource limits in Linux, when available, are currently very limited.

i.e.:
Too many process forks and your system may crash.
This can be capped with threads-max, but may lead you into a lock-out.

Quoted 8 lines
Quoted 5 lines

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

Would this be difficult to implement?

How would you reclaim the resource after that limited time is over ? Kill processes?

That's one way, but really, the issue needs some deep thought. Leaving Linux exposed to a lock-out is rather frightening.

Quoted 2 lines · Neil Horman

Whats insufficient about the per-user limits that can be imposed by the ulimit syscall?

Are they system wide or per-user?

--
Al

#3 lore raw
Quoted 12 lines · Al Boldi
Quoted 11 lines · Rik van Riel
Quoted 10 lines · Al Boldi

Resource limits in Linux, when available, are currently very limited.

i.e.:
Too many process forks and your system may crash.
This can be capped with threads-max, but may lead you into a lock-out.

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Would this be difficult to implement?

How would you reclaim the resource after that limited time is over ? Kill processes?

That's one way, but really, the issue needs some deep thought. Leaving Linux exposed to a lock-out is rather frightening.

What exactly is it that you're worried about here? Do you have a particular concern that a process won't be able to fork or create a thread? Resources that can be allocated to user space processes always run the risk that their allocation will not succede. Its up to the application to deal with that.

Quoted 6 lines
Quoted 2 lines · Neil Horman

Whats insufficient about the per-user limits that can be imposed by the ulimit syscall?

Are they system wide or per-user?

ulimits are per-user.
Neil

Quoted 8 lines

--
Al

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 5 lines
/***************************************************
 *Neil Horman
 *Software Engineer
 *gpg keyid: 1024D / 0x92A74FA1 - http://pgp.mit.edu
 ***************************************************/
author
#4 lore raw
Quoted 20 lines · Neil Horman
Quoted 19 lines · Al Boldi
Quoted 17 lines · Rik van Riel
Quoted 8 lines · Al Boldi

Resource limits in Linux, when available, are currently very limited.

i.e.:
Too many process forks and your system may crash. This can be capped with threads-max, but may lead you into a lock-out.

Quoted 5 lines

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

Would this be difficult to implement?

How would you reclaim the resource after that limited time is over ? Kill processes?

Quoted 8 lines
Quoted 2 lines

That's one way, but really, the issue needs some deep thought. Leaving Linux exposed to a lock-out is rather frightening.

What exactly is it that you're worried about here? Do you have a particular concern that a process won't be able to fork or create a thread? Resources that can be allocated to user space processes always run the risk that their allocation will not succede. Its up to the application to deal with that.

Think about a DoS attack.

Thanks!

--
Al

#5 lore raw
Quoted 30 lines · Al Boldi
Quoted 20 lines · Neil Horman
Quoted 19 lines · Al Boldi
Quoted 17 lines · Rik van Riel
Quoted 8 lines · Al Boldi

Resource limits in Linux, when available, are currently very limited.

i.e.:
Too many process forks and your system may crash. This can be capped with threads-max, but may lead you into a lock-out.

Quoted 5 lines

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

Would this be difficult to implement?

How would you reclaim the resource after that limited time is over ? Kill processes?

Quoted 8 lines
Quoted 2 lines

That's one way, but really, the issue needs some deep thought. Leaving Linux exposed to a lock-out is rather frightening.

What exactly is it that you're worried about here? Do you have a particular concern that a process won't be able to fork or create a thread? Resources that can be allocated to user space processes always run the risk that their allocation will not succede. Its up to the application to deal with that.

Quoted 4 lines

Think about a DoS attack.

Thanks!

Be more specific. Are you talking about a fork bomb, a ICMP flood, what? preventing resource starvation/exhaustion is often handled in a way thats dovetailed to the semantics of how that resources is allocated (i.e. you prevent syn-flood attacks differently than you manage excessive disk usage).

Regards
Neil

Quoted 8 lines

--
Al

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 5 lines
/***************************************************
 *Neil Horman
 *Software Engineer
 *gpg keyid: 1024D / 0x92A74FA1 - http://pgp.mit.edu
 ***************************************************/
author
#6 lore raw
Quoted 16 lines · Neil Horman
Quoted 15 lines · Al Boldi
Quoted 14 lines · Neil Horman
Quoted 13 lines · Al Boldi
Quoted 11 lines · Rik van Riel
Quoted 7 lines · Al Boldi

Too many process forks and your system may crash. This can be capped with threads-max, but may lead you into a lock-out.

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

How would you reclaim the resource after that limited time is over ? Kill processes?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

That's one way, but really, the issue needs some deep thought. Leaving Linux exposed to a lock-out is rather frightening.

What exactly is it that you're worried about here?

Think about a DoS attack.

Be more specific. Are you talking about a fork bomb, a ICMP flood, what?

How would you deal with a situation where the system hit the threads-max ceiling?

Quoted 4 lines

preventing resource starvation/exhaustion is often handled in a way thats dovetailed to the semantics of how that resources is allocated (i.e. you prevent syn-flood attacks differently than you manage excessive disk usage).

The issue here is a general lack of proper kernel support for resource limits. The fork problem is just an example.

Thanks!

--
Al

#7 lore raw
Quoted 20 lines · Al Boldi
Quoted 19 lines · Neil Horman
Quoted 18 lines · Al Boldi
Quoted 17 lines · Neil Horman
Quoted 15 lines · Al Boldi
Quoted 11 lines · Rik van Riel
Quoted 7 lines · Al Boldi

Too many process forks and your system may crash. This can be capped with threads-max, but may lead you into a lock-out.

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

How would you reclaim the resource after that limited time is over ? Kill processes?

That's one way, but really, the issue needs some deep thought. Leaving Linux exposed to a lock-out is rather frightening.

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

What exactly is it that you're worried about here?

Think about a DoS attack.

Be more specific. Are you talking about a fork bomb, a ICMP flood, what?

How would you deal with a situation where the system hit the threads-max ceiling?

Nominally I would log the inability to successfully create a new process/thread, attempt to free some of my applications resources, and try again.

Quoted 8 lines
Quoted 4 lines

preventing resource starvation/exhaustion is often handled in a way thats dovetailed to the semantics of how that resources is allocated (i.e. you prevent syn-flood attacks differently than you manage excessive disk usage).

The issue here is a general lack of proper kernel support for resource limits. The fork problem is just an example.

Thats not really true. As Mr. Helsley pointed out, CKRM is available to provide a level of class based resource management if you need it. By default you can also create a level of resource limitation with ulimits as I mentioned. But no matter what you do, the only way you can guarantee that a system will be able to provide the resources your workload needs is to limit the number of resources your workload asks for, and in the event it asks for too much, make sure it can handle the denial of the resource gracefully.

Thanks and regards
Neil

Quoted 10 lines

Thanks!

--
Al

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 5 lines
/***************************************************
 *Neil Horman
 *Software Engineer
 *gpg keyid: 1024D / 0x92A74FA1 - http://pgp.mit.edu
 ***************************************************/
author
#8 lore raw
Quoted 28 lines · Neil Horman
Quoted 27 lines · Al Boldi
Quoted 25 lines · Neil Horman
Quoted 15 lines · Al Boldi
Quoted 14 lines · Neil Horman
Quoted 13 lines · Al Boldi
Quoted 11 lines · Rik van Riel
Quoted 7 lines · Al Boldi

Too many process forks and your system may crash. This can be capped with threads-max, but may lead you into a lock-out.

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

How would you reclaim the resource after that limited time is over ? Kill processes?

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

That's one way, but really, the issue needs some deep thought. Leaving Linux exposed to a lock-out is rather frightening.

What exactly is it that you're worried about here?

Think about a DoS attack.

Be more specific. Are you talking about a fork bomb, a ICMP flood, what?

Quoted 6 lines
Quoted 2 lines

How would you deal with a situation where the system hit the threads-max ceiling?

Nominally I would log the inability to successfully create a new process/thread, attempt to free some of my applications resources, and try again.

Consider this dilemma:
Runaway proc/s hit the limit.
Try to kill some and you are denied due to the resource limit. Use some previously running app like top, hope it hasn't been killed by some OOM situation, try killing some procs and another one takes it's place because of the runaway situation.
Raise the limit, and it gets filled by the runaways.
You are pretty much stuck.

You may get around the problem by a user-space solution, but this will always run the risks associated with user-space.

Quoted 4 lines
Quoted 2 lines

The issue here is a general lack of proper kernel support for resource limits. The fork problem is just an example.

Thats not really true. As Mr. Helsley pointed out, CKRM is available

Quoted 4 lines · Matthew Helsley

Have you looked at Class-Based Kernel Resource Managment (CKRM)
(http://ckrm.sf.net) to see if it fits your needs? My initial thought is that the CKRM numtasks controller may help limit forks in the way you describe.

Thanks for the link! CKRM is great!

Is there a CKRM-lite version? This would make it easier to be included into the mainline, something that would concentrate on the pressing issues, like lock-out prevention, and leave all the management features as an option.

Thanks!

--
Al

#9 lore raw
Quoted 36 lines · Al Boldi
Quoted 28 lines · Neil Horman
Quoted 27 lines · Al Boldi
Quoted 25 lines · Neil Horman
Quoted 15 lines · Al Boldi
Quoted 14 lines · Neil Horman
Quoted 13 lines · Al Boldi
Quoted 11 lines · Rik van Riel
Quoted 7 lines · Al Boldi

Too many process forks and your system may crash. This can be capped with threads-max, but may lead you into a lock-out.

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

How would you reclaim the resource after that limited time is over ? Kill processes?

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

That's one way, but really, the issue needs some deep thought. Leaving Linux exposed to a lock-out is rather frightening.

What exactly is it that you're worried about here?

Think about a DoS attack.

Be more specific. Are you talking about a fork bomb, a ICMP flood, what?

Quoted 6 lines
Quoted 2 lines

How would you deal with a situation where the system hit the threads-max ceiling?

Nominally I would log the inability to successfully create a new process/thread, attempt to free some of my applications resources, and try again.

Quoted 9 lines

Consider this dilemma:
Runaway proc/s hit the limit.
Try to kill some and you are denied due to the resource limit. Use some previously running app like top, hope it hasn't been killed by some OOM situation, try killing some procs and another one takes it's place because of the runaway situation.
Raise the limit, and it gets filled by the runaways.
You are pretty much stuck.

Not really, this is the sort of thing ulimit is meant for. To keep processes from any one user from running away. It lets you limit the damage it can do, until such time as you can control it and fix the runaway application.

Quoted 3 lines

You may get around the problem by a user-space solution, but this will always run the risks associated with user-space.

Ulimit isn't a user-space solution, its a user-_based_ restriction mechanism for resources. It allows you to prevent user X (or group X, IIRC) from creating more than A MB of files, or B processes, or allocating C KB of memory, etc. man 3 ulimit.

Quoted 21 lines
Quoted 4 lines
Quoted 2 lines

The issue here is a general lack of proper kernel support for resource limits. The fork problem is just an example.

Thats not really true. As Mr. Helsley pointed out, CKRM is available

Quoted 4 lines · Matthew Helsley

Have you looked at Class-Based Kernel Resource Managment (CKRM)
(http://ckrm.sf.net) to see if it fits your needs? My initial thought is that the CKRM numtasks controller may help limit forks in the way you describe.

Thanks for the link! CKRM is great!

Is there a CKRM-lite version? This would make it easier to be included into the mainline, something that would concentrate on the pressing issues, like lock-out prevention, and leave all the management features as an option.

Thanks!

--
Al

Signature · 5 lines
/***************************************************
 *Neil Horman
 *Software Engineer
 *gpg keyid: 1024D / 0x92A74FA1 - http://pgp.mit.edu
 ***************************************************/
author
#10 lore raw
Quoted 40 lines · Neil Horman
Quoted 30 lines · Al Boldi
Quoted 22 lines · Neil Horman
Quoted 21 lines · Al Boldi
Quoted 20 lines · Neil Horman
Quoted 19 lines · Al Boldi
Quoted 18 lines · Neil Horman
Quoted 16 lines · Al Boldi
Quoted 11 lines · Rik van Riel
Quoted 7 lines · Al Boldi

Too many process forks and your system may crash. This can be capped with threads-max, but may lead you into a lock-out.

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

How would you reclaim the resource after that limited time is over ? Kill processes?

That's one way, but really, the issue needs some deep thought. Leaving Linux exposed to a lock-out is rather frightening.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

What exactly is it that you're worried about here?

Think about a DoS attack.

Be more specific. Are you talking about a fork bomb, a ICMP flood, what?

Quoted 8 lines

Consider this dilemma:
Runaway proc/s hit the limit.
Try to kill some and you are denied due to the resource limit. Use some previously running app like top, hope it hasn't been killed by some OOM situation, try killing some procs and another one takes it's place because of the runaway situation.
Raise the limit, and it gets filled by the runaways.
You are pretty much stuck.

Quoted 4 lines

Not really, this is the sort of thing ulimit is meant for. To keep processes from any one user from running away. It lets you limit the damage it can do, until such time as you can control it and fix the runaway application.

threads-max = 1024
ulimit = 100 forks
11 runaway procs hitting the threads-max limit

This example is extreme, but it's possible, and there should be a safe and easy way out.

What do you think?

Thanks!
--
Al

#11 lore raw
Quoted 41 lines · Al Boldi
Quoted 40 lines · Neil Horman
Quoted 38 lines · Al Boldi
Quoted 22 lines · Neil Horman
Quoted 21 lines · Al Boldi
Quoted 20 lines · Neil Horman
Quoted 19 lines · Al Boldi
Quoted 18 lines · Neil Horman
Quoted 16 lines · Al Boldi
Quoted 11 lines · Rik van Riel
Quoted 7 lines · Al Boldi

Too many process forks and your system may crash. This can be capped with threads-max, but may lead you into a lock-out.

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

How would you reclaim the resource after that limited time is over ? Kill processes?

That's one way, but really, the issue needs some deep thought. Leaving Linux exposed to a lock-out is rather frightening.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

What exactly is it that you're worried about here?

Think about a DoS attack.

Be more specific. Are you talking about a fork bomb, a ICMP flood, what?

Consider this dilemma:
Runaway proc/s hit the limit.
Try to kill some and you are denied due to the resource limit. Use some previously running app like top, hope it hasn't been killed by some OOM situation, try killing some procs and another one takes it's place because of the runaway situation.
Raise the limit, and it gets filled by the runaways.
You are pretty much stuck.

Quoted 9 lines
Quoted 4 lines

Not really, this is the sort of thing ulimit is meant for. To keep processes from any one user from running away. It lets you limit the damage it can do, until such time as you can control it and fix the runaway application.

threads-max = 1024
ulimit = 100 forks
11 runaway procs hitting the threads-max limit

This is incorrect. If you ulimit a user to 100 forks, and 11 processes running with that uid start to fork repeatedly, they will get fork failures after they have, in aggregate called fork 89 times. That user can have no more than 100 processes running in the system at any given time. Another user (or root) can fork another process to kill one of the runaways.

If you have a user process that for some reason legitimately needs to try use every process resource available in the system, the yes, you are prone to a lock out condition, if you have no way of killing those processes from a controlling terminal, then yes, you are prone to lock out. In those conditions I would set my ulimit on processes for the user running this process to something less than threads-max, so that I could have some wiggle room to get out of that situation. I would of course also file a bug report with the application author, but thats another discussion :).
Regards
Neil

Quoted 8 lines

This example is extreme, but it's possible, and there should be a safe and easy way out.

What do you think?

Thanks!
--
Al

Signature · 5 lines
/***************************************************
 *Neil Horman
 *Software Engineer
 *gpg keyid: 1024D / 0x92A74FA1 - http://pgp.mit.edu
 ***************************************************/
author
#12 lore raw
Quoted 48 lines · Neil Horman
Quoted 47 lines · Al Boldi
Quoted 45 lines · Neil Horman
Quoted 31 lines · Al Boldi
Quoted 23 lines · Neil Horman
Quoted 22 lines · Al Boldi
Quoted 21 lines · Neil Horman
Quoted 20 lines · Al Boldi
Quoted 19 lines · Neil Horman
Quoted 17 lines · Al Boldi
Quoted 5 lines · Rik van Riel
Quoted 4 lines · Al Boldi

Too many process forks and your system may crash. This can be capped with threads-max, but may lead you into a lock-out.

Quoted 7 lines
Quoted 4 lines

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

How would you reclaim the resource after that limited time is over ? Kill processes?

That's one way, but really, the issue needs some deep thought. Leaving Linux exposed to a lock-out is rather frightening.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

What exactly is it that you're worried about here?

Think about a DoS attack.

Be more specific. Are you talking about a fork bomb, a ICMP flood, what?

Quoted 8 lines

Consider this dilemma:
Runaway proc/s hit the limit.
Try to kill some and you are denied due to the resource limit. Use some previously running app like top, hope it hasn't been killed by some OOM situation, try killing some procs and another one takes it's place because of the runaway situation.
Raise the limit, and it gets filled by the runaways.
You are pretty much stuck.

Not really, this is the sort of thing ulimit is meant for. To keep processes from any one user from running away. It lets you limit the damage it can do, until such time as you can control it and fix the runaway application.

Quoted 6 lines
Quoted 3 lines

threads-max = 1024
ulimit = 100 forks
11 runaway procs hitting the threads-max limit

This is incorrect. If you ulimit a user to 100 forks, and 11 processes running with that uid

Different uid.

Quoted 3 lines

If you have a user process that for some reason legitimately needs to try use every process resource available in the system, then yes, you are prone to a lock out condition

Couldn't this be easily fixed in kernel-space?

Thanks!

--
Al

#13 lore raw
Quoted 49 lines · Al Boldi
Quoted 48 lines · Neil Horman
Quoted 47 lines · Al Boldi
Quoted 45 lines · Neil Horman
Quoted 31 lines · Al Boldi
Quoted 23 lines · Neil Horman
Quoted 22 lines · Al Boldi
Quoted 21 lines · Neil Horman
Quoted 20 lines · Al Boldi
Quoted 19 lines · Neil Horman
Quoted 17 lines · Al Boldi
Quoted 5 lines · Rik van Riel
Quoted 4 lines · Al Boldi

Too many process forks and your system may crash.
This can be capped with threads-max, but may lead
you into a lock-out.

Quoted 7 lines
Quoted 4 lines

What is needed is a soft, hard, and a special
emergency limit that would allow you to use the
resource for a limited time to circumvent a
lock-out.

How would you reclaim the resource after that limited time is over ? Kill processes?

That's one way, but really, the issue needs some deep thought. Leaving Linux exposed to a lock-out is rather frightening.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

What exactly is it that you're worried about here?

Think about a DoS attack.

Be more specific. Are you talking about a fork bomb, a ICMP flood, what?

Quoted 8 lines

Consider this dilemma:
Runaway proc/s hit the limit.
Try to kill some and you are denied due to the resource limit. Use some previously running app like top, hope it hasn't been killed by some OOM situation, try killing some procs and another one takes it's place because of the runaway situation.
Raise the limit, and it gets filled by the runaways.
You are pretty much stuck.

Not really, this is the sort of thing ulimit is meant for. To keep processes from any one user from running away. It lets you limit the damage it can do, until such time as you can control it and fix the runaway application.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

threads-max = 1024
ulimit = 100 forks
11 runaway procs hitting the threads-max limit

This is incorrect. If you ulimit a user to 100 forks, and 11 processes running with that uid

Different uid.

Then yes, if you set a system-wide limit that is less than the sum of the limits imposed on each accountable part of the system you can have lock out. But thats your fault for misconfiguring the system. Don't do that.

Quoted 6 lines
Quoted 3 lines

If you have a user process that for some reason legitimately needs to try use every process resource available in the system, then yes, you are prone to a lock out condition

Couldn't this be easily fixed in kernel-space?

You're not getting it. The resource limits applied by ulimit (and CKRM as far as I know), _are_ inforced in kernel space. The ulimit library call and its corresponding setrlimit system call set resource limitations in the rlim array thats part of each task struct. These limits are queried whenever an instance of the corresponding resource is requested by a user space process, if the requesting process is over its limit, the request is deined.

Regards
Neil

Quoted 10 lines

Thanks!

--
Al

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 5 lines
/***************************************************
 *Neil Horman
 *Software Engineer
 *gpg keyid: 1024D / 0x92A74FA1 - http://pgp.mit.edu
 ***************************************************/
#14 lore raw
In reply to Al Boldi

On Tue, 2005-09-27 at 08:08 +0300, Al Boldi wrote:
<snip>

Quoted 8 lines

Consider this dilemma:
Runaway proc/s hit the limit.
Try to kill some and you are denied due to the resource limit. Use some previously running app like top, hope it hasn't been killed by some OOM situation, try killing some procs and another one takes it's place because of the runaway situation.
Raise the limit, and it gets filled by the runaways.
You are pretty much stuck.

CKRM can solve this problem nicely. You can define classes (for example, you can define a class and it attach to a user). Limits will be applied only to that class(user), failures will be seen only by that class(user) and the rest of the system will be free to operate without getting into the situation stated above.

Quoted 9 lines

and associate resources to a class
You may get around the problem by a user-space solution, but this will always run the risks associated with user-space.

Quoted 4 lines
Quoted 2 lines

The issue here is a general lack of proper kernel support for resource limits. The fork problem is just an example.

Thats not really true. As Mr. Helsley pointed out, CKRM is available

Quoted 7 lines
Quoted 4 lines · Matthew Helsley

Have you looked at Class-Based Kernel Resource Managment (CKRM)
(http://ckrm.sf.net) to see if it fits your needs? My initial thought is that the CKRM numtasks controller may help limit forks in the way you describe.

Thanks for the link! CKRM is great!

Thank you!! :)

Quoted 2 lines

Is there a CKRM-lite version? This would make it easier to be included into

we are currently working on reducing the codesize and complexity of CKRM, which will be lot thinner and less complex than what was in -mm tree a while ago. The development is underway and you can follow the progress of the f-series in ckrm-tech mailing list.

Quoted 3 lines

the mainline, something that would concentrate on the pressing issues, like lock-out prevention, and leave all the management features as an option.

You are welcome to join the mailing list and provide feedback on how the f-series shapes up.

Thanks,

chandra

Quoted 12 lines

Thanks!

--
Al

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 4 lines
----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - [email protected]   |      .......you may get it.
----------------------------------------------------------------------
#15 lore raw
In reply to Al Boldi
Quoted 10 lines · Al Boldi

Resource limits in Linux, when available, are currently very limited.

i.e.:
Too many process forks and your system may crash.
This can be capped with threads-max, but may lead you into a lock-out.

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

Whats insufficient about the per-user limits that can be imposed by the ulimit syscall?

Quoted 12 lines

Would this be difficult to implement?

Thanks!

--
Al

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 5 lines
/***************************************************
 *Neil Horman
 *Software Engineer
 *gpg keyid: 1024D / 0x92A74FA1 - http://pgp.mit.edu
 ***************************************************/
#16 lore raw
In reply to Al Boldi

While talking about limits, one of my customers report that if they set "ulimit -d" to be say 8GB, and then a program goes and attempts to allocate 16GB (in one shot), that the process will hang on the 16GB allocate as the machine does not have enough memory+swap to handle this, the process is at this time unkillable, the customers method to kill the process is to send the process a kill signal, and then create enough swap to be able to meet the request, after the request is filled the process terminates.

It would seem that the best thing to do would be to abort on allocates that will by themselves exceed the limit.

This was a custom version of a earlier version of the 2.6 kernel, I would bet that this has not changed in quite a while.

Roger

Quoted 32 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Al Boldi
Sent: Sunday, September 25, 2005 9:13 AM
To: [email protected]
Subject: Resource limits

Resource limits in Linux, when available, are currently very limited.

i.e.:
Too many process forks and your system may crash.
This can be capped with threads-max, but may lead you into a lock-out.

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

Would this be difficult to implement?

Thanks!

--
Al

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#17 lore raw
Quoted 2 lines · Roger Heflin

While talking about limits, one of my customers report that if they set "ulimit -d" to be say 8GB, and then a program goes and

The kernel doesn't yet support rlimit64() - glibc does but it emulates it best effort. Thats a good intro project for someone

Quoted 2 lines

It would seem that the best thing to do would be to abort on allocates that will by themselves exceed the limit.

2.6 supports "no overcommit" modes.

Alan

author
#18 lore raw
Quoted 11 lines · Alan Cox
Quoted 2 lines · Roger Heflin

While talking about limits, one of my customers report that if they set "ulimit -d" to be say 8GB, and then a program goes and

The kernel doesn't yet support rlimit64() - glibc does but it emulates it best effort. Thats a good intro project for someone

Quoted 2 lines

It would seem that the best thing to do would be to abort on allocates that will by themselves exceed the limit.

2.6 supports "no overcommit" modes.

By name only. see "Kswapd flaw" thread.

Thanks!

--
Al

#19 lore raw
In reply to Alan Cox
Quoted 9 lines · Roger Heflin
Quoted 2 lines

While talking about limits, one of my customers report that if they set "ulimit -d" to be say 8GB, and then a program goes and

The kernel doesn't yet support rlimit64() - glibc does but it emulates it best effort. Thats a good intro project for someone

Quoted 8 lines
Quoted 1 line

It would seem that the best thing to do would be to abort

on allocates

Quoted 1 line

that will by themselves exceed the limit.

2.6 supports "no overcommit" modes.

Alan

Ah.

So any limit over 4GB, is emulated through glibc which means the fix would need to be in the emulation that is outside of the kernel.

And I think they were setting the limit to more like 32 or 48GB,
and having single allocation's go over that. Some of the machines
in question have 32GB of ram, others have 64GB of ram, both with fair amounts of swap, and when the event happens they need to create enough swap to get enough swap to process the request.

The overcommit thing may do what they want.

Thanks.
Roger

#20 lore raw
In reply to Roger Heflin
Quoted 10 lines · Roger Heflin

While talking about limits, one of my customers report that if they set "ulimit -d" to be say 8GB, and then a program goes and attempts to allocate 16GB (in one shot), that the process will hang on the 16GB allocate as the machine does not have enough memory+swap to handle this, the process is at this time unkillable, the customers method to kill the process is to send the process a kill signal, and then create enough swap to be able to meet the request, after the request is filled the process terminates.

Quoted 7 lines

It would seem that the best thing to do would be to abort on allocates that will by themselves exceed the limit.

This was a custom version of a earlier version of the 2.6 kernel, I would bet that this has not changed in quite a while.

Roger

It's simple. Set /proc/sys/vm/overcommit_memory to 2 (iirc) to get arround this `bug' .
--
Coywolf Qi Hunt
http://sosdg.org/~coywolf/

#21 lore raw
In reply to Al Boldi
Quoted 9 lines · Al Boldi

Resource limits in Linux, when available, are currently very limited.

i.e.:
Too many process forks and your system may crash.
This can be capped with threads-max, but may lead you into a lock-out.

What is needed is a soft, hard, and a special emergency limit that would allow you to use the resource for a limited time to circumvent a lock-out.

Quoted 6 lines

Would this be difficult to implement?

Thanks!

--
Al

Have you looked at Class-Based Kernel Resource Managment (CKRM)
(http://ckrm.sf.net) to see if it fits your needs? My initial thought is that the CKRM numtasks controller may help limit forks in the way you describe.

If you have any questions about it please join the CKRM-Tech mailing
list ([email protected]) or chat with folks on the OFTC IRC #ckrm channel.

Cheers,
-Matt Helsley