OOM notifications
Hi,
AIX contains the SIGDANGER signal to notify applications to free up some unused cached memory:
http://www.ussg.iu.edu/hypermail/linux/kernel/0007.0/0901.html
There have been a few discussions on implementing such an idea on Linux, but nothing concrete has been achieved.
On the kernel side Rik suggested two notification points: "about to swap" (for desktop scenarios) and "about to OOM" (for embedded-like scenarios).
With that assumption in mind it would be necessary to either have two special devices for notification, or somehow indicate both events through the same file descriptor.
Comments are more than welcome.
Andrew Morton
maintainerQuoted 14 lines · On Thu, 18 Oct 2007 16:15:31 -0400 Marcelo Tosatti <[email protected]> wrote:
Hi,
AIX contains the SIGDANGER signal to notify applications to free up some unused cached memory:
http://www.ussg.iu.edu/hypermail/linux/kernel/0007.0/0901.html
There have been a few discussions on implementing such an idea on Linux, but nothing concrete has been achieved.
On the kernel side Rik suggested two notification points: "about to swap" (for desktop scenarios) and "about to OOM" (for embedded-like scenarios).
Quoted 5 lines
With that assumption in mind it would be necessary to either have two special devices for notification, or somehow indicate both events through the same file descriptor.
Comments are more than welcome.
Martin was talking about some mad scheme wherin you'd create a bunch of pseudo files (say, /proc/foo/0, /proc/foo/1, ..., /proc/foo/9) and each one would become "ready" when the MM scanning priority reaches 10%, 20%, ... 100%.
Obviously there would need to be a lot of abstraction to unhook a permanent userspace feature from a transient kernel implementation, but the basic idea is that a process which wants to know when the VM is getting into the orange zone would select() on the file "7" and a process which wants to know when the VM is getting into the red zone would select on file "9".
It get more complicated with NUMA memory nodes and cgroup memory controllers.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 27 lines · Andrew Morton
Quoted 14 lines · On Thu, 18 Oct 2007 16:15:31 -0400 Marcelo Tosatti <[email protected]> wrote:
Hi,
AIX contains the SIGDANGER signal to notify applications to free up some unused cached memory:
http://www.ussg.iu.edu/hypermail/linux/kernel/0007.0/0901.html
There have been a few discussions on implementing such an idea on Linux, but nothing concrete has been achieved.
On the kernel side Rik suggested two notification points: "about to swap" (for desktop scenarios) and "about to OOM" (for embedded-like scenarios).
Quoted 5 lines
With that assumption in mind it would be necessary to either have two special devices for notification, or somehow indicate both events through the same file descriptor.
Comments are more than welcome.
Martin was talking about some mad scheme wherin you'd create a bunch of pseudo files (say,
/proc/foo/0,/proc/foo/1, ...,/proc/foo/9) and each one would become "ready" when the MM scanning priority reaches 10%, 20%, ... 100%.
Quoted 8 lines
Obviously there would need to be a lot of abstraction to unhook a permanent userspace feature from a transient kernel implementation, but the basic idea is that a process which wants to know when the VM is getting into the orange zone would
select()on the file "7" and a process which wants to know when the VM is getting into the red zone would select on file "9".It get more complicated with NUMA memory nodes and cgroup memory controllers.
We ended up not doing that, but making a scanner that saw what percentage of the LRU was touched in the last n seconds, and printing that to userspace to deal with.
Turns out priority is a horrible metric to use for this - it stays at default for ages, then falls off a cliff far too quickly to react to.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerQuoted 14 lines · On Fri, 26 Oct 2007 14:05:47 -0700 Martin Bligh <[email protected]> wrote:
Quoted 5 lines
Martin was talking about some mad scheme wherin you'd create a bunch of pseudo files (say,
/proc/foo/0,/proc/foo/1, ...,/proc/foo/9) and each one would become "ready" when the MM scanning priority reaches 10%, 20%, ... 100%.Quoted 8 lines
Obviously there would need to be a lot of abstraction to unhook a permanent userspace feature from a transient kernel implementation, but the basic idea is that a process which wants to know when the VM is getting into the orange zone would
select()on the file "7" and a process which wants to know when the VM is getting into the red zone would select on file "9".It get more complicated with NUMA memory nodes and cgroup memory controllers.
Quoted 7 lines
We ended up not doing that, but making a scanner that saw what percentage of the LRU was touched in the last n seconds, and printing that to userspace to deal with.
Turns out priority is a horrible metric to use for this - it stays at default for ages, then falls off a cliff far too quickly to react to.
Sure, but in terms of high-level userspace interface, being able to select() on a group of priority buckets (spread across different nodes, zones and cgroups) seems a lot more flexible than any signal-based approach we could come up with.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines · On Fri, 26 Oct 2007 14:11:12 -0700 Andrew Morton <[email protected]>…
Sure, but in terms of high-level userspace interface, being able to
select()on a group of priority buckets (spread across different nodes, zones and cgroups) seems a lot more flexible than any signal-based approach we could come up with.
Absolutely, the process needs to be able to just poll or select on a file descriptor from the process main loop.
I am not convinced that the magic of NUMA memory distribution and NUMA memory pressure should be visible to userspace. Due to the thundering herd problem we cannot wake up all of the processes that select on the filedescriptor at the same time anyway, so we can (later on) add NUMA magic to the process selection logic in the kernel to only wake up processes on the right NUMA nodes.
The initial patch probably does not need that.
Signature · 9 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 19 lines · Rik van Riel
Quoted 4 lines · On Fri, 26 Oct 2007 14:11:12 -0700 Andrew Morton <[email protected]>…
Sure, but in terms of high-level userspace interface, being able to
select()on a group of priority buckets (spread across different nodes, zones and cgroups) seems a lot more flexible than any signal-based approach we could come up with.Absolutely, the process needs to be able to just poll or select on a file descriptor from the process main loop.
I am not convinced that the magic of NUMA memory distribution and NUMA memory pressure should be visible to userspace. Due to the thundering herd problem we cannot wake up all of the processes that select on the filedescriptor at the same time anyway, so we can (later on) add NUMA magic to the process selection logic in the kernel to only wake up processes on the right NUMA nodes.
Quoted 1 line
The initial patch probably does not need that.
Depends if you're using cpusets or not, I think?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 20 lines · On Fri, 26 Oct 2007 14:59:01 -0700 Martin Bligh <[email protected]> wrote:
Quoted 19 lines · Rik van Riel
Quoted 4 lines · On Fri, 26 Oct 2007 14:11:12 -0700 Andrew Morton <[email protected]>…
Sure, but in terms of high-level userspace interface, being able to
select()on a group of priority buckets (spread across different nodes, zones and cgroups) seems a lot more flexible than any signal-based approach we could come up with.Absolutely, the process needs to be able to just poll or select on a file descriptor from the process main loop.
I am not convinced that the magic of NUMA memory distribution and NUMA memory pressure should be visible to userspace. Due to the thundering herd problem we cannot wake up all of the processes that select on the filedescriptor at the same time anyway, so we can (later on) add NUMA magic to the process selection logic in the kernel to only wake up processes on the right NUMA nodes.
Quoted 3 lines
Quoted 1 line
The initial patch probably does not need that.
Depends if you're using cpusets or not, I think?
The kernel knows on which cpuset a process can run.
The process itself may have been relocated to a different cpuset at runtime, without it even knowing.
Because of that I think the magic of which process(es) to wake up when there is memory pressure in some NUMA node should live in the kernel.
Signature · 9 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Andrew Morton
It get more complicated with NUMA memory nodes and cgroup memory controllers.
At OLS this year, users wanted user space notification of OOM for cgroup memory controller. When a group is about to OOM, a notification can help an external application re-adjust memory limits across the system.
Keeping some memory reserved for handling OOM, this scheme could be extended to handle global OOM conditions as well.
Signature · 10 lines
Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hello,
Quoted 8 lines
AIX contains the SIGDANGER signal to notify applications to free up some unused cached memory:
http://www.ussg.iu.edu/hypermail/linux/kernel/0007.0/0901.html
There have been a few discussions on implementing such an idea on Linux, but nothing concrete has been achieved.
Quoted 7 lines
On the kernel side Rik suggested two notification points: "about to swap" (for desktop scenarios) and "about to OOM" (for embedded-like scenarios).
With that assumption in mind it would be necessary to either have two special devices for notification, or somehow indicate both events through the same file descriptor.
Actually, wouldn't a generic netlink interface be more elegant? Then
we could connect it with DBUS and it would be much easier for applications (Desktop) to handle such events.
I agree that near-to-oom conditions are quite volatile and maybe we
want a technically simple (and thus more reliable) mechanism for the notification but I anyway wanted to point to this possibility.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SuSE CR Labs
Quoted 17 lines · On Tue, 30 Oct 2007 15:57:20 +0100 Jan Kara <[email protected]> wrote:
Hello,
Quoted 8 lines
AIX contains the SIGDANGER signal to notify applications to free up some unused cached memory:
http://www.ussg.iu.edu/hypermail/linux/kernel/0007.0/0901.html
There have been a few discussions on implementing such an idea on Linux, but nothing concrete has been achieved.
Quoted 7 lines
On the kernel side Rik suggested two notification points: "about to swap" (for desktop scenarios) and "about to OOM" (for embedded-like scenarios).
With that assumption in mind it would be necessary to either have two special devices for notification, or somehow indicate both events through the same file descriptor.
Quoted 6 lines
Actually, wouldn't a generic netlink interface be more elegant? Then
we could connect it with DBUS and it would be much easier for applications (Desktop) to handle such events.
I agree that near-to-oom conditions are quite volatile and maybe we
want a technically simple (and thus more reliable) mechanism for the notification but I anyway wanted to point to this possibility.
There's nothing wrong with being able to get this info via DBUS, but we cannot expect every database and JVM out there (big targets for the "reduce your memory footprint" thing on servers) to grow a DBUS interface.
Signature · 1 line
All Rights Reversed
Quoted 27 lines · Rik van Riel
Quoted 17 lines · On Tue, 30 Oct 2007 15:57:20 +0100 Jan Kara <[email protected]> wrote:
Hello,
Quoted 8 lines
AIX contains the SIGDANGER signal to notify applications to free up some unused cached memory:
http://www.ussg.iu.edu/hypermail/linux/kernel/0007.0/0901.html
There have been a few discussions on implementing such an idea on Linux, but nothing concrete has been achieved.
Quoted 7 lines
On the kernel side Rik suggested two notification points: "about to swap" (for desktop scenarios) and "about to OOM" (for embedded-like scenarios).
With that assumption in mind it would be necessary to either have two special devices for notification, or somehow indicate both events through the same file descriptor.
Quoted 6 lines
Actually, wouldn't a generic netlink interface be more elegant? Then
we could connect it with DBUS and it would be much easier for applications (Desktop) to handle such events.
I agree that near-to-oom conditions are quite volatile and maybe we
want a technically simple (and thus more reliable) mechanism for the notification but I anyway wanted to point to this possibility.
Quoted 4 lines
There's nothing wrong with being able to get this info via DBUS, but we cannot expect every database and JVM out there (big targets for the "reduce your memory footprint" thing on servers) to grow a DBUS interface.
Hmm, that's right, but still the kernel->userspace interface could be via
netlink (which is much more flexible than signals etc.) and then in userspace we could implement also some simple interface (UNIX socket?) for server like apps...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 4 lines · On Tue, 30 Oct 2007 16:55:25 +0100 Jan Kara <[email protected]> wrote:
Hmm, that's right, but still the
kernel->userspaceinterface could be via
netlink (which is much more flexible than signals etc.) and then in userspace we could implement also some simple interface (UNIX socket?) for server like apps...
I think we all agree that it should not be a Unix signal, if only because glibc cannot manipulate memory pools from signal handlers :)
The low memory message (for lack of a better word) needs to get to userspace over a file descriptor, which the process can select() or poll() on from its main loop.
Whether that is a device node, a sysfs file, a netlink socket or something else ... I don't particularly care :)
Signature · 1 line
All Rights Reversed