New: page allocation failure. order:0, mode:0x20
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]/
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 22 lines · On Mon, 13 Apr 2009 19:27:27 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=13084
Summary: page allocation failure. order:0, mode:0x20 Product: Memory Management Version: 2.5 Kernel Version: 2.6.17.4 Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: high Priority: P1 Component: Page Allocator AssignedTo: [email protected] ReportedBy: [email protected] Regression: NoCreated an attachment (id=20964)
--> (http://bugzilla.kernel.org/attachment.cgi?id=20964)
kernel config file.
Quoted 7 lines
The system is Intel Xeon Quad core with 8G physical RAM. When it's under UPD loads, e.g., DNS queries, the box is stuck in terms it cannot be pinged or login. By checking syslog, I'm seeing following trace back from various dameon/processes. The network controller is E1000 82571 with NAPI enabled in kernel.
page allocation failure. order:0, mode:0x20
This is very common. e1000 attempts to do large memory allocations from within interrupt context and the page allocator cannot satisfy the allocation and is not allowed to do the necessary work to make the allocation attempt succeed. It's the same with all net drivers, but e1000 is especially prone, apparently because of hardware suckiness.
However the networking stack should just drop the packet and the system will recover.
You report is unclear. Yes, the machine wedges up under the UDP load. But does it recover when the other machine stops spraying UDP packets at this machine? It _should_ recover. If it does not, we have a bug somewhere.
The usual workaround for these problems is to increase the value in /proc/sys/vm/min_free_kbytes.
2.6.17 is fairly old. If we need to do additional work on this report then we'll be asking you to test something more recent - ideally 2.6.29.
Thanks.
Quoted 34 lines · Andrew Morton
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 28 lines · On Mon, 13 Apr 2009 19:27:27 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=13084
Summary: page allocation failure. order:0, mode:0x20 Product: Memory Management Version: 2.5 Kernel Version: 2.6.17.4 Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: high Priority: P1 Component: Page Allocator AssignedTo: [email protected] ReportedBy: [email protected] Regression: NoCreated an attachment (id=20964)
--> (http://bugzilla.kernel.org/attachment.cgi?id=20964)
kernel config file.The system is Intel Xeon Quad core with 8G physical RAM. When it's under UPD loads, e.g., DNS queries, the box is stuck in terms it cannot be pinged or login. By checking syslog, I'm seeing following trace back from various dameon/processes. The network controller is E1000 82571 with NAPI enabled in kernel.
Quoted 7 lines
Quoted 1 line
page allocation failure. order:0, mode:0x20
This is very common. e1000 attempts to do large memory allocations from within interrupt context and the page allocator cannot satisfy the allocation and is not allowed to do the necessary work to make the allocation attempt succeed. It's the same with all net drivers, but e1000 is especially prone, apparently because of hardware suckiness.
while in jumbo mode, andrew's statement is true, but with order:0 allocation failures it is just normal networking goo that causes the memory allocator to run out of free pages, seems much less frequent in newer kernels.
Quoted 2 lines
However the networking stack should just drop the packet and the system will recover.
I think at that point the kernel gets quite busy printing warnings about how much it is out of memory.
Quoted 4 lines
You report is unclear. Yes, the machine wedges up under the UDP load. But does it recover when the other machine stops spraying UDP packets at this machine? It _should_ recover. If it does not, we have a bug somewhere.
In this case kmem_cache_alloc is failing to get memory, being called by the route_dst code, maybe someone on netdev can comment if this has been fixed along the way.
Quoted 2 lines
The usual workaround for these problems is to increase the value in
/proc/sys/vm/min_free_kbytes.
this should help a lot in my experience.
Quoted 3 lines
2.6.17 is fairly old. If we need to do additional work on this report then we'll be asking you to test something more recent - ideally 2.6.29.
If you must run 2.6.17, then you might want to try the e1000e driver (*not e1000*) from sourceforge for your 82571.
Otherwise I also will be asking you to soon try a newer kernel.
Here is the memory snapshot when problem happening:
MemTotal: 8307844 kB MemFree: 6091208 kB Buffers: 6524 kB Cached: 1121528 kB SwapCached: 0 kB Active: 1361052 kB Inactive: 25784 kB HighTotal: 7470464 kB HighFree: 6083688 kB LowTotal: 837380 kB LowFree: 7520 kB SwapTotal: 2047992 kB SwapFree: 2047992 kB Dirty: 744488 kB Writeback: 0 kB Mapped: 285532 kB Slab: 797500 kB CommitLimit: 6201912 kB Committed_AS: 459788 kB PageTables: 3532 kB VmallocTotal: 118776 kB VmallocUsed: 2432 kB VmallocChunk: 116084 kB
You can see I have lots of physical RAM available. The LowFree reduction rate is about 10M/Second.
Quoted 77 lines · Jesse
Quoted 34 lines · Andrew Morton
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 28 lines · On Mon, 13 Apr 2009 19:27:27 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=13084
Summary: page allocation failure. order:0, mode:0x20 Product: Memory Management Version: 2.5 Kernel Version: 2.6.17.4 Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: high Priority: P1 Component: Page Allocator AssignedTo: [email protected] ReportedBy: [email protected] Regression: NoCreated an attachment (id=20964)
--> (http://bugzilla.kernel.org/attachment.cgi?id=20964)
kernel config file.The system is Intel Xeon Quad core with 8G physical RAM. When it's under UPD loads, e.g., DNS queries, the box is stuck in terms it cannot be pinged or login. By checking syslog, I'm seeing following trace back from various dameon/processes. The network controller is E1000 82571 with NAPI enabled in kernel.
Quoted 7 lines
Quoted 1 line
page allocation failure. order:0, mode:0x20
This is very common. e1000 attempts to do large memory allocations from within interrupt context and the page allocator cannot satisfy the allocation and is not allowed to do the necessary work to make the allocation attempt succeed. It's the same with all net drivers, but e1000 is especially prone, apparently because of hardware suckiness.
while in jumbo mode, andrew's statement is true, but with order:0 allocation failures it is just normal networking goo that causes the memory allocator to run out of free pages, seems much less frequent in newer kernels.
Quoted 2 lines
However the networking stack should just drop the packet and the system will recover.
I think at that point the kernel gets quite busy printing warnings about how much it is out of memory.
Quoted 4 lines
You report is unclear. Yes, the machine wedges up under the UDP load. But does it recover when the other machine stops spraying UDP packets at this machine? It _should_ recover. If it does not, we have a bug somewhere.
In this case
kmem_cache_allocis failing to get memory, being called by theroute_dstcode, maybe someone on netdev can comment if this has been fixed along the way.Quoted 2 lines
The usual workaround for these problems is to increase the value in
/proc/sys/vm/min_free_kbytes.this should help a lot in my experience.
Quoted 3 lines
2.6.17 is fairly old. If we need to do additional work on this report then we'll be asking you to test something more recent - ideally 2.6.29.
If you must run 2.6.17, then you might want to try the e1000e driver (*not e1000*) from sourceforge for your 82571.
Otherwise I also will be asking you to soon try a newer kernel.
David Miller
maintainerFrom: "Brandeburg, Jesse" <[email protected]>
Date: Mon, 13 Apr 2009 13:06:04 -0700 (Pacific Daylight Time)
Quoted 9 lines · Andrew Morton
Quoted 4 lines
You report is unclear. Yes, the machine wedges up under the UDP load. But does it recover when the other machine stops spraying UDP packets at this machine? It _should_ recover. If it does not, we have a bug somewhere.
In this case
kmem_cache_allocis failing to get memory, being called by theroute_dstcode, maybe someone on netdev can comment if this has been fixed along the way.
Although I have some level of tolerance, there is zero way I'm going to analyze anything on 2.6.17 kernels nor am I going to encourage other core networking developers to waste their time on this either.
It is easily the case that we've fixed 10's of thousands of VM and networking bugs since then, if not more.