mount.nfs: cannot allocate memory

I've been seeing these messages on my AMD Fusion server running linux-3.3.7-64bit. Does this ring any bells for anyone else? The system is NOT low on memory.

I'm building/installing 3.4.2 now to see if it behaves any better.

[962841.265658] mount.nfs: page allocation failure: order:4, mode:0xc0d0
[962841.265674] Pid: 32116, comm: mount.nfs Not tainted 3.3.7 #2
[962841.265680] Call Trace:
[962841.265700]  [<ffffffff81079363>] ? warn_alloc_failed+0x11a/0x12d
[962841.265713]  [<ffffffff8107b904>] ? __alloc_pages_nodemask+0x6c0/0x702
[962841.265725]  [<ffffffff8114fcb8>] ? timerqueue_del+0x53/0x63
[962841.265758]  [<ffffffff8107b9ba>] ? __get_free_pages+0x10/0x3f
[962841.265805]  [<ffffffffa01ea33d>] ? nfs_idmap_new+0x28/0xde [nfs]
[962841.265836]  [<ffffffffa01c79c9>] ? nfs4_init_client+0x74/0x12a [nfs]
[962841.265863]  [<ffffffffa01c6f3b>] ? nfs_get_client+0x337/0x476 [nfs]
[962841.265874]  [<ffffffff81087ad8>] ? pcpu_alloc+0x796/0x7ad
[962841.265885]  [<ffffffff810403f7>] ? should_resched+0x5/0x23
[962841.265913]  [<ffffffffa01c70f2>] ? nfs4_set_client+0x78/0xca [nfs]
[962841.265944]  [<ffffffffa01c7b5c>] ? nfs4_create_server+0xdd/0x208 [nfs]
[962841.265956]  [<ffffffff812c0de1>] ? _cond_resched+0x7/0x1c
[962841.265990]  [<ffffffffa01cfa3c>] ? nfs4_remote_mount+0x50/0x183 [nfs]
[962841.266007]  [<ffffffff810ab334>] ? mount_fs+0xc/0xa0
[962841.266020]  [<ffffffff810bd3fd>] ? vfs_kern_mount+0x61/0xcc
[962841.266053]  [<ffffffffa01cfc3f>] ? nfs_do_root_mount+0x89/0xaa [nfs]
[962841.266088]  [<ffffffffa01d0702>] ? nfs4_try_mount.isra.27+0x4c/0x9e [nfs]
[962841.266120]  [<ffffffffa01d1145>] ? nfs_fs_mount+0x3e9/0x5ef [nfs]
[962841.266131]  [<ffffffff8114b3bc>] ? ida_get_new_above+0x176/0x198
[962841.266147]  [<ffffffff810ab334>] ? mount_fs+0xc/0xa0
[962841.266159]  [<ffffffff810bd3fd>] ? vfs_kern_mount+0x61/0xcc
[962841.266172]  [<ffffffff810bd7f0>] ? do_kern_mount+0x49/0xd8
[962841.266182]  [<ffffffff810beec8>] ? do_mount+0x630/0x695
[962841.266191]  [<ffffffff810be80a>] ? copy_mount_options+0xc3/0x126
[962841.266201]  [<ffffffff810bf00f>] ? sys_mount+0x88/0xca
[962841.266212]  [<ffffffff812c20a2>] ? system_call_fastpath+0x16/0x1b
[962841.266219] Mem-Info:
[962841.266224] DMA per-cpu:
[962841.266230] CPU    0: hi:    0, btch:   1 usd:   0
[962841.266237] CPU    1: hi:    0, btch:   1 usd:   0
[962841.266242] DMA32 per-cpu:
[962841.266248] CPU    0: hi:  186, btch:  31 usd:   0
[962841.266255] CPU    1: hi:  186, btch:  31 usd:   0
[962841.266260] Normal per-cpu:
[962841.266266] CPU    0: hi:  186, btch:  31 usd:   0
[962841.266272] CPU    1: hi:  186, btch:  31 usd: 167
[962841.266289] active_anon:60865 inactive_anon:15547 isolated_anon:0
[962841.266293]  active_file:221296 inactive_file:1249491 isolated_file:0
[962841.266297]  unevictable:0 dirty:4156 writeback:0 unstable:0
[962841.266300]  free:47696 slab_reclaimable:325469 slab_unreclaimable:6023
[962841.266304]  mapped:8585 shmem:1955 pagetables:4954 bounce:0
[962841.266327] DMA free:15904kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB

active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15648kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

[962841.266344] lowmem_reserve[]: 0 2603 7628 7628
[962841.266371] DMA32 free:102056kB min:23020kB low:28772kB high:34528kB active_anon:20256kB

inactive_anon:19164kB active_file:160472kB inactive_file:1952944kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2666288kB mlocked:0kB dirty:5804kB writeback:0kB mapped:8052kB shmem:4192kB slab_reclaimable:400280kB slab_unreclaimable:3080kB kernel_stack:104kB pagetables:1400kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

[962841.266391] lowmem_reserve[]: 0 0 5024 5024
[962841.266417] Normal free:72824kB min:44424kB low:55528kB high:66636kB active_anon:223204kB

inactive_anon:43024kB active_file:724712kB inactive_file:3045020kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5144828kB mlocked:0kB dirty:10820kB writeback:0kB mapped:26288kB shmem:3628kB slab_reclaimable:901596kB slab_unreclaimable:21012kB kernel_stack:2640kB pagetables:18416kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:135 all_unreclaimable? no

[962841.266438] lowmem_reserve[]: 0 0 0 0
[962841.266448] DMA: 0*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB
3*4096kB = 15904kB
[962841.266475] DMA32: 18658*4kB 2005*8kB 437*16kB 112*32kB 13*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 102080kB
[962841.266501] Normal: 8194*4kB 4068*8kB 395*16kB 31*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 72824kB
[962841.266526] 1472725 total pagecache pages
[962841.266532] 0 pages in swap cache
[962841.266538] Swap cache stats: add 0, delete 0, find 0/0
[962841.266544] Free swap  = 7846908kB
[962841.266549] Total swap = 7846908kB
[962841.381943] 1994736 pages RAM
[962841.381951] 53376 pages reserved
[962841.381956] 1122458 pages shared
[962841.381961] 819713 pages non-shared

Quoted 67 lines

I've been seeing these messages on my AMD Fusion server running linux-3.3.7-64bit. Does this ring any bells for anyone else? The system is NOT low on memory.

I'm building/installing 3.4.2 now to see if it behaves any better.

[962841.265658] mount.nfs: page allocation failure: order:4, mode:0xc0d0
[962841.265674] Pid: 32116, comm: mount.nfs Not tainted 3.3.7 #2
[962841.265680] Call Trace:
[962841.265700]  [<ffffffff81079363>] ? warn_alloc_failed+0x11a/0x12d
[962841.265713]  [<ffffffff8107b904>] ? __alloc_pages_nodemask+0x6c0/0x702
[962841.265725]  [<ffffffff8114fcb8>] ? timerqueue_del+0x53/0x63
[962841.265758]  [<ffffffff8107b9ba>] ? __get_free_pages+0x10/0x3f
[962841.265805]  [<ffffffffa01ea33d>] ? nfs_idmap_new+0x28/0xde [nfs]
[962841.265836]  [<ffffffffa01c79c9>] ? nfs4_init_client+0x74/0x12a [nfs]
[962841.265863]  [<ffffffffa01c6f3b>] ? nfs_get_client+0x337/0x476 [nfs]
[962841.265874]  [<ffffffff81087ad8>] ? pcpu_alloc+0x796/0x7ad
[962841.265885]  [<ffffffff810403f7>] ? should_resched+0x5/0x23
[962841.265913]  [<ffffffffa01c70f2>] ? nfs4_set_client+0x78/0xca [nfs]
[962841.265944]  [<ffffffffa01c7b5c>] ? nfs4_create_server+0xdd/0x208 [nfs]
[962841.265956]  [<ffffffff812c0de1>] ? _cond_resched+0x7/0x1c
[962841.265990]  [<ffffffffa01cfa3c>] ? nfs4_remote_mount+0x50/0x183 [nfs]
[962841.266007]  [<ffffffff810ab334>] ? mount_fs+0xc/0xa0
[962841.266020]  [<ffffffff810bd3fd>] ? vfs_kern_mount+0x61/0xcc
[962841.266053]  [<ffffffffa01cfc3f>] ? nfs_do_root_mount+0x89/0xaa [nfs]
[962841.266088]  [<ffffffffa01d0702>] ? nfs4_try_mount.isra.27+0x4c/0x9e [nfs]
[962841.266120]  [<ffffffffa01d1145>] ? nfs_fs_mount+0x3e9/0x5ef [nfs]
[962841.266131]  [<ffffffff8114b3bc>] ? ida_get_new_above+0x176/0x198
[962841.266147]  [<ffffffff810ab334>] ? mount_fs+0xc/0xa0
[962841.266159]  [<ffffffff810bd3fd>] ? vfs_kern_mount+0x61/0xcc
[962841.266172]  [<ffffffff810bd7f0>] ? do_kern_mount+0x49/0xd8
[962841.266182]  [<ffffffff810beec8>] ? do_mount+0x630/0x695
[962841.266191]  [<ffffffff810be80a>] ? copy_mount_options+0xc3/0x126
[962841.266201]  [<ffffffff810bf00f>] ? sys_mount+0x88/0xca
[962841.266212]  [<ffffffff812c20a2>] ? system_call_fastpath+0x16/0x1b
[962841.266219] Mem-Info:
[962841.266224] DMA per-cpu:
[962841.266230] CPU    0: hi:    0, btch:   1 usd:   0
[962841.266237] CPU    1: hi:    0, btch:   1 usd:   0
[962841.266242] DMA32 per-cpu:
[962841.266248] CPU    0: hi:  186, btch:  31 usd:   0
[962841.266255] CPU    1: hi:  186, btch:  31 usd:   0
[962841.266260] Normal per-cpu:
[962841.266266] CPU    0: hi:  186, btch:  31 usd:   0
[962841.266272] CPU    1: hi:  186, btch:  31 usd: 167
[962841.266289] active_anon:60865 inactive_anon:15547 isolated_anon:0
[962841.266293]  active_file:221296 inactive_file:1249491 isolated_file:0
[962841.266297]  unevictable:0 dirty:4156 writeback:0 unstable:0
[962841.266300]  free:47696 slab_reclaimable:325469 slab_unreclaimable:6023
[962841.266304]  mapped:8585 shmem:1955 pagetables:4954 bounce:0
[962841.266327] DMA free:15904kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB

active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15648kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

[962841.266344] lowmem_reserve[]: 0 2603 7628 7628
[962841.266371] DMA32 free:102056kB min:23020kB low:28772kB high:34528kB active_anon:20256kB

inactive_anon:19164kB active_file:160472kB inactive_file:1952944kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2666288kB mlocked:0kB dirty:5804kB writeback:0kB mapped:8052kB shmem:4192kB slab_reclaimable:400280kB slab_unreclaimable:3080kB kernel_stack:104kB pagetables:1400kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

[962841.266391] lowmem_reserve[]: 0 0 5024 5024
[962841.266417] Normal free:72824kB min:44424kB low:55528kB high:66636kB active_anon:223204kB

inactive_anon:43024kB active_file:724712kB inactive_file:3045020kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5144828kB mlocked:0kB dirty:10820kB writeback:0kB mapped:26288kB shmem:3628kB slab_reclaimable:901596kB slab_unreclaimable:21012kB kernel_stack:2640kB pagetables:18416kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:135 all_unreclaimable? no

Quoted 16 lines

[962841.266438] lowmem_reserve[]: 0 0 0 0
[962841.266448] DMA: 0*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB
3*4096kB = 15904kB
[962841.266475] DMA32: 18658*4kB 2005*8kB 437*16kB 112*32kB 13*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 102080kB
[962841.266501] Normal: 8194*4kB 4068*8kB 395*16kB 31*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 72824kB
[962841.266526] 1472725 total pagecache pages
[962841.266532] 0 pages in swap cache
[962841.266538] Swap cache stats: add 0, delete 0, find 0/0
[962841.266544] Free swap  = 7846908kB
[962841.266549] Total swap = 7846908kB
[962841.381943] 1994736 pages RAM
[962841.381951] 53376 pages reserved
[962841.381956] 1122458 pages shared
[962841.381961] 819713 pages non-shared

Looks like an old bug, perhaps:
https://bugzilla.redhat.com/show_bug.cgi?id=593035
https://bugzilla.redhat.com/show_bug.cgi?id=728003

I wonder what the underlying cause is?
And I also wonder if the NFS code should be more clever about handing order-4 allocation failures.

Adding Ben Hutchings to CC: as he seems to have looked into this before.

Quoted 16 lines · Mark Lord

Quoted 15 lines

I've been seeing these messages on my AMD Fusion server running linux-3.3.7-64bit. Does this ring any bells for anyone else? The system is NOT low on memory.

I'm building/installing 3.4.2 now to see if it behaves any better.

[962841.265658] mount.nfs: page allocation failure: order:4, mode:0xc0d0
[962841.265674] Pid: 32116, comm: mount.nfs Not tainted 3.3.7 #2
[962841.265680] Call Trace:
[962841.265700]  [<ffffffff81079363>] ? warn_alloc_failed+0x11a/0x12d
[962841.265713]  [<ffffffff8107b904>] ? __alloc_pages_nodemask+0x6c0/0x702
[962841.265725]  [<ffffffff8114fcb8>] ? timerqueue_del+0x53/0x63
[962841.265758]  [<ffffffff8107b9ba>] ? __get_free_pages+0x10/0x3f
[962841.265805]  [<ffffffffa01ea33d>] ? nfs_idmap_new+0x28/0xde [nfs]
[962841.265836]  [<ffffffffa01c79c9>] ? nfs4_init_client+0x74/0x12a [nfs]

Quoted 7 lines

Looks like an old bug, perhaps:
https://bugzilla.redhat.com/show_bug.cgi?id=593035
https://bugzilla.redhat.com/show_bug.cgi?id=728003

I wonder what the underlying cause is?
And I also wonder if the NFS code should be more clever about handing order-4 allocation failures.

More references:
http://www.mail-archive.com/[email protected]/msg71281.html
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1004619
http://codemonkey.org.uk/2012/02/17/fedora-16-kernel-bugzilla-status-report-20120210-20120217/
https://lkml.org/lkml/2011/2/24/73

Quoted 18 lines · Mark Lord

Adding Ben Hutchings to CC: as he seems to have looked into this before.

Quoted 16 lines · Mark Lord

Quoted 15 lines

I've been seeing these messages on my AMD Fusion server running linux-3.3.7-64bit. Does this ring any bells for anyone else? The system is NOT low on memory.

I'm building/installing 3.4.2 now to see if it behaves any better.

[962841.265658] mount.nfs: page allocation failure: order:4, mode:0xc0d0
[962841.265674] Pid: 32116, comm: mount.nfs Not tainted 3.3.7 #2
[962841.265680] Call Trace:
[962841.265700]  [<ffffffff81079363>] ? warn_alloc_failed+0x11a/0x12d
[962841.265713]  [<ffffffff8107b904>] ? __alloc_pages_nodemask+0x6c0/0x702
[962841.265725]  [<ffffffff8114fcb8>] ? timerqueue_del+0x53/0x63
[962841.265758]  [<ffffffff8107b9ba>] ? __get_free_pages+0x10/0x3f
[962841.265805]  [<ffffffffa01ea33d>] ? nfs_idmap_new+0x28/0xde [nfs]
[962841.265836]  [<ffffffffa01c79c9>] ? nfs4_init_client+0x74/0x12a [nfs]

...

Quoted 8 lines

Quoted 7 lines

Looks like an old bug, perhaps:
https://bugzilla.redhat.com/show_bug.cgi?id=593035
https://bugzilla.redhat.com/show_bug.cgi?id=728003

I wonder what the underlying cause is?
And I also wonder if the NFS code should be more clever about handing order-4 allocation failures.

Quoted 5 lines

More references:
http://www.mail-archive.com/[email protected]/msg71281.html
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1004619
http://codemonkey.org.uk/2012/02/17/fedora-16-kernel-bugzilla-status-report-20120210-20120217/
https://lkml.org/lkml/2011/2/24/73

Ahh.. this one seems to explain it, kind of:

   http://permalink.gmane.org/gmane.linux.nfs/41686

So in 3.3.7, I didn't have CONFIG_NFS_USE_NEW_IDMAPPER enabled, and the kernel seems to be buggy without that.

In 3.4.2, that option has disappeared entirely, most likely because it is now the default behaviour. Right?

Quoted 34 lines · Mark Lord

Quoted 18 lines · Mark Lord

Adding Ben Hutchings to CC: as he seems to have looked into this before.

Quoted 16 lines · Mark Lord

Quoted 15 lines

I've been seeing these messages on my AMD Fusion server running linux-3.3.7-64bit. Does this ring any bells for anyone else? The system is NOT low on memory.

I'm building/installing 3.4.2 now to see if it behaves any better.

[962841.265658] mount.nfs: page allocation failure: order:4, mode:0xc0d0
[962841.265674] Pid: 32116, comm: mount.nfs Not tainted 3.3.7 #2
[962841.265680] Call Trace:
[962841.265700]  [<ffffffff81079363>] ? warn_alloc_failed+0x11a/0x12d
[962841.265713]  [<ffffffff8107b904>] ? __alloc_pages_nodemask+0x6c0/0x702
[962841.265725]  [<ffffffff8114fcb8>] ? timerqueue_del+0x53/0x63
[962841.265758]  [<ffffffff8107b9ba>] ? __get_free_pages+0x10/0x3f
[962841.265805]  [<ffffffffa01ea33d>] ? nfs_idmap_new+0x28/0xde [nfs]
[962841.265836]  [<ffffffffa01c79c9>] ? nfs4_init_client+0x74/0x12a [nfs]

...

Quoted 8 lines

Quoted 7 lines

Looks like an old bug, perhaps:
https://bugzilla.redhat.com/show_bug.cgi?id=593035
https://bugzilla.redhat.com/show_bug.cgi?id=728003

I wonder what the underlying cause is?
And I also wonder if the NFS code should be more clever about handing order-4 allocation failures.

Quoted 5 lines

More references:
http://www.mail-archive.com/[email protected]/msg71281.html
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1004619
http://codemonkey.org.uk/2012/02/17/fedora-16-kernel-bugzilla-status-report-20120210-20120217/
https://lkml.org/lkml/2011/2/24/73

Quoted 8 lines

Ahh.. this one seems to explain it, kind of:

   http://permalink.gmane.org/gmane.linux.nfs/41686

So in 3.3.7, I didn't have CONFIG_NFS_USE_NEW_IDMAPPER enabled, and the kernel seems to be buggy without that.

In 3.4.2, that option has disappeared entirely, most likely because it is now the default behaviour. Right?

Yes, but we also added some patches to Linux 3.4 in order to fix this bug.

Please see commit d073e9b541e1ac3f52d72c3a153855d9a9ee3278 (NFSv4:
Reduce the footprint of the idmapper) and commit 685f50f9188ac1e8244d0340a9d6ea36b6136cec (NFSv4: Further reduce the footprint of the idmapper)

Looks like an old bug, perhaps:
https://bugzilla.redhat.com/show_bug.cgi?id=593035
https://bugzilla.redhat.com/show_bug.cgi?id=728003

I wonder what the underlying cause is?
And I also wonder if the NFS code should be more clever about handing order-4 allocation failures.

From memory, I don't think we've seen new reports of this since we switched to (now dead)