Skip to contentSkip to conversation

NFS client large rsize/wsize (tcp?) problems

DISCUSSION Erik Slagter 3 participants 6 messages Dec 30, 2012 – Jan 2, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello All,

I am almost complete NOOB on this matter, so please be gentle ;-) I do believe there is some sort of problem inside the NFS code though.

Background: I have:
- linux server x86_64, vanilla kernel 3.6.7 sharing a few exports
- several set-top-boxes running linux, arch is mipsel32, they're
almost vanilla, but they have prioprietary closed source drivers for the DVB frontends:
* MaxDigital XP1000, kernel 3.5.1
* DMM DM8000, kernel 3.2.0
* VU+ Ultimo, kernel 3.1.1

These all suffer from the same problem. When they have a share mounted with default parameters, using tcp, they crash sooner or later, notably after heavy share access. The dm8000 has it's gui killed by the OOM-killer whilst the xp1000 and the ultimo simply lock up.

The OOM-killer reports it needs blocks of 128k (probably for NFS, but it doesn't say it), but can't find them. For the other stb'es it's not clear as they lock up.

I've "discovered" a few interesting things:
- adding swap to the dm8000 makes the problem almost go away, although
without NFS it definitely doesn't need swap, ever.
- when I ran my laptop (x86_64!) with a slightly older kernel (2.6.35
iirc) from a rescue cd, at a certain point I also got nasty dmesg reports and the "dd" proces got stuck in D state, this was reproducable over reboots.
- all clients work flawlessly (over extended perdiods of time) if
mounted using udp and smaller rsize/wsize values (max 32k). Tcp seems to work as well, as long as the size values are kept under 32k.
- the x86_64 laptop also worked fine when mounted this way
- so apparently it's not a stb/mipsel32/proprietary driver issue.
- stb's running older kernels (notably 2.6.18) don't suffer from this
problem

Can please anyone enlighten me? I can't find similar reports other than from fellow stb users.

Thanks!

#1 lore raw
Quoted 3 lines · Erik Slagter

Hello All,

I am almost complete NOOB on this matter, so please be gentle ;-)

Hah! Hah! Hah!

Quoted 12 lines

I
do believe there is some sort of problem inside the NFS code though.

Background: I have:
- linux server x86_64, vanilla kernel 3.6.7 sharing a few exports
- several set-top-boxes running linux, arch is mipsel32, they're
almost vanilla, but they have prioprietary closed source drivers for the DVB frontends:
* MaxDigital XP1000, kernel 3.5.1
* DMM DM8000, kernel 3.2.0
* VU+ Ultimo, kernel 3.1.1

Quoted 8 lines

These all suffer from the same problem. When they have a share mounted with default parameters, using tcp, they crash sooner or later, notably after heavy share access. The dm8000 has it's gui killed by the OOM-killer whilst the xp1000 and the ultimo simply lock up.

The OOM-killer reports it needs blocks of 128k (probably for NFS, but it doesn't say it), but can't find them.

Details? (Could you show us the log messages?) Anything else interesting in the logs before then? (E.g. any "order-n allocation failed" messages?)

Quoted 10 lines

For the other stb'es
it's not clear as they lock up.

I've "discovered" a few interesting things:
- adding swap to the dm8000 makes the problem almost go away,
although without NFS it definitely doesn't need swap, ever.
- when I ran my laptop (x86_64!) with a slightly older kernel
(2.6.35 iirc) from a rescue cd, at a certain point I also got nasty dmesg reports and the "dd" proces got stuck in D state, this was reproducable over reboots.

Why do you believe that's the same problem?

Quoted 8 lines

- all clients work flawlessly (over extended perdiods of time) if
mounted using udp and smaller rsize/wsize values (max 32k). Tcp seems to work as well, as long as the size values are kept under 32k.
- the x86_64 laptop also worked fine when mounted this way
- so apparently it's not a stb/mipsel32/proprietary driver issue.
- stb's running older kernels (notably 2.6.18) don't suffer from
this problem

OK, thanks for the reports, let us know i you're able to narrow it down farther. It's not familiar off the top of my head.

--b.

Quoted 9 lines

Can please anyone enlighten me? I can't find similar reports other than from fellow stb users.

Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 6 lines · J. Bruce Fields
Quoted 2 lines

The OOM-killer reports it needs blocks of 128k (probably for NFS, but it doesn't say it), but can't find them.

Details? (Could you show us the log messages?) Anything else interesting in the logs before then? (E.g. any "order-n allocation failed" messages?)

Hmmm, that will be tricky. The one box that produces OOM-messages has this after about a week of usage, and they only log in memory :-(

Ah, I've found one!

Quoted 64 lines

enigma2 invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0

Call Trace:
[<80485708>] dump_stack+0x8/0x34
[<80081f60>] dump_header.isra.9+0x88/0x1a4
[<80082268>] oom_kill_process.constprop.16+0xc4/0x2b8
[<800828c4>] out_of_memory+0x2a8/0x3a8
[<80085e78>] __alloc_pages_nodemask+0x640/0x654
[<8048683c>] cache_alloc_refill+0x350/0x668
[<800b1f10>] kmem_cache_alloc+0xe0/0x104
[<80185360>] nfs_create_request+0x40/0x178
[<80187544>] readpage_async_filler+0x9c/0x1bc
[<80089b98>] read_cache_pages+0xe4/0x144
[<801886ac>] nfs_readpages+0xd4/0x1cc
[<80089928>] __do_page_cache_readahead+0x218/0x2e4
[<80089d58>] ra_submit+0x28/0x34
[<8008a138>] page_cache_sync_readahead+0x48/0x70
[<80080ae0>] generic_file_aio_read+0x55c/0x858
[<80179560>] nfs_file_read+0xac/0x194
[<800b5004>] do_sync_read+0xb8/0x120
[<800b5ca0>] vfs_read+0xa0/0x180
[<800b5dcc>] sys_read+0x4c/0x90
[<8000c61c>] stack_done+0x20/0x40

Mem-Info:
Normal per-cpu:
CPU 0: hi: 90, btch: 15 usd: 14
CPU 1: hi: 90, btch: 15 usd: 0
active_anon:22459 inactive_anon:57 isolated_anon:0
active_file:972 inactive_file:1968 isolated_file:0
unevictable:0 dirty:0 writeback:144 unstable:0
free:501 slab_reclaimable:526 slab_unreclaimable:2701
mapped:686 shmem:142 pagetables:137 bounce:0
Normal free:2004kB min:2036kB low:2544kB high:3052kB active_anon:89836kB inactive_anon:228kB active_file:3888kB inactive_file:7872kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:260096kB mlocked:0kB dirty:0kB writeback:576kB mapped:2744kB shmem:568kB slab_reclaimable:2104kB slab_unreclaimable:10804kB kernel_stack:792kB pagetables:548kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:14594 all_unreclaimable? yes lowmem_reserve[]: 0 0
Normal: 317*4kB 90*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2004kB
3101 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
65536 pages RAM
28149 pages reserved
3039 pages shared
33680 pages non-shared
[ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name

[  254]     0   254      474       16   1       0             0 wdog
[  263]     0   263     1225       88   0       0             0 tpmd
[  327]     0   327     1026      255   1       0             0 nmbd
[  329]     0   329     1803      175   1       0             0 smbd
[  349]     0   349     1803      175   0       0             0 smbd
[  372]     1   372      499       19   1       0             0 portmap
[  383]   998   383      762       37   1       0             0 dbus-daemon
[  387]     0   387      666       24   1       0             0 dropbear
[  392]     0   392      664       48   0       0             0 crond
[  398]     0   398      758       22   1       0             0 inetd
[  401]     0   401      664       35   1       0             0 syslogd
[  403]     0   403      664       52   0       0             0 klogd
[  410]   997   410      922       95   1       0             0 avahi-daemon
[  411]   997   411      922       42   0       0             0 avahi-daemon
[ 7811] 65534  7811     7424      187   1       0             0 msgd
[ 7819]     0  7819     1266       45   0       0             0 oscam
[ 7820]     0  7820     6733     2491   1       0             0 oscam
[ 7821]     0  7821      664       16   1       0             0 enigma2.sh
[ 7828]     0  7828    44920    19651   1       0             0 enigma2
Quoted 2 lines

Out of memory: Kill process 7828 (enigma2) score 496 or sacrifice child Killed process 7828 (enigma2) total-vm:179680kB, anon-rss:77180kB, file-rss:1424kB

The other boxes simply lock up.

This does NOT happen with NFS mounted using smaller buffers!

Quoted 9 lines
Quoted 7 lines

I've "discovered" a few interesting things:
- adding swap to the dm8000 makes the problem almost go away,
although without NFS it definitely doesn't need swap, ever.
- when I ran my laptop (x86_64!) with a slightly older kernel
(2.6.35 iirc) from a rescue cd, at a certain point I also got nasty dmesg reports and the "dd" proces got stuck in D state, this was reproducable over reboots.

Why do you believe that's the same problem?

Because all are solved with smaller nfs mount buffers. That is as much as I understand.

Quoted 2 lines

OK, thanks for the reports, let us know i you're able to narrow it down farther. It's not familiar off the top of my head.

Okay, at least it's good to know it's not a known problem with a known solution / workaround. I hope the kernel message helps.

As a temporary workaround (for "dumb users" that don't know what a mount option is, yes it's awful!) I'd like to modify the kernel of the clients to negotiate a smaller buffer size, 32k would probably suffice. I've had a few shots but have not been successful yet, can you give me a pointer please?

Thanks!

#3 lore raw
Quoted 81 lines · Erik Slagter
Quoted 6 lines · J. Bruce Fields
Quoted 2 lines

The OOM-killer reports it needs blocks of 128k (probably for NFS, but it doesn't say it), but can't find them.

Details? (Could you show us the log messages?) Anything else interesting in the logs before then? (E.g. any "order-n allocation failed" messages?)

Hmmm, that will be tricky. The one box that produces OOM-messages has this after about a week of usage, and they only log in memory :-(

Ah, I've found one!

Quoted 64 lines

enigma2 invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0

Call Trace:

[<80485708>] dump_stack+-0x8/0x34 [<80081f60>] dump_header.isra.9+-0x88/0x1a4 [<80082268>] oom_kill_process.constprop.16+-0xc4/0x2b8 [<800828c4>] out_of_memory+-0x2a8/0x3a8 [<80085e78>] __alloc_pages_nodemask+-0x640/0x654 [<8048683c>] cache_alloc_refill+-0x350/0x668 [<800b1f10>] kmem_cache_alloc+-0xe0/0x104 [<80185360>] nfs_create_request+-0x40/0x178 [<80187544>] readpage_async_filler+-0x9c/0x1bc [<80089b98>] read_cache_pages+-0xe4/0x144 [<801886ac>] nfs_readpages+-0xd4/0x1cc [<80089928>] __do_page_cache_readahead+-0x218/0x2e4 [<80089d58>] ra_submit+-0x28/0x34 [<8008a138>] page_cache_sync_readahead+-0x48/0x70 [<80080ae0>] generic_file_aio_read+-0x55c/0x858 [<80179560>] nfs_file_read+-0xac/0x194 [<800b5004>] do_sync_read+-0xb8/0x120 [<800b5ca0>] vfs_read+-0xa0/0x180 [<800b5dcc>] sys_read+-0x4c/0x90
[<8000c61c>] stack_done+-0x20/0x40

Mem-Info:
Normal per-cpu:
CPU 0: hi: 90, btch: 15 usd: 14
CPU 1: hi: 90, btch: 15 usd: 0
active_anon:22459 inactive_anon:57 isolated_anon:0
active_file:972 inactive_file:1968 isolated_file:0
unevictable:0 dirty:0 writeback:144 unstable:0
free:501 slab_reclaimable:526 slab_unreclaimable:2701
mapped:686 shmem:142 pagetables:137 bounce:0
Normal free:2004kB min:2036kB low:2544kB high:3052kB active_anon:89836kB inactive_anon:228kB active_file:3888kB inactive_file:7872kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:260096kB mlocked:0kB dirty:0kB writeback:576kB mapped:2744kB shmem:568kB slab_reclaimable:2104kB slab_unreclaimable:10804kB kernel_stack:792kB pagetables:548kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:14594 all_unreclaimable? yes lowmem_reserve[]: 0 0
Normal: 317*4kB 90*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2004kB
3101 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
65536 pages RAM
28149 pages reserved
3039 pages shared
33680 pages non-shared
[ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name

[  254]     0   254      474       16   1       0             0 wdog
[  263]     0   263     1225       88   0       0             0 tpmd
[  327]     0   327     1026      255   1       0             0 nmbd
[  329]     0   329     1803      175   1       0             0 smbd
[  349]     0   349     1803      175   0       0             0 smbd
[  372]     1   372      499       19   1       0             0 portmap
[  383]   998   383      762       37   1       0             0 dbus-daemon
[  387]     0   387      666       24   1       0             0 dropbear
[  392]     0   392      664       48   0       0             0 crond
[  398]     0   398      758       22   1       0             0 inetd
[  401]     0   401      664       35   1       0             0 syslogd
[  403]     0   403      664       52   0       0             0 klogd
[  410]   997   410      922       95   1       0             0 avahi-daemon
[  411]   997   411      922       42   0       0             0 avahi-daemon
[ 7811] 65534  7811     7424      187   1       0             0 msgd
[ 7819]     0  7819     1266       45   0       0             0 oscam
[ 7820]     0  7820     6733     2491   1       0             0 oscam
[ 7821]     0  7821      664       16   1       0             0 enigma2.sh
[ 7828]     0  7828    44920    19651   1       0             0 enigma2
Quoted 2 lines

Out of memory: Kill process 7828 (enigma2) score 496 or sacrifice child Killed process 7828 (enigma2) total-vm:179680kB, anon-rss:77180kB, file-rss:1424kB

Quoted 3 lines

The other boxes simply lock up.

This does NOT happen with NFS mounted using smaller buffers!

You probably have a NIC that doesn't support scatter-gather.

Quoted 16 lines
Quoted 9 lines
Quoted 7 lines

I've "discovered" a few interesting things:
- adding swap to the dm8000 makes the problem almost go away,
although without NFS it definitely doesn't need swap, ever.
- when I ran my laptop (x86_64!) with a slightly older kernel
(2.6.35 iirc) from a rescue cd, at a certain point I also got nasty dmesg reports and the "dd" proces got stuck in D state, this was reproducable over reboots.

Why do you believe that's the same problem?

Because all are solved with smaller nfs mount buffers. That is as much as I understand.

Quoted 2 lines

OK, thanks for the reports, let us know i you're able to narrow it down farther. It's not familiar off the top of my head.

Quoted 9 lines

Okay, at least it's good to know it's not a known problem with a known solution / workaround. I hope the kernel message helps.

As a temporary workaround (for "dumb users" that don't know what a mount option is, yes it's awful!) I'd like to modify the kernel of the clients to negotiate a smaller buffer size, 32k would probably suffice. I've had a few shots but have not been successful yet, can you give me a pointer please?

man nfsmount.conf

Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com
#4 lore raw
Quoted 1 line · Trond

You probably have a NIC that doesn't support scatter-gather.

I am not 100% sure, but as it's a satellite set-top-box, with very basic ethernet connectivity, I'd say you're right on the spot.

Is there a way to workaround that?

Quoted 7 lines
Quoted 5 lines

As a temporary workaround (for "dumb users" that don't know what a mount option is, yes it's awful!) I'd like to modify the kernel of the clients to negotiate a smaller buffer size, 32k would probably suffice. I've had a few shots but have not been successful yet, can you give me a pointer please?

man nfsmount.conf

Thanks for the hint, I didn't know that! Unfortunately I can't use it, because the mount command on the stb is a "busybox" version, so it's very basic, and doesn't check this file (checked it using strace...)

#5 lore raw
Quoted 15 lines · Erik Slagter
Quoted 1 line · Trond

You probably have a NIC that doesn't support scatter-gather.

I am not 100% sure, but as it's a satellite set-top-box, with very basic ethernet connectivity, I'd say you're right on the spot.

Is there a way to workaround that?

Quoted 5 lines
Quoted 5 lines

As a temporary workaround (for "dumb users" that don't know what a mount option is, yes it's awful!) I'd like to modify the kernel of the clients to negotiate a smaller buffer size, 32k would probably suffice. I've had a few shots but have not been successful yet, can you give me a pointer please?

Quoted 5 lines
Quoted 1 line

man nfsmount.conf

Thanks for the hint, I didn't know that! Unfortunately I can't use it, because the mount command on the stb is a "busybox" version, so it's very basic, and doesn't check this file (checked it using strace...)

You can also configure this server-side by writing to /proc/fs/nfsd/max_block_size (or your distro may have some config file where that's set). But then of course it limits all clients whether they need the workaround or not.

--b.