Skip to contentSkip to conversation

Errors writing Large files to CIFS shares

DISCUSSION Neil Sikka 3 participants 4 messages Nov 5, 2013 – Nov 6, 2013 5 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): mailman.1277.1383636787.1861.linux-cifs-client-w/[email protected]

Neil Sikka

author
#2 lore raw

Hello. I am running Debian 7.2, and I get errors writing "large" (32.5KB) files to shares mounted via CIFS. Writing "small" files to the remote share works. I am not sure if its relevant, but this share is mounted over a slow pptp connection. Here are some of the details of what I am running:

3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux

The Kernel log contains the following data:

[ 2243.047442] CIFS VFS: Server <IP_REMOVED> has not responded in 300
seconds. Reconnecting...
[ 2263.693713] CIFS VFS: sends on sock ffff88020e477d40 stuck for 15 seconds
[ 2263.693721] CIFS VFS: Error -11 sending data on socket to server
[ 2279.434990] INFO: task cp:4807 blocked for more than 120 seconds.
[ 2279.434996] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2279.435000] cp              D ffff88023fc93780     0  4807   4412 0x00000000
[ 2279.435009]  ffff8802336281c0 0000000000000086 ffff8802239067f0
ffff8802333a5650
[ 2279.435017]  0000000000013780 ffff880235ba7fd8 ffff880235ba7fd8
ffff8802336281c0
[ 2279.435025]  ffffffff810135d2 ffffffff81066245 ffff880233348d70
ffff88023fc93fd0
[ 2279.435033] Call Trace:
[ 2279.435045]  [<ffffffff810135d2>] ? read_tsc+0x5/0x14
[ 2279.435052]  [<ffffffff81066245>] ? timekeeping_get_ns+0xd/0x2a
[ 2279.435060]  [<ffffffff810b4afb>] ? lock_page+0x20/0x20
[ 2279.435067]  [<ffffffff8134e141>] ? io_schedule+0x59/0x71
[ 2279.435073]  [<ffffffff810b4b01>] ? sleep_on_page+0x6/0xa
[ 2279.435078]  [<ffffffff8134e584>] ? __wait_on_bit+0x3e/0x71
[ 2279.435085]  [<ffffffff810b4c3d>] ? wait_on_page_bit+0x6e/0x73
[ 2279.435093]  [<ffffffff8105fcad>] ? autoremove_wake_function+0x2a/0x2a
[ 2279.435100]  [<ffffffff810b4d15>] ? filemap_fdatawait_range+0x74/0x139
[ 2279.435108]  [<ffffffff810b64c9>] ? filemap_write_and_wait+0x24/0x30
[ 2279.435124]  [<ffffffffa063d3ce>] ? cifs_flush+0x20/0x50 [cifs]
[ 2279.435132]  [<ffffffff810f922d>] ? filp_close+0x3b/0x6a
[ 2279.435137]  [<ffffffff810f92ea>] ? sys_close+0x8e/0xcb
[ 2279.435143]  [<ffffffff81354212>] ? system_call_fastpath+0x16/0x1b
[ 2294.192963] CIFS VFS: sends on sock ffff88020e477d40 stuck for 15 seconds
[ 2294.192971] CIFS VFS: Error -11 sending data on socket to server

I mounted the CIFS share using the following command:
mount -t cifs //<IP_REMOVED>/share /mnt/net2/ -o username=<USERNAME_REMOVED>

I see many other people had this problem in the past (almost 4 years ago). A similar description can be found here:

https://bugzilla.redhat.com/show_bug.cgi?id=522867

That bug was marked as WONTFIX

Was there ever a workaround or resolution for this bug? I am obviously not running software from 2009 (like in that bugzilla post), so I would have thought this would have been resolved. Could this bug be a regression?

Thanks.

Signature · 2 lines
My Blog: http://www.neilscomputerblog.blogspot.com/
Twitter: @neilsikka

A message that is not in the archive (sent off-list or not yet received): CAHPMNWdLy7cfJdRGp=2-pUQqFO=PgUYT8E2_uW9koNdv1x_D_g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#4 lore raw
Quoted 8 lines · On Tue, 5 Nov 2013 10:23:41 -0800 Neil Sikka…

Hello. I am running Debian 7.2, and I get errors writing "large" (32.5KB) files to shares mounted via CIFS. Writing "small" files to the remote share works. I am not sure if its relevant, but this share is mounted over a slow pptp connection. Here are some of the details of what I am running:

3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux

3.2 is ancient history now ;)

Quoted 6 lines

The Kernel log contains the following data:

[ 2243.047442] CIFS VFS: Server <IP_REMOVED> has not responded in 300
seconds. Reconnecting...
[ 2263.693713] CIFS VFS: sends on sock ffff88020e477d40 stuck for 15 seconds
[ 2263.693721] CIFS VFS: Error -11 sending data on socket to server

Looks like you're losing connectivity to the server. I'd have a hard look at your network connectivity. Maybe sniff traffic and see if you can figure out what's happening on the wire.

Quoted 45 lines
[ 2279.434990] INFO: task cp:4807 blocked for more than 120 seconds.
[ 2279.434996] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2279.435000] cp              D ffff88023fc93780     0  4807   4412 0x00000000
[ 2279.435009]  ffff8802336281c0 0000000000000086 ffff8802239067f0
ffff8802333a5650
[ 2279.435017]  0000000000013780 ffff880235ba7fd8 ffff880235ba7fd8
ffff8802336281c0
[ 2279.435025]  ffffffff810135d2 ffffffff81066245 ffff880233348d70
ffff88023fc93fd0
[ 2279.435033] Call Trace:
[ 2279.435045]  [<ffffffff810135d2>] ? read_tsc+0x5/0x14
[ 2279.435052]  [<ffffffff81066245>] ? timekeeping_get_ns+0xd/0x2a
[ 2279.435060]  [<ffffffff810b4afb>] ? lock_page+0x20/0x20
[ 2279.435067]  [<ffffffff8134e141>] ? io_schedule+0x59/0x71
[ 2279.435073]  [<ffffffff810b4b01>] ? sleep_on_page+0x6/0xa
[ 2279.435078]  [<ffffffff8134e584>] ? __wait_on_bit+0x3e/0x71
[ 2279.435085]  [<ffffffff810b4c3d>] ? wait_on_page_bit+0x6e/0x73
[ 2279.435093]  [<ffffffff8105fcad>] ? autoremove_wake_function+0x2a/0x2a
[ 2279.435100]  [<ffffffff810b4d15>] ? filemap_fdatawait_range+0x74/0x139
[ 2279.435108]  [<ffffffff810b64c9>] ? filemap_write_and_wait+0x24/0x30
[ 2279.435124]  [<ffffffffa063d3ce>] ? cifs_flush+0x20/0x50 [cifs]
[ 2279.435132]  [<ffffffff810f922d>] ? filp_close+0x3b/0x6a
[ 2279.435137]  [<ffffffff810f92ea>] ? sys_close+0x8e/0xcb
[ 2279.435143]  [<ffffffff81354212>] ? system_call_fastpath+0x16/0x1b
[ 2294.192963] CIFS VFS: sends on sock ffff88020e477d40 stuck for 15 seconds
[ 2294.192971] CIFS VFS: Error -11 sending data on socket to server

I mounted the CIFS share using the following command:
mount -t cifs //<IP_REMOVED>/share /mnt/net2/ -o username=<USERNAME_REMOVED>

I see many other people had this problem in the past (almost 4 years ago). A similar description can be found here:

https://bugzilla.redhat.com/show_bug.cgi?id=522867

That bug was marked as WONTFIX

Was there ever a workaround or resolution for this bug? I am obviously not running software from 2009 (like in that bugzilla post), so I would have thought this would have been resolved. Could this bug be a regression?

Thanks.

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20131105142533.2b9bd00e-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

author
#6 lore raw

Here are the ping statistics where I starting pinging before I started a "cp" of a 32KB file and ending after VFS hung in the copy process:
170 packets transmitted, 170 received, 0% packet loss, time 169241ms rtt min/avg/max/mdev = 120.447/158.142/397.196/36.544 ms

The network connection seems fine--there were no dropped packets.

Although 3.2 is old, it is still more recent than the link i shared where they first reported this bug. I would have thought they would have fixed this bug or something. Does anyone on this list know the what happened to the bug reported
here (https://bugzilla.redhat.com/show_bug.cgi?id=522867)?

Quoted 74 lines · Jeff Layton
Quoted 8 lines · On Tue, 5 Nov 2013 10:23:41 -0800 Neil Sikka…

Hello. I am running Debian 7.2, and I get errors writing "large" (32.5KB) files to shares mounted via CIFS. Writing "small" files to the remote share works. I am not sure if its relevant, but this share is mounted over a slow pptp connection. Here are some of the details of what I am running:

3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux

3.2 is ancient history now ;)

Quoted 6 lines

The Kernel log contains the following data:

[ 2243.047442] CIFS VFS: Server <IP_REMOVED> has not responded in 300
seconds. Reconnecting...
[ 2263.693713] CIFS VFS: sends on sock ffff88020e477d40 stuck for 15 seconds
[ 2263.693721] CIFS VFS: Error -11 sending data on socket to server

Looks like you're losing connectivity to the server. I'd have a hard look at your network connectivity. Maybe sniff traffic and see if you can figure out what's happening on the wire.

Quoted 36 lines
[ 2279.434990] INFO: task cp:4807 blocked for more than 120 seconds.
[ 2279.434996] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2279.435000] cp              D ffff88023fc93780     0  4807   4412 0x00000000
[ 2279.435009]  ffff8802336281c0 0000000000000086 ffff8802239067f0
ffff8802333a5650
[ 2279.435017]  0000000000013780 ffff880235ba7fd8 ffff880235ba7fd8
ffff8802336281c0
[ 2279.435025]  ffffffff810135d2 ffffffff81066245 ffff880233348d70
ffff88023fc93fd0
[ 2279.435033] Call Trace:
[ 2279.435045]  [<ffffffff810135d2>] ? read_tsc+0x5/0x14
[ 2279.435052]  [<ffffffff81066245>] ? timekeeping_get_ns+0xd/0x2a
[ 2279.435060]  [<ffffffff810b4afb>] ? lock_page+0x20/0x20
[ 2279.435067]  [<ffffffff8134e141>] ? io_schedule+0x59/0x71
[ 2279.435073]  [<ffffffff810b4b01>] ? sleep_on_page+0x6/0xa
[ 2279.435078]  [<ffffffff8134e584>] ? __wait_on_bit+0x3e/0x71
[ 2279.435085]  [<ffffffff810b4c3d>] ? wait_on_page_bit+0x6e/0x73
[ 2279.435093]  [<ffffffff8105fcad>] ? autoremove_wake_function+0x2a/0x2a
[ 2279.435100]  [<ffffffff810b4d15>] ? filemap_fdatawait_range+0x74/0x139
[ 2279.435108]  [<ffffffff810b64c9>] ? filemap_write_and_wait+0x24/0x30
[ 2279.435124]  [<ffffffffa063d3ce>] ? cifs_flush+0x20/0x50 [cifs]
[ 2279.435132]  [<ffffffff810f922d>] ? filp_close+0x3b/0x6a
[ 2279.435137]  [<ffffffff810f92ea>] ? sys_close+0x8e/0xcb
[ 2279.435143]  [<ffffffff81354212>] ? system_call_fastpath+0x16/0x1b
[ 2294.192963] CIFS VFS: sends on sock ffff88020e477d40 stuck for 15 seconds
[ 2294.192971] CIFS VFS: Error -11 sending data on socket to server

I mounted the CIFS share using the following command:
mount -t cifs //<IP_REMOVED>/share /mnt/net2/ -o username=<USERNAME_REMOVED>

I see many other people had this problem in the past (almost 4 years ago). A similar description can be found here:

https://bugzilla.redhat.com/show_bug.cgi?id=522867

Quoted 9 lines

That bug was marked as WONTFIX

Was there ever a workaround or resolution for this bug? I am obviously not running software from 2009 (like in that bugzilla post), so I would have thought this would have been resolved. Could this bug be a regression?

Thanks.

--
Jeff Layton <[email protected]>

Signature · 2 lines
My Blog: http://www.neilscomputerblog.blogspot.com/
Twitter: @neilsikka

A message that is not in the archive (sent off-list or not yet received): CAHPMNWcY05L+ikBDmOb9SaVM5nA24rY00Hfjy3zjowmaE2p1+g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#8 lore raw
Quoted 5 lines · On Tue, 5 Nov 2013 20:34:52 -0800 Neil Sikka…

Here are the ping statistics where I starting pinging before I started a "cp" of a 32KB file and ending after VFS hung in the copy process:
170 packets transmitted, 170 received, 0% packet loss, time 169241ms rtt min/avg/max/mdev = 120.447/158.142/397.196/36.544 ms

Quoted 10 lines

The network connection seems fine--there were no dropped packets.

Although 3.2 is old, it is still more recent than the link i shared where they first reported this bug. I would have thought they would have fixed this bug or something. Does anyone on this list know the what happened to the bug reported
here (https://bugzilla.redhat.com/show_bug.cgi?id=522867)?

The bug got autoclosed because no one bothered to keep it going (see comment #5 in that bug).

comment #4 in that bug strongly suggests that the problem was with the lower-level networking and not really with cifs, per-se. Your problem may or may not be different. You'll have to do some investigation to be sure.

cifs.ko's usage of the networking layer is very userland-like. It really doesn't do much special there. You have these lines in the log you posted:

[ 2294.192963] CIFS VFS: sends on sock ffff88020e477d40 stuck for 15 seconds
[ 2294.192971] CIFS VFS: Error -11 sending data on socket to server

Basically that means that kernel_sendmsg is continually returning -EAGAIN. That likely means that the socket send buffer is full, typically because the server can't be reached or isn't ACK'ing TCP packets.

If pinging works ok, then your problem may be at the TCP layer. Possibly a problem related to bufferbloat given that you are running this over a long-haul link:

http://en.wikipedia.org/wiki/Bufferbloat

...but it could be something else entirely.

Signature · 1 line
Jeff Layton <[email protected]>