Skip to contentSkip to conversation

"CIFS VFS: SMB response too short (9 bytes)" when copying data from CIFS mount

DISCUSSION Andrew Martin 2 participants 5 messages Sep 28, 2015 – Sep 30, 2015 6 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 1936200018.81178.1443479619664.JavaMail.zimbra-AQeFf1F/[email protected]

#2 lore raw

Hello,

I have tested this setup on Ubuntu 14.04 using 3.13.0-32-generic and
3.19.0-28-generic with the same results (cifs-utils 6.0 in both cases). I have a
CIFS mountpoint set up on the server with the following options:
ro,relatime,vers=1.0,cache=strict,username=myusername,domain=EXAMPLE,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.1.4,unix,posixpaths,serverino,acl,rsize=1048576,wsize=65536,actimeo=1 0 0

I am then using rsync to sync data from the mountpoint to a local directory. Shortly after starting rsync, I see dmesg fill up with this message repeated hundreds of times:
CIFS VFS: SMB response too short (9 bytes)

At this point, if I watch the df output I see that no new data is being synced locally. If I strace the rsync process, this sometimes makes rsync continue syncing data for a minute or two, but eventually it stops again. Note that I encountered this same issue a few months ago using perl:
http://permalink.gmane.org/gmane.linux.kernel.cifs/10640

What can I do to debug this? Is there a bug in the CIFS kernel driver?

Thanks,

Andrew Martin

A message that is not in the archive (sent off-list or not yet received): 1731839987.81462.1443479968995.JavaMail.zimbra-AQeFf1F/[email protected]

#4 lore raw

It is possible that it is a cifs client bug or a server bug but hard to tell without seeing the trace. Obviously the client message indicates that the SMB response from the server was invalid (malformed due to server bug) but hard to prove that without looking at a network trace. See the mention of wireshark at

https://wiki.samba.org/index.php/LinuxCIFS_troubleshooting

Also it may be better for many use cases to mount with SMB3 ("vers=3.0") rather than the ancient cifs dialect (unless the server is Samba where there are "posix extensions" that the client can take advantage of).

What is the server version?

Quoted 27 lines · Andrew Martin

Hello,

I have tested this setup on Ubuntu 14.04 using 3.13.0-32-generic and
3.19.0-28-generic with the same results (cifs-utils 6.0 in both cases). I have a
CIFS mountpoint set up on the server with the following options:
ro,relatime,vers=1.0,cache=strict,username=myusername,domain=EXAMPLE,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.1.4,unix,posixpaths,serverino,acl,rsize=1048576,wsize=65536,actimeo=1 0 0

I am then using rsync to sync data from the mountpoint to a local directory. Shortly after starting rsync, I see dmesg fill up with this message repeated hundreds of times:
CIFS VFS: SMB response too short (9 bytes)

At this point, if I watch the df output I see that no new data is being synced locally. If I strace the rsync process, this sometimes makes rsync continue syncing data for a minute or two, but eventually it stops again. Note that I encountered this same issue a few months ago using perl:
http://permalink.gmane.org/gmane.linux.kernel.cifs/10640

What can I do to debug this? Is there a bug in the CIFS kernel driver?

Thanks,

Andrew Martin
--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines
Thanks,

Steve

A message that is not in the archive (sent off-list or not yet received): CAH2r5mvsZJ3Nn45Fqjhc8nd-PZZ_KYJxsPg19v7DCpkuMb1OLg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#6 lore raw

----- Original Message -----

Quoted 15 lines

From: "Steve French" <[email protected]>
To: "Andrew Martin" <amartin-AQeFf1F/[email protected]>
Cc: [email protected]
Sent: Monday, September 28, 2015 6:32:51 PM
Subject: Re: "CIFS VFS: SMB response too short (9 bytes)" when copying data from CIFS mount

It is possible that it is a cifs client bug or a server bug but hard to tell without seeing the trace. Obviously the client message indicates that the SMB response from the server was invalid (malformed due to server bug) but hard to prove that without looking at a network trace. See the mention of wireshark at

https://wiki.samba.org/index.php/LinuxCIFS_troubleshooting

Quoted 7 lines

Also it may be better for many use cases to mount with SMB3 ("vers=3.0") rather than the ancient cifs dialect (unless the server is Samba where there are "posix extensions" that the client can take advantage of).

What is the server version?

Steve,

The server is Ubuntu 12.04 with Samba 3.6.3, so I am limited to vers=1.0. I attempted to pass vers=2.0 (since it looks like basic SMB2 support was available in Samba 3.6.3), however it failed to mount with this error:
mount error(112): Host is down
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Looking at the Wireshark capture when the "SMB response too short" message is being printed, I see the following queries sent before the malformed packet:

<client> <server> SMB 268 Trans2 Request, QUERY_PATH_INFO, Query File Unix Basic, Path: /path/to/file.txt
<server> <client> SMB 230 Trans2 Response, QUERY_PATH_INFO
<client> <server> SMB 129 Read AndX Request, FID: 0x3134, 131022 bytes at offset 262144
<server> <client> SMB 79 [Malformed Packet] - Expert Info (Error/Malformed): Malformed Packet (Exception occurred)

The server's samba log for the client during this period:
http://pastebin.com/Y0diqhGY

I also set /proc/fs/cifs/cifsFYI to 7 and captured the following output:
http://pastebin.com/CfPUtBQg

Note that the Samba share definition is as follows:
http://pastebin.com/0aVCaRJH

Thanks,

Andrew

A message that is not in the archive (sent off-list or not yet received): 1744736878.52548.1443542655076.JavaMail.zimbra-AQeFf1F/[email protected]

#8 lore raw

It sounds pretty straightforward if both wireshark and the Linux kernel client agree and say that this old Samba server sent a malformed response (presumably a CIFS ReadX response). Note that this is an old version of Samba server - Samba 3.6 was released more than 4 years ago.

Try turning off unix extensions on the mount and/or setting rsize smaller (which might change the behavior of the Samba server read response). So try
1) adding something like "rsize=60000" to the options you pass to the mount command
2) and as a different experiment try mounting with "nounix" to disable CIFS unix extensions which would also have the affect of reducing the maximum read size
3) another option to try is to set max protocols in the server's smb.conf to enable SMB2.0 support and try "vers=2.0" on mount

My guess is that the bug is related to large CIFS read processing in this old version of Samba.

I also want to --- strongly --- encourage upgrade from Samba 3.6.3 to at least Samba 4.1 (I use Samba 4.3). The additional security features and capabilities which SMB2.1 and SMB3 bring (not just the new protocol features and improved performance) are VERY helpful and very important for security.

Quoted 52 lines · Andrew Martin

----- Original Message -----

Quoted 15 lines

From: "Steve French" <[email protected]>
To: "Andrew Martin" <amartin-AQeFf1F/[email protected]>
Cc: [email protected]
Sent: Monday, September 28, 2015 6:32:51 PM
Subject: Re: "CIFS VFS: SMB response too short (9 bytes)" when copying data from CIFS mount

It is possible that it is a cifs client bug or a server bug but hard to tell without seeing the trace. Obviously the client message indicates that the SMB response from the server was invalid (malformed due to server bug) but hard to prove that without looking at a network trace. See the mention of wireshark at

https://wiki.samba.org/index.php/LinuxCIFS_troubleshooting

Quoted 7 lines

Also it may be better for many use cases to mount with SMB3 ("vers=3.0") rather than the ancient cifs dialect (unless the server is Samba where there are "posix extensions" that the client can take advantage of).

What is the server version?

Steve,

The server is Ubuntu 12.04 with Samba 3.6.3, so I am limited to vers=1.0. I attempted to pass vers=2.0 (since it looks like basic SMB2 support was available in Samba 3.6.3), however it failed to mount with this error:
mount error(112): Host is down
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Looking at the Wireshark capture when the "SMB response too short" message is being printed, I see the following queries sent before the malformed packet:

<client>        <server>        SMB     268     Trans2 Request, QUERY_PATH_INFO, Query File Unix Basic, Path: /path/to/file.txt
<server>        <client>        SMB     230     Trans2 Response, QUERY_PATH_INFO
<client>        <server>        SMB     129     Read AndX Request, FID: 0x3134, 131022 bytes at offset 262144
<server>        <client>        SMB     79      [Malformed Packet] - Expert Info (Error/Malformed): Malformed Packet (Exception occurred)

The server's samba log for the client during this period:
http://pastebin.com/Y0diqhGY

I also set /proc/fs/cifs/cifsFYI to 7 and captured the following output:
http://pastebin.com/CfPUtBQg

Note that the Samba share definition is as follows:
http://pastebin.com/0aVCaRJH

Thanks,

Andrew

Signature · 3 lines
Thanks,

Steve

A message that is not in the archive (sent off-list or not yet received): CAH2r5ms0bbkhy_D4tRXqwN2nH5vyBH9Q0OrCTV7cm93=P3fgRw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#10 lore raw

----- Original Message -----

Quoted 23 lines

From: "Steve French" <[email protected]>
To: "Andrew Martin" <amartin-AQeFf1F/[email protected]>
Cc: [email protected], "samba-technical" <samba-technical-w/[email protected]>
Sent: Tuesday, September 29, 2015 11:21:25 AM
Subject: Re: "CIFS VFS: SMB response too short (9 bytes)" when copying data from CIFS mount

It sounds pretty straightforward if both wireshark and the Linux kernel client agree and say that this old Samba server sent a malformed response (presumably a CIFS ReadX response). Note that this is an old version of Samba server - Samba 3.6 was released more than 4 years ago.

Try turning off unix extensions on the mount and/or setting rsize smaller (which might change the behavior of the Samba server read response). So try
1) adding something like "rsize=60000" to the options you pass to the mount command
2) and as a different experiment try mounting with "nounix" to disable CIFS unix extensions which would also have the affect of reducing the maximum read size
3) another option to try is to set max protocols in the server's smb.conf to enable SMB2.0 support and try "vers=2.0" on mount

Quoted 9 lines

My guess is that the bug is related to large CIFS read processing in this old version of Samba.

I also want to --- strongly --- encourage upgrade from Samba 3.6.3 to at least Samba 4.1 (I use Samba 4.3). The additional security features and capabilities which SMB2.1 and SMB3 bring (not just the new protocol features and improved performance) are VERY helpful and very important for security.

Steve,

It looks like using either "nounix" or "rsize=60000" fixes the read error! I will also look into upgrading to Samba 4.1 ASAP - I would like to take advantage of the security and performance improvement of a more modern SMB protocol. Thanks again!

Andrew