Linux CIFS client, Windows 2012 CIFS file server => files not found?

Folks;

the last few years we've been locally serving files using a NetApp storage system exposing CIFS and several (Windows, Linux) clients mounting these shares in order to host data and run an internal business application.

The application itself (binaries, libraries, resources, a bunch of ksh scripts) entirely lives on the CIFS share. So far (having NetApp CIFS mounted), this worked on all machines.

Now, these CIFS shares are to be provided by a Windows 2012 Storage Server. Outcome: Linux machines can't run this application anymore. Having mounted the W2012 CIFS share, the application won't start complaining that some resources (scripts and libraries) are missing, even while they are obviously around and also, permission-wise, accessible.

It's difficult to really figure out what happens here. Basically, looking at strace output, it seems the application does stat64() one of these files which in both cases seems to provide a valid outcome...

stat64("/opt/.../file.oeb", {st_mode=S_IFREG|0755, st_size=36405, ...}) = 0

... but while mounting the share off the NetApp the application works as desired, mounting the share off the W2012 server the application will terminate here complaining about this particular resource missing.

Vague as it is, does anyone have any idea what could be causing this? I've been playing with various parameters of mount.cifs so far, but I'm not deep enough into either NetApps nor Windows 2012s understanding of CIFS to have an idea what might have changed here...

TIA,
Kristian

any chance you could send me a small capture file of each (preferably the smallest possible reproduction scenario so the trace isn't big and cluttered) so I can compare the two server's responses to see what might be going on.

(see the "wire captures" section of
https://wiki.samba.org/index.php/LinuxCIFS_troubleshooting)

Thanks,

Steve

Hi Steve;

and first off, thanks loads for your feedback.

Am 20.08.2015 um 18:11 schrieb Steve French:

Sure, it's easy to reproduce, and I ended up with two pcap files. Mind if I send these off-list, or should I rather upload them anywhere?

Thanks again and all the best,
Kristian

Last thing in the trace is a parallelized read with lots of reads in flight at once which should be ok and I do see a response.

Two obvious things to ask:

1) What dialect are you using when you mount to Windows? It looks like SMB2.0 (and would be MUCH better to use SMB2.1 or SMB3 or SMB3.02, or even CIFS when mounting to Windows, SMB2.0 is old and does not support large reads/writes and it looks like it is failing ) rather than the default CIFS. To NetApp you are mounting using CIFS (not SMB2 or SMB3) which seems to work ok for your workload looking at the trace.

Note that you should try the same dialect on both the mount to NetApp and Windows to make the comparison easier: e.g. after -o in the mount command specify "vers=2.1" or "vers=3.0" (or don't specify any vers (or "vers=1.0") and it should default to cifs)

2) Are there any things logged in the message log on the Linux client (type "dmesg" to see what is logged) by cifs.ko during the failure on
running to Windows. Does the status of the session show disconnected
at the end of the failure to Windows? (See the "SMB session status"
and "TCP status" in the output of "cat /proc/fs/cifs/DebugData")

Thanks,

Steve

Hi Steve, *;

Am 20.08.2015 um 22:23 schrieb Steve French:

Ok. It's a bit strange, we used to have a similar behaviour a while ago after switching the NetApp filer to SMB2.0, which would end up in a situation in which some (Linux) part applications created files on the CIFS share which this particular application was unable to see. In that particular case however it seemed a timing problem in the application and happened when creating files on CIFS from Linux and trying to access them from Windows clients.

I tried all dialects while connecting to Windows, which didn't change this behaviour, just mounting with vers=3.0 makes it fail considerably faster.

Actually, to use the same dialect for both mounts means using vers=1.0; though the NetApp filer does support some understanding of at least SMB2.0 (and the shares can be accessed using SMB2.0 from Windows clients), mount.cifs can't mount any of our NetApp shares using SMB2.0.

We figured this out only a few weeks ago when starting moving our stores and tried to mount things consistently using SMB2.0 for performance reasons, and mounting the NetApp shares using anything else than CIFS will reproducibly end up in errors like this:

-------------------
mount error(95): Operation not supported
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

[3959136.173555] CIFS VFS: cifs_read_super: get root inode failed

-------------------

That's a different problem however and I am unsure whether this is a general problem or just an issue our fairly old NetApp / ONTAP version is to blame for. I surely can provide you with more information on that if that is of interest.

No. Unfortunately not.

Unsure. This is what DebugData so far says for this particular filer connection:

CIFS Version 2.03
Features: dfs fscache lanman posix spnego xattr acl Active VFS Requests: 0
Servers:
1) entry for 192.168.1.138 not fully displayed

        TCP status: 1
        Local Users To Server: 1 SecMode: 0x3 Req On Wire: 0
        Shares:
        1) \\192.168.1.138\Store Mounts: 1 Type: NTFS DevInfo: 0x60020
		Attributes: 0xc700ff
        PathComponentMax: 255 Status: 0x1 type: DISK
        MIDs:

Can you make anything of this?

Thanks bunches and all the best,
Kristian

Hi Benjamin;

and thanks for your feedback. :)

Am 21.08.2015 um 10:57 schrieb Benjamin Bellec:

Oh, ok. I see. Well, so I'm sorted here - not much of a problem as our FAS2020 (running ONTAP 7.3.something) is on its way out also due to other issues. Not sure whether a Windows based file server is a better choice, though. Let's see... ;)

Best,
Kristian