Skip to contentSkip to conversation

issues with CIFS on Ubuntu 20 mounting Windows 2019

DISCUSSION Scott M. Lewandowski 3 participants 10 messages May 1, 2020 – May 4, 2020 3 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file):

//file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,nounix,iocharset=utf8 0 0

My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.

Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:

[  179.625296] SMB2_read: 9 callbacks suppressed
[  179.625298] CIFS VFS: Send error in read = -4
[  179.704191] CIFS VFS: Send error in read = -4
[  179.847514] CIFS VFS: Send error in read = -4
[  180.031516] CIFS VFS: Send error in read = -4
[  180.214848] CIFS VFS: Send error in read = -4
[  180.282909] CIFS VFS: Send error in read = -4
[  180.474829] CIFS VFS: Send error in read = -4
[  180.621349] CIFS VFS: Send error in read = -4
[  180.780465] CIFS VFS: Send error in read = -4
[  180.823249] CIFS VFS: Send error in read = -4
[  182.339351] CIFS VFS: Close unmatched open
[  182.726829] CIFS VFS: Close unmatched open
[  184.292663] CIFS VFS: Close unmatched open
[  184.501185] CIFS VFS: Close unmatched open
[  191.784922] SMB2_read: 4 callbacks suppressed
[  191.784925] CIFS VFS: Send error in read = -4
[  191.884186] CIFS VFS: Send error in read = -4
[  192.270851] CIFS VFS: Send error in read = -4
[  192.323467] CIFS VFS: Send error in read = -4
[  192.381336] CIFS VFS: Send error in read = -4
[  192.446711] CIFS VFS: Send error in read = -4
[  192.507605] CIFS VFS: Send error in read = -4
[  193.244397] CIFS VFS: Send error in read = -4
[  193.307588] CIFS VFS: Send error in read = -4
[  193.403410] CIFS VFS: Send error in read = -4
[  193.797038] CIFS VFS: Close unmatched open
[  196.784082] CIFS VFS: Close unmatched open
[  198.247594] CIFS VFS: Close unmatched open

Here is the duplicate cookie info:

[   21.713307] FS-Cache: Duplicate cookie detected
[   21.713840] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[   21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[   21.714779] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 fl=2 nc=0 na=1]
[   21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5
[   21.716141] FS-Cache: N-key=[8] '020001bdc0a801af'
[   21.716591] FS-Cache: Duplicate cookie detected
[   21.717217] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[   21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[   21.718220] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 fl=2 nc=0 na=1]
[   21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917
[   21.719621] FS-Cache: N-key=[8] '020001bdc0a801af'
[   21.720097] FS-Cache: Duplicate cookie detected
[   21.720565] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[   21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[   21.721518] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 fl=2 nc=0 na=1]
[   21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa
[   21.722941] FS-Cache: N-key=[8] '020001bdc0a801af'

Any ideas what could be going on?

Thanks for any help!

DebugData:
CIFS Version 2.23
Features: DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,CIFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect 0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x100080000021
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

Number of credits: 8190 Dialect 0x300

2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000019
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: 0x10007c000061
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:
        State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489
        State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

Number of credits: 8190 Dialect 0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000055
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

A message that is not in the archive (sent off-list or not yet received): CAH2r5msOBGS7QYagUkby4upxFKbxCdQg=T65K4s9VhRKkWY+Xw@mail.gmail.com

#2 lore raw
Quoted 1 line

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

Not using scp at the moment.

Quoted 1 line

Could an app be sending signals trying to kill the process doing the reads?

Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.

One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105

-scott

From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file):

//file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,nounix,iocharset=utf8 0 0

My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.

Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:

[  179.625296] SMB2_read: 9 callbacks suppressed
[  179.625298] CIFS VFS: Send error in read = -4
[  179.704191] CIFS VFS: Send error in read = -4
[  179.847514] CIFS VFS: Send error in read = -4
[  180.031516] CIFS VFS: Send error in read = -4
[  180.214848] CIFS VFS: Send error in read = -4
[  180.282909] CIFS VFS: Send error in read = -4
[  180.474829] CIFS VFS: Send error in read = -4
[  180.621349] CIFS VFS: Send error in read = -4
[  180.780465] CIFS VFS: Send error in read = -4
[  180.823249] CIFS VFS: Send error in read = -4
[  182.339351] CIFS VFS: Close unmatched open
[  182.726829] CIFS VFS: Close unmatched open
[  184.292663] CIFS VFS: Close unmatched open
[  184.501185] CIFS VFS: Close unmatched open
[  191.784922] SMB2_read: 4 callbacks suppressed
[  191.784925] CIFS VFS: Send error in read = -4
[  191.884186] CIFS VFS: Send error in read = -4
[  192.270851] CIFS VFS: Send error in read = -4
[  192.323467] CIFS VFS: Send error in read = -4
[  192.381336] CIFS VFS: Send error in read = -4
[  192.446711] CIFS VFS: Send error in read = -4
[  192.507605] CIFS VFS: Send error in read = -4
[  193.244397] CIFS VFS: Send error in read = -4
[  193.307588] CIFS VFS: Send error in read = -4
[  193.403410] CIFS VFS: Send error in read = -4
[  193.797038] CIFS VFS: Close unmatched open
[  196.784082] CIFS VFS: Close unmatched open
[  198.247594] CIFS VFS: Close unmatched open

Here is the duplicate cookie info:

[   21.713307] FS-Cache: Duplicate cookie detected
[   21.713840] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [   21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[   21.714779] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 fl=2 nc=0 na=1] [   21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5
[   21.716141] FS-Cache: N-key=[8] '020001bdc0a801af'
[   21.716591] FS-Cache: Duplicate cookie detected
[   21.717217] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [   21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[   21.718220] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 fl=2 nc=0 na=1] [   21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917
[   21.719621] FS-Cache: N-key=[8] '020001bdc0a801af'
[   21.720097] FS-Cache: Duplicate cookie detected
[   21.720565] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [   21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[   21.721518] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 fl=2 nc=0 na=1] [   21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa
[   21.722941] FS-Cache: N-key=[8] '020001bdc0a801af'

Any ideas what could be going on?

Thanks for any help!

DebugData:
CIFS Version 2.23
Features: DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,CIFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect 0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067     Session Status: 1 TCP status: 1 Instance: 1 Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x100080000021 Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1)      Speed: 10000000000 bps Capabilities:
                IPv4: 192.168.1.175
Number of credits: 8190 Dialect 0x300

2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067     Session Status: 1 TCP status: 1 Instance: 1 Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000019 Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1)      Speed: 10000000000 bps Capabilities:
                IPv4: 192.168.1.175

3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067     Session Status: 1 TCP status: 1 Instance: 1 Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: 0x10007c000061 Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:
        State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489
        State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553

        Server interfaces: 2
        0)      Speed: 10000000000 bps Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1)      Speed: 10000000000 bps Capabilities:
                IPv4: 192.168.1.175

Number of credits: 8190 Dialect 0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067     Session Status: 1 TCP status: 1 Instance: 1 Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000055 Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1)      Speed: 10000000000 bps Capabilities:
                IPv4: 192.168.1.175

#3 lore raw

I’m also seeing entries like the following mixed in with the other messages:

[27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd: 5

And

[31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate inode 00000000b4733e96
Quoted 219 lines

-----Original Message-----
From: Scott M. Lewandowski <[email protected]>
Sent: Friday, May 1, 2020 12:20 AM
To: 'Steve French' <[email protected]>
Cc: CIFS <[email protected]>
Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019

Quoted 2 lines

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

Not using scp at the moment.

Quoted 1 line

Could an app be sending signals trying to kill the process doing the reads?

Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.

One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105

-scott

From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file):

//file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,n ounix,iocharset=utf8 0 0

My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.

Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:

[  179.625296] SMB2_read: 9 callbacks suppressed [  179.625298] CIFS VFS:
Send error in read = -4 [  179.704191] CIFS VFS: Send error in read = -4 [  179.847514] CIFS VFS: Send error in read = -4 [  180.031516] CIFS VFS: Send error in read = -4 [  180.214848] CIFS VFS: Send error in read = -4 [  180.282909] CIFS VFS: Send error in read = -4 [  180.474829] CIFS VFS: Send error in read = -4 [  180.621349] CIFS VFS: Send error in read = -4 [  180.780465] CIFS VFS: Send error in read = -4 [  180.823249] CIFS VFS: Send error in read = -4 [  182.339351] CIFS VFS: Close unmatched open [  182.726829] CIFS VFS: Close unmatched open [  184.292663] CIFS VFS: Close unmatched open [  184.501185] CIFS VFS: Close unmatched open
[  191.784922] SMB2_read: 4 callbacks suppressed [  191.784925] CIFS VFS:
Send error in read = -4 [  191.884186] CIFS VFS: Send error in read = -4 [  192.270851] CIFS VFS: Send error in read = -4 [  192.323467] CIFS VFS: Send error in read = -4 [  192.381336] CIFS VFS: Send error in read = -4 [  192.446711] CIFS VFS: Send error in read = -4 [  192.507605] CIFS VFS: Send error in read = -4 [  193.244397] CIFS VFS: Send error in read = -4 [  193.307588] CIFS VFS: Send error in read = -4 [  193.403410] CIFS VFS: Send error in read = -4 [  193.797038] CIFS VFS: Close unmatched open [  196.784082] CIFS VFS: Close unmatched open [  198.247594] CIFS VFS: Close unmatched open

Here is the duplicate cookie info:

[   21.713307] FS-Cache: Duplicate cookie detected [   21.713840] FS-Cache: O- cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[   21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [   21.714779] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 fl=2 nc=0 na=1] [   21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5 [   21.716141] FS-Cache: N-key=[8] '020001bdc0a801af' [   21.716591] FS-Cache: Duplicate cookie detected [   21.717217] FS-Cache: O- cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[   21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [   21.718220] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 fl=2 nc=0 na=1] [   21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917 [   21.719621] FS-Cache: N-key=[8] '020001bdc0a801af' [   21.720097] FS-Cache: Duplicate cookie detected [   21.720565] FS-Cache: O- cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[   21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [   21.721518] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 fl=2 nc=0 na=1] [   21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa [   21.722941] FS-Cache: N-key=[8] '020001bdc0a801af'

Any ideas what could be going on?

Thanks for any help!

DebugData:
CIFS Version 2.23
Features:
DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect 0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067     Session Status: 1 TCP
status: 1 Instance: 1
        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x100080000021
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1)      Speed: 10000000000 bps Capabilities:
                IPv4: 192.168.1.175
Number of credits: 8190 Dialect 0x300

2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067     Session Status: 1 TCP
status: 1 Instance: 1
        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x10006c000019
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1)      Speed: 10000000000 bps Capabilities:
                IPv4: 192.168.1.175

3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067     Session Status: 1 TCP
status: 1 Instance: 1
        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId:
0x10007c000061
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:
        State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489
        State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553

        Server interfaces: 2
        0)      Speed: 10000000000 bps Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1)      Speed: 10000000000 bps Capabilities:
                IPv4: 192.168.1.175

Number of credits: 8190 Dialect 0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067     Session Status: 1 TCP
status: 1 Instance: 1
        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x10006c000055
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1)      Speed: 10000000000 bps Capabilities:
                IPv4: 192.168.1.175

#4 lore raw

Are you comfortable with dynamic tracing ("trace-cmd record -e ..." ... the list of trace points can be seen in /sys/kernel/debug/tracing/events/cifs)?

Quoted 231 lines · Scott M. Lewandowski

I’m also seeing entries like the following mixed in with the other messages:

[27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd: 5

And

[31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate inode 00000000b4733e96
Quoted 219 lines

-----Original Message-----
From: Scott M. Lewandowski <[email protected]>
Sent: Friday, May 1, 2020 12:20 AM
To: 'Steve French' <[email protected]>
Cc: CIFS <[email protected]>
Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019

Quoted 2 lines

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

Not using scp at the moment.

Quoted 1 line

Could an app be sending signals trying to kill the process doing the reads?

Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.

One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105

-scott

From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file):

//file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,n ounix,iocharset=utf8 0 0

My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.

Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:

[  179.625296] SMB2_read: 9 callbacks suppressed [  179.625298] CIFS VFS:
Send error in read = -4 [  179.704191] CIFS VFS: Send error in read = -4
[  179.847514] CIFS VFS: Send error in read = -4 [  180.031516] CIFS VFS: Send
error in read = -4 [  180.214848] CIFS VFS: Send error in read = -4
[  180.282909] CIFS VFS: Send error in read = -4 [  180.474829] CIFS VFS: Send
error in read = -4 [  180.621349] CIFS VFS: Send error in read = -4
[  180.780465] CIFS VFS: Send error in read = -4 [  180.823249] CIFS VFS: Send
error in read = -4 [  182.339351] CIFS VFS: Close unmatched open
[  182.726829] CIFS VFS: Close unmatched open [  184.292663] CIFS VFS: Close
unmatched open [  184.501185] CIFS VFS: Close unmatched open
[  191.784922] SMB2_read: 4 callbacks suppressed [  191.784925] CIFS VFS:
Send error in read = -4 [  191.884186] CIFS VFS: Send error in read = -4
[  192.270851] CIFS VFS: Send error in read = -4 [  192.323467] CIFS VFS: Send
error in read = -4 [  192.381336] CIFS VFS: Send error in read = -4
[  192.446711] CIFS VFS: Send error in read = -4 [  192.507605] CIFS VFS: Send
error in read = -4 [  193.244397] CIFS VFS: Send error in read = -4
[  193.307588] CIFS VFS: Send error in read = -4 [  193.403410] CIFS VFS: Send
error in read = -4 [  193.797038] CIFS VFS: Close unmatched open
[  196.784082] CIFS VFS: Close unmatched open [  198.247594] CIFS VFS: Close

unmatched open

Here is the duplicate cookie info:

[   21.713307] FS-Cache: Duplicate cookie detected [   21.713840] FS-Cache: O-
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[   21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[   21.714779] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365
fl=2 nc=0 na=1] [   21.715672] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000dd1465c5 [   21.716141] FS-Cache: N-key=[8] '020001bdc0a801af'
[   21.716591] FS-Cache: Duplicate cookie detected [   21.717217] FS-Cache: O-
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[   21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[   21.718220] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365
fl=2 nc=0 na=1] [   21.719166] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000e757f917 [   21.719621] FS-Cache: N-key=[8] '020001bdc0a801af'
[   21.720097] FS-Cache: Duplicate cookie detected [   21.720565] FS-Cache: O-
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[   21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[   21.721518] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365

fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8] '020001bdc0a801af'

Any ideas what could be going on?

Thanks for any help!

DebugData:
CIFS Version 2.23
Features:
DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect 0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP
status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x100080000021
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

Number of credits: 8190 Dialect 0x300

2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP
status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x10006c000019
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP
status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId:
0x10007c000061
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:
        State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489
        State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

Number of credits: 8190 Dialect 0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP
status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x10006c000055
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175
Signature · 3 lines
Thanks,

Steve
#5 lore raw

I'm trying to become comfortable now 😊

I have it setup to capture all CIFS events, but I haven't seen the messages in dmesg since. I'm not sure if I've just gotten unlucky, or if the instrumentation is changing the behavior.

Quoted 281 lines

-----Original Message-----
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 10:57 AM
To: Scott M. Lewandowski <[email protected]>
Cc: Scott M. Lewandowski <[email protected]>; CIFS <linux- [email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

Are you comfortable with dynamic tracing ("trace-cmd record -e ..." ... the list of trace points can be seen in /sys/kernel/debug/tracing/events/cifs)?

Quoted 27 lines · Scott M. Lewandowski

I’m also seeing entries like the following mixed in with the other messages:

[27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd:

5

And

[31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate

inode 00000000b4733e96

Quoted 13 lines

-----Original Message-----
From: Scott M. Lewandowski <[email protected]>
Sent: Friday, May 1, 2020 12:20 AM
To: 'Steve French' <[email protected]>
Cc: CIFS <[email protected]>
Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019

Quoted 2 lines

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

Not using scp at the moment.

Quoted 1 line

Could an app be sending signals trying to kill the process doing the

reads?

Quoted 18 lines
Quoted 18 lines

Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.

One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105

-scott

From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

Quoted 6 lines
Quoted 6 lines

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019

server.

Quoted 3 lines
Quoted 3 lines

Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move

the credential file):

Quoted 13 lines
Quoted 13 lines

//file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserveri no,n
ounix,iocharset=utf8 0 0

My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.

Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:

Quoted 11 lines
Quoted 11 lines
[  179.625296] SMB2_read: 9 callbacks suppressed [  179.625298] CIFS VFS:

Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4

[  180.474829] CIFS VFS: Send error in read = -4 [  180.621349] CIFS

VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS:
Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open [ 191.784922] SMB2_read:

4 callbacks suppressed [ 191.784925] CIFS VFS:

Quoted 10 lines
Quoted 10 lines

Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4

[  192.507605] CIFS VFS: Send error in read = -4 [  193.244397] CIFS

VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS:
Close unmatched open [ 198.247594] CIFS VFS: Close unmatched open

Quoted 3 lines
Quoted 3 lines

Here is the duplicate cookie info:

[   21.713307] FS-Cache: Duplicate cookie detected [   21.713840] FS-

Cache: O-

Quoted 2 lines
Quoted 2 lines

cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]

[   21.714330] FS-Cache: O-cookie d=00000000173ea0ee

n=00000000243f210c

Quoted 2 lines
Quoted 2 lines
[   21.714779] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.715219] FS-Cache: N-cookie c=00000000afb49d9c

[p=0000000079ef0365

Quoted 2 lines
Quoted 2 lines

fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8]

'020001bdc0a801af'

Quoted 1 line
Quoted 1 line
[   21.716591] FS-Cache: Duplicate cookie detected [   21.717217] FS-

Cache: O-

Quoted 2 lines
Quoted 2 lines

cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]

[   21.717679] FS-Cache: O-cookie d=00000000173ea0ee

n=00000000243f210c

Quoted 2 lines
Quoted 2 lines
[   21.718220] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.718681] FS-Cache: N-cookie c=0000000063665cf7

[p=0000000079ef0365

Quoted 2 lines
Quoted 2 lines

fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8]

'020001bdc0a801af'

Quoted 1 line
Quoted 1 line
[   21.720097] FS-Cache: Duplicate cookie detected [   21.720565] FS-

Cache: O-

Quoted 2 lines
Quoted 2 lines

cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]

[   21.721053] FS-Cache: O-cookie d=00000000173ea0ee

n=00000000243f210c

Quoted 2 lines
Quoted 2 lines
[   21.721518] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.721994] FS-Cache: N-cookie c=00000000d479c294

[p=0000000079ef0365

Quoted 2 lines
Quoted 2 lines

fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8]

'020001bdc0a801af'

Quoted 10 lines
Quoted 10 lines

Any ideas what could be going on?

Thanks for any help!

Quoted 4 lines
Quoted 4 lines

DebugData:
CIFS Version 2.23
Features:

DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C

Quoted 6 lines
Quoted 6 lines

IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect 0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:

0x100080000021
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff

1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 14 lines
Quoted 14 lines
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

Number of credits: 8190 Dialect 0x300

Quoted 1 line
Quoted 1 line

2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:

0x10006c000019
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff

1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 14 lines
Quoted 14 lines
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175
Quoted 1 line
Quoted 1 line

3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId:

0x10007c000061
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff

1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 16 lines
Quoted 16 lines
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:
        State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489
        State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid

1211553

Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175

Quoted 2 lines
Quoted 2 lines

Number of credits: 8190 Dialect 0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:

0x10006c000055
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff

1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 5 lines
Quoted 5 lines
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:
Quoted 7 lines
Quoted 7 lines

Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175

--
Thanks,

Steve

A message that is not in the archive (sent off-list or not yet received): CAH2r5mvscNft+arZuzzupNeB7df+c7fUmj=XRNZ5Y6PEts+Efw@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw

For anyone interested, there is an excerpt with the second trace set at https://www.dropbox.com/s/53fewmvsj8w2fau/trace-report-filtered-trimmed?dl=0 (it’s still not exactly succinct so I didn’t want to spam the list with it)

I included the relevant dmesg output at the top of the file

Steve, thx for your help and thoughts so far!

From: Steve French <mailto:[email protected]>
Sent: Saturday, May 2, 2020 1:08 AM
To: Scott M. Lewandowski <mailto:[email protected]>
Cc: CIFS <mailto:[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

if you have a short repro scenario then in window 1) trace-cmd record -e cifs
window 2) repro the bug
window 3) (immediately after the repro) trace-cmd show

Or if you want to make it more narrow maybe something like

"trace-cmd record -e smb3_read* -e smb3_reconnect* -e  smb3_partial_send_reconnect"
or perhaps a somewhat noisier one:
"trace-cmd record -e smb3_read* -e smb3_reconnect* -e smb3_partial_send_reconnect -e smb3_cmd* -e smb3_slow_rsp -e smb3_enter -e smb3_exit*"

On Fri, May 1, 2020 at 1:20 PM Scott M. Lewandowski <mailto:[email protected]> wrote:
I'm trying to become comfortable now 😊

I have it setup to capture all CIFS events, but I haven't seen the messages in dmesg since. I'm not sure if I've just gotten unlucky, or if the instrumentation is changing the behavior.

Quoted 281 lines

-----Original Message-----
From: Steve French <mailto:[email protected]>
Sent: Friday, May 1, 2020 10:57 AM
To: Scott M. Lewandowski <mailto:[email protected]>
Cc: Scott M. Lewandowski <mailto:[email protected]>; CIFS <linux- mailto:[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

Are you comfortable with dynamic tracing ("trace-cmd record -e ..." ... the list of trace points can be seen in /sys/kernel/debug/tracing/events/cifs)?

Quoted 27 lines · Scott M. Lewandowski

I’m also seeing entries like the following mixed in with the other messages:

[27078.665192] CIFS VFS: file://file1 Cancelling wait for mid 13443257 cmd:

5

And

[31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate

inode 00000000b4733e96

Quoted 13 lines

-----Original Message-----
From: Scott M. Lewandowski <mailto:[email protected]>
Sent: Friday, May 1, 2020 12:20 AM
To: 'Steve French' <mailto:[email protected]>
Cc: CIFS <mailto:[email protected]>
Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019

Quoted 2 lines

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

Not using scp at the moment.

Quoted 1 line

Could an app be sending signals trying to kill the process doing the

reads?

Quoted 18 lines
Quoted 18 lines

Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.

One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105

-scott

From: Steve French <mailto:[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <mailto:[email protected]>
Cc: CIFS <mailto:[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

Quoted 6 lines
Quoted 6 lines

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019

server.

Quoted 3 lines
Quoted 3 lines

Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move

the credential file):

Quoted 13 lines
Quoted 13 lines

//file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserveri no,n
ounix,iocharset=utf8 0 0

My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.

Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:

Quoted 11 lines
Quoted 11 lines

[  179.625296] SMB2_read: 9 callbacks suppressed [  179.625298] CIFS VFS:
Send error in read = -4 [  179.704191] CIFS VFS: Send error in read = -4 [  179.847514] CIFS VFS: Send error in read = -4 [  180.031516] CIFS VFS: Send error in read = -4 [  180.214848] CIFS VFS: Send error in read = -4 [  180.282909] CIFS VFS: Send error in read = -4 [  180.474829] CIFS VFS: Send error in read = -4 [  180.621349] CIFS
VFS: Send error in read = -4 [  180.780465] CIFS VFS: Send error in read = -4 [  180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [  182.726829] CIFS VFS:
Close unmatched open [  184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open [  191.784922] SMB2_read:

4 callbacks suppressed [  191.784925] CIFS VFS:

Quoted 10 lines
Quoted 10 lines

Send error in read = -4 [  191.884186] CIFS VFS: Send error in read = -4 [  192.270851] CIFS VFS: Send error in read = -4 [  192.323467] CIFS VFS: Send error in read = -4 [  192.381336] CIFS VFS: Send error in read = -4 [  192.446711] CIFS VFS: Send error in read = -4 [  192.507605] CIFS VFS: Send error in read = -4 [  193.244397] CIFS
VFS: Send error in read = -4 [  193.307588] CIFS VFS: Send error in read = -4 [  193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [  196.784082] CIFS VFS:
Close unmatched open [  198.247594] CIFS VFS: Close unmatched open

Quoted 3 lines
Quoted 3 lines

Here is the duplicate cookie info:

[   21.713307] FS-Cache: Duplicate cookie detected [   21.713840] FS-

Cache: O-

Quoted 2 lines
Quoted 2 lines

cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [   21.714330] FS-Cache: O-cookie d=00000000173ea0ee

n=00000000243f210c

Quoted 2 lines
Quoted 2 lines

[   21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' [   21.715219] FS-Cache: N-cookie c=00000000afb49d9c

[p=0000000079ef0365

Quoted 2 lines
Quoted 2 lines

fl=2 nc=0 na=1] [   21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5 [   21.716141] FS-Cache: N-key=[8]

'020001bdc0a801af'

Quoted 1 line
Quoted 1 line

[   21.716591] FS-Cache: Duplicate cookie detected [   21.717217] FS-

Cache: O-

Quoted 2 lines
Quoted 2 lines

cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [   21.717679] FS-Cache: O-cookie d=00000000173ea0ee

n=00000000243f210c

Quoted 2 lines
Quoted 2 lines

[   21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' [   21.718681] FS-Cache: N-cookie c=0000000063665cf7

[p=0000000079ef0365

Quoted 2 lines
Quoted 2 lines

fl=2 nc=0 na=1] [   21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917 [   21.719621] FS-Cache: N-key=[8]

'020001bdc0a801af'

Quoted 1 line
Quoted 1 line

[   21.720097] FS-Cache: Duplicate cookie detected [   21.720565] FS-

Cache: O-

Quoted 2 lines
Quoted 2 lines

cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [   21.721053] FS-Cache: O-cookie d=00000000173ea0ee

n=00000000243f210c

Quoted 2 lines
Quoted 2 lines

[   21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' [   21.721994] FS-Cache: N-cookie c=00000000d479c294

[p=0000000079ef0365

Quoted 2 lines
Quoted 2 lines

fl=2 nc=0 na=1] [   21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa [   21.722941] FS-Cache: N-key=[8]

'020001bdc0a801af'

Quoted 10 lines
Quoted 10 lines

Any ideas what could be going on?

Thanks for any help!

Quoted 4 lines
Quoted 4 lines

DebugData:
CIFS Version 2.23
Features:

DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C

Quoted 6 lines
Quoted 6 lines

IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect 0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067     Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1
         Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x100080000021
         Shares:
         0) IPC: file://file1/IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
         PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None        Share Flags: 0x30
         tid: 0x1        Maximal Access: 0x11f01ff

         1) file://file1/XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
         PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 14 lines
Quoted 14 lines

         Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
         tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

         MIDs:

         Server interfaces: 2
         0)      Speed: 10000000000 bps Capabilities:
                 IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1)      Speed: 10000000000 bps Capabilities:
                 IPv4: 192.168.1.175
Number of credits: 8190 Dialect 0x300

Quoted 1 line
Quoted 1 line

2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067     Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1
         Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x10006c000019
         Shares:
         0) IPC: file://file1/IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
         PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None        Share Flags: 0x30
         tid: 0x1        Maximal Access: 0x11f01ff

         1) file://file1/usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
         PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 14 lines
Quoted 14 lines

         Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
         tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

         MIDs:

         Server interfaces: 2
         0)      Speed: 10000000000 bps Capabilities:
                 IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1)      Speed: 10000000000 bps Capabilities:
                 IPv4: 192.168.1.175

Quoted 1 line
Quoted 1 line

3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067     Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1
         Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId:
0x10007c000061
         Shares:
         0) IPC: file://file1/IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
         PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None        Share Flags: 0x30
         tid: 0x1        Maximal Access: 0x11f01ff

         1) file://file1/YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
         PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 16 lines
Quoted 16 lines

         Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
         tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

         MIDs:
         State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489
         State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553

         Server interfaces: 2
         0)      Speed: 10000000000 bps Capabilities:
                 IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1)      Speed: 10000000000 bps Capabilities:
                 IPv4: 192.168.1.175

Quoted 2 lines
Quoted 2 lines

Number of credits: 8190 Dialect 0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067     Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1
         Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x10006c000055
         Shares:
         0) IPC: file://file1/IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
         PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None        Share Flags: 0x30
         tid: 0x1        Maximal Access: 0x11f01ff

         1) file://file1/ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
         PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 5 lines
Quoted 5 lines

         Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
         tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

         MIDs:

Quoted 7 lines
Quoted 7 lines

         Server interfaces: 2
         0)      Speed: 10000000000 bps Capabilities:
                 IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1)      Speed: 10000000000 bps Capabilities:
                 IPv4: 192.168.1.175

--
Thanks,

Steve

Signature · 3 lines
Thanks,

Steve
#9 lore raw
In reply to Steve French

I've now got a trace that corresponds to the dmesg output below. I will take a look later, but upon quick glance, this doesn't look like anything that will mean much to me, unfortunately. If someone would like to look at the file I can find a way to get it to you.

[53223.099221] CIFS VFS: Send error in read = -4
[53223.232373] CIFS VFS: Send error in read = -4
[53254.762914] CIFS VFS: Send error in read = -4
[53301.403925] CIFS VFS: Send error in read = -4
[53301.772568] CIFS VFS: Send error in read = -4
[53301.818566] CIFS VFS: Send error in read = -4
[53301.885510] CIFS VFS: Send error in read = -4
[53302.055959] CIFS VFS: Send error in read = -4
[53302.146337] CIFS VFS: Send error in read = -4
[53302.236322] CIFS VFS: Send error in read = -4
[53302.329525] CIFS VFS: Send error in read = -4
[53303.000869] CIFS VFS: Send error in read = -4
[53303.106961] CIFS VFS: Send error in read = -4
[53304.756998] CIFS VFS: Close unmatched open
[53670.776832] CIFS VFS: Close unmatched open
Quoted 281 lines

-----Original Message-----
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 10:57 AM
To: Scott M. Lewandowski <[email protected]>
Cc: Scott M. Lewandowski <[email protected]>; CIFS <linux- [email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

Are you comfortable with dynamic tracing ("trace-cmd record -e ..." ... the list of trace points can be seen in /sys/kernel/debug/tracing/events/cifs)?

Quoted 27 lines · Scott M. Lewandowski

I’m also seeing entries like the following mixed in with the other messages:

[27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd:

5

And

[31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate

inode 00000000b4733e96

Quoted 13 lines

-----Original Message-----
From: Scott M. Lewandowski <[email protected]>
Sent: Friday, May 1, 2020 12:20 AM
To: 'Steve French' <[email protected]>
Cc: CIFS <[email protected]>
Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019

Quoted 2 lines

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

Not using scp at the moment.

Quoted 1 line

Could an app be sending signals trying to kill the process doing the

reads?

Quoted 18 lines
Quoted 18 lines

Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.

One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105

-scott

From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

Quoted 6 lines
Quoted 6 lines

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019

server.

Quoted 3 lines
Quoted 3 lines

Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move

the credential file):

Quoted 13 lines
Quoted 13 lines

//file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserveri no,n
ounix,iocharset=utf8 0 0

My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.

Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:

Quoted 11 lines
Quoted 11 lines
[  179.625296] SMB2_read: 9 callbacks suppressed [  179.625298] CIFS VFS:

Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4

[  180.474829] CIFS VFS: Send error in read = -4 [  180.621349] CIFS

VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS:
Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open [ 191.784922] SMB2_read:

4 callbacks suppressed [ 191.784925] CIFS VFS:

Quoted 10 lines
Quoted 10 lines

Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4

[  192.507605] CIFS VFS: Send error in read = -4 [  193.244397] CIFS

VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS:
Close unmatched open [ 198.247594] CIFS VFS: Close unmatched open

Quoted 3 lines
Quoted 3 lines

Here is the duplicate cookie info:

[   21.713307] FS-Cache: Duplicate cookie detected [   21.713840] FS-

Cache: O-

Quoted 2 lines
Quoted 2 lines

cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]

[   21.714330] FS-Cache: O-cookie d=00000000173ea0ee

n=00000000243f210c

Quoted 2 lines
Quoted 2 lines
[   21.714779] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.715219] FS-Cache: N-cookie c=00000000afb49d9c

[p=0000000079ef0365

Quoted 2 lines
Quoted 2 lines

fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8]

'020001bdc0a801af'

Quoted 1 line
Quoted 1 line
[   21.716591] FS-Cache: Duplicate cookie detected [   21.717217] FS-

Cache: O-

Quoted 2 lines
Quoted 2 lines

cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]

[   21.717679] FS-Cache: O-cookie d=00000000173ea0ee

n=00000000243f210c

Quoted 2 lines
Quoted 2 lines
[   21.718220] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.718681] FS-Cache: N-cookie c=0000000063665cf7

[p=0000000079ef0365

Quoted 2 lines
Quoted 2 lines

fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8]

'020001bdc0a801af'

Quoted 1 line
Quoted 1 line
[   21.720097] FS-Cache: Duplicate cookie detected [   21.720565] FS-

Cache: O-

Quoted 2 lines
Quoted 2 lines

cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]

[   21.721053] FS-Cache: O-cookie d=00000000173ea0ee

n=00000000243f210c

Quoted 2 lines
Quoted 2 lines
[   21.721518] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.721994] FS-Cache: N-cookie c=00000000d479c294

[p=0000000079ef0365

Quoted 2 lines
Quoted 2 lines

fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8]

'020001bdc0a801af'

Quoted 10 lines
Quoted 10 lines

Any ideas what could be going on?

Thanks for any help!

Quoted 4 lines
Quoted 4 lines

DebugData:
CIFS Version 2.23
Features:

DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C

Quoted 6 lines
Quoted 6 lines

IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect 0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:

0x100080000021
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff

1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 14 lines
Quoted 14 lines
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

Number of credits: 8190 Dialect 0x300

Quoted 1 line
Quoted 1 line

2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:

0x10006c000019
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff

1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 14 lines
Quoted 14 lines
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175
Quoted 1 line
Quoted 1 line

3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId:

0x10007c000061
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff

1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 16 lines
Quoted 16 lines
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:
        State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489
        State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid

1211553

Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175

Quoted 2 lines
Quoted 2 lines

Number of credits: 8190 Dialect 0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:

0x10006c000055
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff

1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 5 lines
Quoted 5 lines
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:
Quoted 7 lines
Quoted 7 lines

Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175

--
Thanks,

Steve

#10 lore raw

I would expect the close unmatched open to be related to the session being closed (thus all files implicitly closed on the server) but the session drop overlapping the application closing a file.

Quoted 302 lines · Scott M. Lewandowski

I've now got a trace that corresponds to the dmesg output below. I will take a look later, but upon quick glance, this doesn't look like anything that will mean much to me, unfortunately. If someone would like to look at the file I can find a way to get it to you.

[53223.099221] CIFS VFS: Send error in read = -4
[53223.232373] CIFS VFS: Send error in read = -4
[53254.762914] CIFS VFS: Send error in read = -4
[53301.403925] CIFS VFS: Send error in read = -4
[53301.772568] CIFS VFS: Send error in read = -4
[53301.818566] CIFS VFS: Send error in read = -4
[53301.885510] CIFS VFS: Send error in read = -4
[53302.055959] CIFS VFS: Send error in read = -4
[53302.146337] CIFS VFS: Send error in read = -4
[53302.236322] CIFS VFS: Send error in read = -4
[53302.329525] CIFS VFS: Send error in read = -4
[53303.000869] CIFS VFS: Send error in read = -4
[53303.106961] CIFS VFS: Send error in read = -4
[53304.756998] CIFS VFS: Close unmatched open
[53670.776832] CIFS VFS: Close unmatched open
Quoted 281 lines

-----Original Message-----
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 10:57 AM
To: Scott M. Lewandowski <[email protected]>
Cc: Scott M. Lewandowski <[email protected]>; CIFS <linux- [email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

Are you comfortable with dynamic tracing ("trace-cmd record -e ..." ... the list of trace points can be seen in /sys/kernel/debug/tracing/events/cifs)?

Quoted 27 lines · Scott M. Lewandowski

I’m also seeing entries like the following mixed in with the other messages:

[27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd:

5

And

[31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate

inode 00000000b4733e96

Quoted 13 lines

-----Original Message-----
From: Scott M. Lewandowski <[email protected]>
Sent: Friday, May 1, 2020 12:20 AM
To: 'Steve French' <[email protected]>
Cc: CIFS <[email protected]>
Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019

Quoted 2 lines

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

Not using scp at the moment.

Quoted 1 line

Could an app be sending signals trying to kill the process doing the

reads?

Quoted 18 lines
Quoted 18 lines

Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.

One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105

-scott

From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

Quoted 6 lines
Quoted 6 lines

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019

server.

Quoted 3 lines
Quoted 3 lines

Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move

the credential file):

Quoted 13 lines
Quoted 13 lines

//file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserveri no,n
ounix,iocharset=utf8 0 0

My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.

Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:

Quoted 11 lines
Quoted 11 lines
[  179.625296] SMB2_read: 9 callbacks suppressed [  179.625298] CIFS VFS:

Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4

[  180.474829] CIFS VFS: Send error in read = -4 [  180.621349] CIFS

VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS:
Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open [ 191.784922] SMB2_read:

4 callbacks suppressed [ 191.784925] CIFS VFS:

Quoted 10 lines
Quoted 10 lines

Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4

[  192.507605] CIFS VFS: Send error in read = -4 [  193.244397] CIFS

VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS:
Close unmatched open [ 198.247594] CIFS VFS: Close unmatched open

Quoted 3 lines
Quoted 3 lines

Here is the duplicate cookie info:

[   21.713307] FS-Cache: Duplicate cookie detected [   21.713840] FS-

Cache: O-

Quoted 2 lines
Quoted 2 lines

cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]

[   21.714330] FS-Cache: O-cookie d=00000000173ea0ee

n=00000000243f210c

Quoted 2 lines
Quoted 2 lines
[   21.714779] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.715219] FS-Cache: N-cookie c=00000000afb49d9c

[p=0000000079ef0365

Quoted 2 lines
Quoted 2 lines

fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8]

'020001bdc0a801af'

Quoted 1 line
Quoted 1 line
[   21.716591] FS-Cache: Duplicate cookie detected [   21.717217] FS-

Cache: O-

Quoted 2 lines
Quoted 2 lines

cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]

[   21.717679] FS-Cache: O-cookie d=00000000173ea0ee

n=00000000243f210c

Quoted 2 lines
Quoted 2 lines
[   21.718220] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.718681] FS-Cache: N-cookie c=0000000063665cf7

[p=0000000079ef0365

Quoted 2 lines
Quoted 2 lines

fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8]

'020001bdc0a801af'

Quoted 1 line
Quoted 1 line
[   21.720097] FS-Cache: Duplicate cookie detected [   21.720565] FS-

Cache: O-

Quoted 2 lines
Quoted 2 lines

cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]

[   21.721053] FS-Cache: O-cookie d=00000000173ea0ee

n=00000000243f210c

Quoted 2 lines
Quoted 2 lines
[   21.721518] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.721994] FS-Cache: N-cookie c=00000000d479c294

[p=0000000079ef0365

Quoted 2 lines
Quoted 2 lines

fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8]

'020001bdc0a801af'

Quoted 10 lines
Quoted 10 lines

Any ideas what could be going on?

Thanks for any help!

Quoted 4 lines
Quoted 4 lines

DebugData:
CIFS Version 2.23
Features:

DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C

Quoted 6 lines
Quoted 6 lines

IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect 0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:

0x100080000021
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff

1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 14 lines
Quoted 14 lines
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

Number of credits: 8190 Dialect 0x300

Quoted 1 line
Quoted 1 line

2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:

0x10006c000019
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff

1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 14 lines
Quoted 14 lines
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175
Quoted 1 line
Quoted 1 line

3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId:

0x10007c000061
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff

1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 16 lines
Quoted 16 lines
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:
        State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489
        State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid

1211553

Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175

Quoted 2 lines
Quoted 2 lines

Number of credits: 8190 Dialect 0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1

TCP

Quoted 11 lines
Quoted 11 lines

status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:

0x10006c000055
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff

1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:

0xa3c339a0

Quoted 5 lines
Quoted 5 lines
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:
Quoted 7 lines
Quoted 7 lines

Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175

--
Thanks,

Steve

Signature · 3 lines
Thanks,

Steve
#11 lore raw
In reply to Scott M. Lewandowski

I don't know much about mono but wonder if signal handling problems (in mono) could be related "EINTR" is suspicious on read ...

Don't know if this is related but google search turned up this:

"Mono uses signals on certain platforms to handle NullReferenceExceptions. It’s also becoming more and more common to add third-party libraries that detect and report crashes. The problem is when those third-party libraries install their own signal handlers, overwriting Mono’s signal handlers. When this happens, any NullReferenceException would be detected by such libraries as a fatal SIGSEGV, and the libraries would report it as a crash and terminate the process."

Quoted 199 lines · Scott M. Lewandowski
Quoted 2 lines

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

Not using scp at the moment.

Quoted 1 line

Could an app be sending signals trying to kill the process doing the reads?

Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.

One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105

-scott

From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019

EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads

On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file):

//file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,nounix,iocharset=utf8 0 0

My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.

Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:

[  179.625296] SMB2_read: 9 callbacks suppressed
[  179.625298] CIFS VFS: Send error in read = -4
[  179.704191] CIFS VFS: Send error in read = -4
[  179.847514] CIFS VFS: Send error in read = -4
[  180.031516] CIFS VFS: Send error in read = -4
[  180.214848] CIFS VFS: Send error in read = -4
[  180.282909] CIFS VFS: Send error in read = -4
[  180.474829] CIFS VFS: Send error in read = -4
[  180.621349] CIFS VFS: Send error in read = -4
[  180.780465] CIFS VFS: Send error in read = -4
[  180.823249] CIFS VFS: Send error in read = -4
[  182.339351] CIFS VFS: Close unmatched open
[  182.726829] CIFS VFS: Close unmatched open
[  184.292663] CIFS VFS: Close unmatched open
[  184.501185] CIFS VFS: Close unmatched open
[  191.784922] SMB2_read: 4 callbacks suppressed
[  191.784925] CIFS VFS: Send error in read = -4
[  191.884186] CIFS VFS: Send error in read = -4
[  192.270851] CIFS VFS: Send error in read = -4
[  192.323467] CIFS VFS: Send error in read = -4
[  192.381336] CIFS VFS: Send error in read = -4
[  192.446711] CIFS VFS: Send error in read = -4
[  192.507605] CIFS VFS: Send error in read = -4
[  193.244397] CIFS VFS: Send error in read = -4
[  193.307588] CIFS VFS: Send error in read = -4
[  193.403410] CIFS VFS: Send error in read = -4
[  193.797038] CIFS VFS: Close unmatched open
[  196.784082] CIFS VFS: Close unmatched open
[  198.247594] CIFS VFS: Close unmatched open

Here is the duplicate cookie info:

[   21.713307] FS-Cache: Duplicate cookie detected
[   21.713840] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[   21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[   21.714779] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 fl=2 nc=0 na=1]
[   21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5
[   21.716141] FS-Cache: N-key=[8] '020001bdc0a801af'
[   21.716591] FS-Cache: Duplicate cookie detected
[   21.717217] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[   21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[   21.718220] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 fl=2 nc=0 na=1]
[   21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917
[   21.719621] FS-Cache: N-key=[8] '020001bdc0a801af'
[   21.720097] FS-Cache: Duplicate cookie detected
[   21.720565] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[   21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[   21.721518] FS-Cache: O-key=[8] '020001bdc0a801af'
[   21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 fl=2 nc=0 na=1]
[   21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa
[   21.722941] FS-Cache: N-key=[8] '020001bdc0a801af'

Any ideas what could be going on?

Thanks for any help!

DebugData:
CIFS Version 2.23
Features: DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,CIFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect 0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x100080000021
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

Number of credits: 8190 Dialect 0x300

2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000019
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: 0x10007c000061
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:
        State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489
        State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175

Number of credits: 8190 Dialect 0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1

        Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000055
        Shares:
        0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
        PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
        Share Capabilities: None        Share Flags: 0x30
        tid: 0x1        Maximal Access: 0x11f01ff

        1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
        PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
        Share Capabilities: None Aligned, Partition Aligned,    Share Flags: 0x0
        tid: 0x5        Optimal sector size: 0x200      Maximal Access: 0x1f01ff

        MIDs:

        Server interfaces: 2
        0)      Speed: 10000000000 bps
                Capabilities:
                IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
        1)      Speed: 10000000000 bps
                Capabilities:
                IPv4: 192.168.1.175
Signature · 3 lines
Thanks,

Steve
#12 lore raw
In reply to Scott M. Lewandowski

Hi Scott,

If you can reproduce this, could you try making a capture of mounting + waiting for the bug?

https://wiki.samba.org/index.php/Bug_Reporting#cifs.ko

Cheers,

Signature · 4 lines
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)