issues with CIFS on Ubuntu 20 mounting Windows 2019
Subsystems (2)
I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file):
//file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,nounix,iocharset=utf8 0 0
My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.
Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:
[ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS: Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4 [ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFS VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS: Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open [ 191.784922] SMB2_read: 4 callbacks suppressed [ 191.784925] CIFS VFS: Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4 [ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFS VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS: Close unmatched open [ 198.247594] CIFS VFS: Close unmatched open
Here is the duplicate cookie info:
[ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8] '020001bdc0a801af' [ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8] '020001bdc0a801af' [ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8] '020001bdc0a801af'
Any ideas what could be going on?
Thanks for any help!
DebugData:
CIFS Version 2.23
Features: DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,CIFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect 0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x100080000021
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff
1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff
MIDs:
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175Number of credits: 8190 Dialect 0x300
2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000019
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff
1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff
MIDs:
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.1753) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: 0x10007c000061
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff
1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff
MIDs:
State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489
State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175Number of credits: 8190 Dialect 0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000055
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff
1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff
MIDs:
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175A message that is not in the archive (sent off-list or not yet received): CAH2r5msOBGS7QYagUkby4upxFKbxCdQg=T65K4s9VhRKkWY+Xw@mail.gmail.com
Quoted 1 line
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
Not using scp at the moment.
Quoted 1 line
Could an app be sending signals trying to kill the process doing the reads?
Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.
One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105
-scott
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file):
//file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,nounix,iocharset=utf8 0 0
My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.
Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:
[ 179.625296] SMB2_read: 9 callbacks suppressed
[ 179.625298] CIFS VFS: Send error in read = -4
[ 179.704191] CIFS VFS: Send error in read = -4
[ 179.847514] CIFS VFS: Send error in read = -4
[ 180.031516] CIFS VFS: Send error in read = -4
[ 180.214848] CIFS VFS: Send error in read = -4
[ 180.282909] CIFS VFS: Send error in read = -4
[ 180.474829] CIFS VFS: Send error in read = -4
[ 180.621349] CIFS VFS: Send error in read = -4
[ 180.780465] CIFS VFS: Send error in read = -4
[ 180.823249] CIFS VFS: Send error in read = -4
[ 182.339351] CIFS VFS: Close unmatched open
[ 182.726829] CIFS VFS: Close unmatched open
[ 184.292663] CIFS VFS: Close unmatched open
[ 184.501185] CIFS VFS: Close unmatched open
[ 191.784922] SMB2_read: 4 callbacks suppressed
[ 191.784925] CIFS VFS: Send error in read = -4
[ 191.884186] CIFS VFS: Send error in read = -4
[ 192.270851] CIFS VFS: Send error in read = -4
[ 192.323467] CIFS VFS: Send error in read = -4
[ 192.381336] CIFS VFS: Send error in read = -4
[ 192.446711] CIFS VFS: Send error in read = -4
[ 192.507605] CIFS VFS: Send error in read = -4
[ 193.244397] CIFS VFS: Send error in read = -4
[ 193.307588] CIFS VFS: Send error in read = -4
[ 193.403410] CIFS VFS: Send error in read = -4
[ 193.797038] CIFS VFS: Close unmatched open
[ 196.784082] CIFS VFS: Close unmatched open
[ 198.247594] CIFS VFS: Close unmatched open
Here is the duplicate cookie info:
[ 21.713307] FS-Cache: Duplicate cookie detected
[ 21.713840] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af'
[ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5
[ 21.716141] FS-Cache: N-key=[8] '020001bdc0a801af'
[ 21.716591] FS-Cache: Duplicate cookie detected
[ 21.717217] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af'
[ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917
[ 21.719621] FS-Cache: N-key=[8] '020001bdc0a801af'
[ 21.720097] FS-Cache: Duplicate cookie detected
[ 21.720565] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af'
[ 21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa
[ 21.722941] FS-Cache: N-key=[8] '020001bdc0a801af'
Any ideas what could be going on?
Thanks for any help!
DebugData:
CIFS Version 2.23
Features: DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,CIFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect 0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1 Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x100080000021 Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff
1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff
MIDs:
Server interfaces: 2
0) Speed: 10000000000 bps Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities:
IPv4: 192.168.1.175
Number of credits: 8190 Dialect 0x300
2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1 Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000019 Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff
1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff
MIDs:
Server interfaces: 2
0) Speed: 10000000000 bps Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities:
IPv4: 192.168.1.175
3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1 Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: 0x10007c000061 Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff
1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff
MIDs:
State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489
State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553
Server interfaces: 2
0) Speed: 10000000000 bps Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities:
IPv4: 192.168.1.175
Number of credits: 8190 Dialect 0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1 Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000055 Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff
1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff
MIDs:
Server interfaces: 2
0) Speed: 10000000000 bps Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities:
IPv4: 192.168.1.175
I’m also seeing entries like the following mixed in with the other messages:
[27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd: 5
And
[31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate inode 00000000b4733e96
Quoted 219 lines
-----Original Message-----
From: Scott M. Lewandowski <[email protected]>
Sent: Friday, May 1, 2020 12:20 AM
To: 'Steve French' <[email protected]>
Cc: CIFS <[email protected]>
Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019Quoted 2 lines
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
Not using scp at the moment.
Quoted 1 line
Could an app be sending signals trying to kill the process doing the reads?
Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.
One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105
-scott
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file)://file1/usenet
/mnt/usenetcifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,n ounix,iocharset=utf8 0 0My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.
Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:
[ 179.625296]
SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS:
Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4 [ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFS VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS: Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open
[ 191.784922]SMB2_read: 4 callbacks suppressed [ 191.784925] CIFS VFS:
Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4 [ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFS VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS: Close unmatched open [ 198.247594] CIFS VFS: Close unmatched openHere is the duplicate cookie info:
[ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS-Cache: O- cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af'
[ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8] '020001bdc0a801af' [ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS-Cache: O- cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af'
[ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8] '020001bdc0a801af' [ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS-Cache: O- cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af'
[ 21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8] '020001bdc0a801af'Any ideas what could be going on?
Thanks for any help!
DebugData:
CIFS Version 2.23
Features:
DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP
status: 1 Instance: 1
Local Users To Server: 1 SecMode:0x1Req On Wire: 0 SessionId:0x100080000021
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0Share Capabilities: None Share Flags:0x30
tid: 0x1 Maximal Access:0x11f01ff1) \\file1\XXX Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:0xa3c339a0Share Capabilities: None Aligned, Partition Aligned, Share Flags:0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access:0x1f01ffMIDs:
Server interfaces: 2
0) Speed: 10000000000 bps Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities:
IPv4: 192.168.1.175
Number of credits: 8190 Dialect0x3002) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP
status: 1 Instance: 1
Local Users To Server: 1 SecMode:0x1Req On Wire: 0 SessionId:0x10006c000019
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0Share Capabilities: None Share Flags:0x30
tid: 0x1 Maximal Access:0x11f01ff1) \\file1\usenet Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:0xa3c339a0Share Capabilities: None Aligned, Partition Aligned, Share Flags:0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access:0x1f01ffMIDs:
Server interfaces: 2
0) Speed: 10000000000 bps Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities:
IPv4: 192.168.1.1753) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP
status: 1 Instance: 1
Local Users To Server: 1 SecMode:0x1Req On Wire: 2 SessionId:0x10007c000061
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0Share Capabilities: None Share Flags:0x30
tid: 0x1 Maximal Access:0x11f01ff1) \\file1\YYY Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:0xa3c339a0Share Capabilities: None Aligned, Partition Aligned, Share Flags:0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access:0x1f01ffMIDs:
State: 2 com: 8 pid: 1218 cbdata:00000000e45c59d2mid 1211489
State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553Server interfaces: 2
0) Speed: 10000000000 bps Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities:
IPv4: 192.168.1.175Number of credits: 8190 Dialect
0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP
status: 1 Instance: 1
Local Users To Server: 1 SecMode:0x1Req On Wire: 0 SessionId:0x10006c000055
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0Share Capabilities: None Share Flags:0x30
tid: 0x1 Maximal Access:0x11f01ff1) \\file1\ZZZ Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:0xa3c339a0Share Capabilities: None Aligned, Partition Aligned, Share Flags:0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access:0x1f01ffMIDs:
Server interfaces: 2
0) Speed: 10000000000 bps Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities:
IPv4: 192.168.1.175
Are you comfortable with dynamic tracing ("trace-cmd record -e ..." ... the list of trace points can be seen in /sys/kernel/debug/tracing/events/cifs)?
Quoted 231 lines · Scott M. Lewandowski
I’m also seeing entries like the following mixed in with the other messages:
[27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd: 5And
[31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate inode 00000000b4733e96Quoted 219 lines
-----Original Message-----
From: Scott M. Lewandowski <[email protected]>
Sent: Friday, May 1, 2020 12:20 AM
To: 'Steve French' <[email protected]>
Cc: CIFS <[email protected]>
Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019Quoted 2 lines
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
Not using scp at the moment.
Quoted 1 line
Could an app be sending signals trying to kill the process doing the reads?
Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.
One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105
-scott
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file)://file1/usenet
/mnt/usenetcifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,n ounix,iocharset=utf8 0 0My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.
Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:
[ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS: Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4 [ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFS VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS: Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open [ 191.784922] SMB2_read: 4 callbacks suppressed [ 191.784925] CIFS VFS: Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4 [ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFS VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS: Close unmatched open [ 198.247594] CIFS VFS: Closeunmatched open
Here is the duplicate cookie info:
[ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS-Cache: O- cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8] '020001bdc0a801af' [ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS-Cache: O- cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8] '020001bdc0a801af' [ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS-Cache: O- cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8] '020001bdc0a801af'Any ideas what could be going on?
Thanks for any help!
DebugData:
CIFS Version 2.23
Features:
DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect0x300
1) Name: 192.168.1.175 Uses: 1 Capability:0x300067Session Status: 1 TCP
status: 1 Instance: 1Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x100080000021 Shares: 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None Share Flags: 0x30 tid: 0x1 Maximal Access: 0x11f01ff 1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175Number of credits: 8190 Dialect
0x3002) Name: 192.168.1.175 Uses: 1 Capability:
0x300067Session Status: 1 TCP
status: 1 Instance: 1Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000019 Shares: 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None Share Flags: 0x30 tid: 0x1 Maximal Access: 0x11f01ff 1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.1753) Name: 192.168.1.175 Uses: 1 Capability:
0x300067Session Status: 1 TCP
status: 1 Instance: 1Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: 0x10007c000061 Shares: 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None Share Flags: 0x30 tid: 0x1 Maximal Access: 0x11f01ff 1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553 Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175Number of credits: 8190 Dialect
0x300
4) Name: 192.168.1.175 Uses: 1 Capability:0x300067Session Status: 1 TCP
status: 1 Instance: 1Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000055 Shares: 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None Share Flags: 0x30 tid: 0x1 Maximal Access: 0x11f01ff 1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175
Signature · 3 lines
Thanks, Steve
I'm trying to become comfortable now 😊
I have it setup to capture all CIFS events, but I haven't seen the messages in dmesg since. I'm not sure if I've just gotten unlucky, or if the instrumentation is changing the behavior.
Quoted 281 lines
-----Original Message-----
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 10:57 AM
To: Scott M. Lewandowski <[email protected]>
Cc: Scott M. Lewandowski <[email protected]>; CIFS <linux- [email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019Are you comfortable with dynamic tracing ("trace-cmd record -e ..." ... the list of trace points can be seen in
/sys/kernel/debug/tracing/events/cifs)?Quoted 27 lines · Scott M. Lewandowski
I’m also seeing entries like the following mixed in with the other messages:
[27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd:5
And
[31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidateinode
00000000b4733e96Quoted 13 lines
-----Original Message-----
From: Scott M. Lewandowski <[email protected]>
Sent: Friday, May 1, 2020 12:20 AM
To: 'Steve French' <[email protected]>
Cc: CIFS <[email protected]>
Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019Quoted 2 lines
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
Not using scp at the moment.
Quoted 1 line
Could an app be sending signals trying to kill the process doing the
reads?
Quoted 18 lines
Quoted 18 lines
Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.
One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105
-scott
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019Quoted 6 lines
Quoted 6 lines
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019server.
Quoted 3 lines
Quoted 3 lines
Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move
the credential file):
Quoted 13 lines
Quoted 13 lines
//file1/usenet
/mnt/usenetcifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserveri no,n
ounix,iocharset=utf8 0 0My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.
Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:
Quoted 11 lines
Quoted 11 lines
[ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS:Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4
[ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFSVFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS:
Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open [ 191.784922]SMB2_read:4 callbacks suppressed [ 191.784925] CIFS VFS:
Quoted 10 lines
Quoted 10 lines
Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4
[ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFSVFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS:
Close unmatched open [ 198.247594] CIFS VFS: Close unmatched openQuoted 3 lines
Quoted 3 lines
Here is the duplicate cookie info:
[ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS-Cache: O-
Quoted 2 lines
Quoted 2 lines
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.714330] FS-Cache: O-cookie d=00000000173ea0een=00000000243f210c
Quoted 2 lines
Quoted 2 lines
[ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c[p=0000000079ef0365
Quoted 2 lines
Quoted 2 lines
fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8]'
020001bdc0a801af'Quoted 1 line
Quoted 1 line
[ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS-Cache: O-
Quoted 2 lines
Quoted 2 lines
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.717679] FS-Cache: O-cookie d=00000000173ea0een=00000000243f210c
Quoted 2 lines
Quoted 2 lines
[ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7[p=0000000079ef0365
Quoted 2 lines
Quoted 2 lines
fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8]'
020001bdc0a801af'Quoted 1 line
Quoted 1 line
[ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS-Cache: O-
Quoted 2 lines
Quoted 2 lines
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.721053] FS-Cache: O-cookie d=00000000173ea0een=00000000243f210c
Quoted 2 lines
Quoted 2 lines
[ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.721994] FS-Cache: N-cookie c=00000000d479c294[p=0000000079ef0365
Quoted 2 lines
Quoted 2 lines
fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8]'
020001bdc0a801af'Quoted 10 lines
Quoted 10 lines
Any ideas what could be going on?
Thanks for any help!
Quoted 4 lines
Quoted 4 lines
DebugData:
CIFS Version 2.23
Features:DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C
Quoted 6 lines
Quoted 6 lines
IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect0x300
1) Name: 192.168.1.175 Uses: 1 Capability:0x300067Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x100080000021
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0
Share Capabilities: None Share Flags:0x30
tid:0x1Maximal Access:0x11f01ff1) \\file1\XXX Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 14 lines
Quoted 14 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175Number of credits: 8190 Dialect
0x300Quoted 1 line
Quoted 1 line
2) Name: 192.168.1.175 Uses: 1 Capability:
0x300067Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x10006c000019
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0
Share Capabilities: None Share Flags:0x30
tid:0x1Maximal Access:0x11f01ff1) \\file1\usenet Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 14 lines
Quoted 14 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175Quoted 1 line
Quoted 1 line
3) Name: 192.168.1.175 Uses: 1 Capability:
0x300067Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId:
0x10007c000061
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0
Share Capabilities: None Share Flags:0x30
tid:0x1Maximal Access:0x11f01ff1) \\file1\YYY Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 16 lines
Quoted 16 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid1211553
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175Quoted 2 lines
Quoted 2 lines
Number of credits: 8190 Dialect
0x300
4) Name: 192.168.1.175 Uses: 1 Capability:0x300067Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x10006c000055
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0
Share Capabilities: None Share Flags:0x30
tid:0x1Maximal Access:0x11f01ff1) \\file1\ZZZ Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 5 lines
Quoted 5 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs:Quoted 7 lines
Quoted 7 lines
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175--
Thanks,Steve
A message that is not in the archive (sent off-list or not yet received): CAH2r5mvscNft+arZuzzupNeB7df+c7fUmj=XRNZ5Y6PEts+Efw@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
For anyone interested, there is an excerpt with the second trace set at https://www.dropbox.com/s/53fewmvsj8w2fau/trace-report-filtered-trimmed?dl=0 (it’s still not exactly succinct so I didn’t want to spam the list with it)
I included the relevant dmesg output at the top of the file
Steve, thx for your help and thoughts so far!
From: Steve French <mailto:[email protected]>
Sent: Saturday, May 2, 2020 1:08 AM
To: Scott M. Lewandowski <mailto:[email protected]>
Cc: CIFS <mailto:[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019
if you have a short repro scenario then in window 1) trace-cmd record -e cifs
window 2) repro the bug
window 3) (immediately after the repro) trace-cmd show
Or if you want to make it more narrow maybe something like
"trace-cmd record -e smb3_read* -e smb3_reconnect* -e smb3_partial_send_reconnect"
or perhaps a somewhat noisier one:
"trace-cmd record -e smb3_read* -e smb3_reconnect* -e smb3_partial_send_reconnect -e smb3_cmd* -e smb3_slow_rsp -e smb3_enter -e smb3_exit*"
On Fri, May 1, 2020 at 1:20 PM Scott M. Lewandowski <mailto:[email protected]> wrote:
I'm trying to become comfortable now 😊
I have it setup to capture all CIFS events, but I haven't seen the messages in dmesg since. I'm not sure if I've just gotten unlucky, or if the instrumentation is changing the behavior.
Quoted 281 lines
-----Original Message-----
From: Steve French <mailto:[email protected]>
Sent: Friday, May 1, 2020 10:57 AM
To: Scott M. Lewandowski <mailto:[email protected]>
Cc: Scott M. Lewandowski <mailto:[email protected]>; CIFS <linux- mailto:[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019Are you comfortable with dynamic tracing ("trace-cmd record -e ..." ... the list of trace points can be seen in
/sys/kernel/debug/tracing/events/cifs)?Quoted 27 lines · Scott M. Lewandowski
I’m also seeing entries like the following mixed in with the other messages:
[27078.665192] CIFS VFS: file://file1 Cancelling wait for mid 13443257 cmd:5
And
[31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidateinode
00000000b4733e96Quoted 13 lines
-----Original Message-----
From: Scott M. Lewandowski <mailto:[email protected]>
Sent: Friday, May 1, 2020 12:20 AM
To: 'Steve French' <mailto:[email protected]>
Cc: CIFS <mailto:[email protected]>
Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019Quoted 2 lines
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
Not using scp at the moment.
Quoted 1 line
Could an app be sending signals trying to kill the process doing the
reads?
Quoted 18 lines
Quoted 18 lines
Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.
One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105
-scott
From: Steve French <mailto:[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <mailto:[email protected]>
Cc: CIFS <mailto:[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019Quoted 6 lines
Quoted 6 lines
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019server.
Quoted 3 lines
Quoted 3 lines
Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move
the credential file):
Quoted 13 lines
Quoted 13 lines
//file1/usenet
/mnt/usenetcifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserveri no,n
ounix,iocharset=utf8 0 0My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.
Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:
Quoted 11 lines
Quoted 11 lines
[ 179.625296]
SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS:
Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4 [ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFS
VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS:
Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open [ 191.784922]SMB2_read:4 callbacks suppressed [ 191.784925] CIFS VFS:
Quoted 10 lines
Quoted 10 lines
Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4 [ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFS
VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS:
Close unmatched open [ 198.247594] CIFS VFS: Close unmatched openQuoted 3 lines
Quoted 3 lines
Here is the duplicate cookie info:
[ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS-
Cache: O-
Quoted 2 lines
Quoted 2 lines
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee
n=00000000243f210c
Quoted 2 lines
Quoted 2 lines
[ 21.714779] FS-Cache: O-key=[8] '
020001bdc0a801af' [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c[p=0000000079ef0365
Quoted 2 lines
Quoted 2 lines
fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8]
'
020001bdc0a801af'Quoted 1 line
Quoted 1 line
[ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS-
Cache: O-
Quoted 2 lines
Quoted 2 lines
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee
n=00000000243f210c
Quoted 2 lines
Quoted 2 lines
[ 21.718220] FS-Cache: O-key=[8] '
020001bdc0a801af' [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7[p=0000000079ef0365
Quoted 2 lines
Quoted 2 lines
fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8]
'
020001bdc0a801af'Quoted 1 line
Quoted 1 line
[ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS-
Cache: O-
Quoted 2 lines
Quoted 2 lines
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee
n=00000000243f210c
Quoted 2 lines
Quoted 2 lines
[ 21.721518] FS-Cache: O-key=[8] '
020001bdc0a801af' [ 21.721994] FS-Cache: N-cookie c=00000000d479c294[p=0000000079ef0365
Quoted 2 lines
Quoted 2 lines
fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8]
'
020001bdc0a801af'Quoted 10 lines
Quoted 10 lines
Any ideas what could be going on?
Thanks for any help!
Quoted 4 lines
Quoted 4 lines
DebugData:
CIFS Version 2.23
Features:DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C
Quoted 6 lines
Quoted 6 lines
IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode:0x1Req On Wire: 0 SessionId:0x100080000021
Shares:
0) IPC: file://file1/IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0Share Capabilities: None Share Flags:0x30
tid: 0x1 Maximal Access:0x11f01ff1) file://file1/XXX Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 14 lines
Quoted 14 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags:
0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access:0x1f01ffMIDs:
Server interfaces: 2
0) Speed: 10000000000 bps Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities:
IPv4: 192.168.1.175
Number of credits: 8190 Dialect0x300Quoted 1 line
Quoted 1 line
2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1
TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode:0x1Req On Wire: 0 SessionId:0x10006c000019
Shares:
0) IPC: file://file1/IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0Share Capabilities: None Share Flags:0x30
tid: 0x1 Maximal Access:0x11f01ff1) file://file1/usenet Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 14 lines
Quoted 14 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags:
0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access:0x1f01ffMIDs:
Server interfaces: 2
0) Speed: 10000000000 bps Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities:
IPv4: 192.168.1.175Quoted 1 line
Quoted 1 line
3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1
TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode:0x1Req On Wire: 2 SessionId:0x10007c000061
Shares:
0) IPC: file://file1/IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0Share Capabilities: None Share Flags:0x30
tid: 0x1 Maximal Access:0x11f01ff1) file://file1/YYY Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 16 lines
Quoted 16 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags:
0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access:0x1f01ffMIDs:
State: 2 com: 8 pid: 1218 cbdata:00000000e45c59d2mid 1211489
State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553Server interfaces: 2
0) Speed: 10000000000 bps Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities:
IPv4: 192.168.1.175Quoted 2 lines
Quoted 2 lines
Number of credits: 8190 Dialect
0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode:0x1Req On Wire: 0 SessionId:0x10006c000055
Shares:
0) IPC: file://file1/IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0Share Capabilities: None Share Flags:0x30
tid: 0x1 Maximal Access:0x11f01ff1) file://file1/ZZZ Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 5 lines
Quoted 5 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags:
0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access:0x1f01ffMIDs:
Quoted 7 lines
Quoted 7 lines
Server interfaces: 2
0) Speed: 10000000000 bps Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities:
IPv4: 192.168.1.175--
Thanks,Steve
Signature · 3 lines
Thanks, Steve
I've now got a trace that corresponds to the dmesg output below. I will take a look later, but upon quick glance, this doesn't look like anything that will mean much to me, unfortunately. If someone would like to look at the file I can find a way to get it to you.
[53223.099221] CIFS VFS: Send error in read = -4 [53223.232373] CIFS VFS: Send error in read = -4 [53254.762914] CIFS VFS: Send error in read = -4 [53301.403925] CIFS VFS: Send error in read = -4 [53301.772568] CIFS VFS: Send error in read = -4 [53301.818566] CIFS VFS: Send error in read = -4 [53301.885510] CIFS VFS: Send error in read = -4 [53302.055959] CIFS VFS: Send error in read = -4 [53302.146337] CIFS VFS: Send error in read = -4 [53302.236322] CIFS VFS: Send error in read = -4 [53302.329525] CIFS VFS: Send error in read = -4 [53303.000869] CIFS VFS: Send error in read = -4 [53303.106961] CIFS VFS: Send error in read = -4 [53304.756998] CIFS VFS: Close unmatched open [53670.776832] CIFS VFS: Close unmatched open
Quoted 281 lines
-----Original Message-----
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 10:57 AM
To: Scott M. Lewandowski <[email protected]>
Cc: Scott M. Lewandowski <[email protected]>; CIFS <linux- [email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019Are you comfortable with dynamic tracing ("trace-cmd record -e ..." ... the list of trace points can be seen in
/sys/kernel/debug/tracing/events/cifs)?Quoted 27 lines · Scott M. Lewandowski
I’m also seeing entries like the following mixed in with the other messages:
[27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd:5
And
[31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidateinode
00000000b4733e96Quoted 13 lines
-----Original Message-----
From: Scott M. Lewandowski <[email protected]>
Sent: Friday, May 1, 2020 12:20 AM
To: 'Steve French' <[email protected]>
Cc: CIFS <[email protected]>
Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019Quoted 2 lines
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
Not using scp at the moment.
Quoted 1 line
Could an app be sending signals trying to kill the process doing the
reads?
Quoted 18 lines
Quoted 18 lines
Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.
One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105
-scott
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019Quoted 6 lines
Quoted 6 lines
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019server.
Quoted 3 lines
Quoted 3 lines
Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move
the credential file):
Quoted 13 lines
Quoted 13 lines
//file1/usenet
/mnt/usenetcifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserveri no,n
ounix,iocharset=utf8 0 0My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.
Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:
Quoted 11 lines
Quoted 11 lines
[ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS:Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4
[ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFSVFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS:
Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open [ 191.784922]SMB2_read:4 callbacks suppressed [ 191.784925] CIFS VFS:
Quoted 10 lines
Quoted 10 lines
Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4
[ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFSVFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS:
Close unmatched open [ 198.247594] CIFS VFS: Close unmatched openQuoted 3 lines
Quoted 3 lines
Here is the duplicate cookie info:
[ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS-Cache: O-
Quoted 2 lines
Quoted 2 lines
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.714330] FS-Cache: O-cookie d=00000000173ea0een=00000000243f210c
Quoted 2 lines
Quoted 2 lines
[ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c[p=0000000079ef0365
Quoted 2 lines
Quoted 2 lines
fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8]'
020001bdc0a801af'Quoted 1 line
Quoted 1 line
[ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS-Cache: O-
Quoted 2 lines
Quoted 2 lines
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.717679] FS-Cache: O-cookie d=00000000173ea0een=00000000243f210c
Quoted 2 lines
Quoted 2 lines
[ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7[p=0000000079ef0365
Quoted 2 lines
Quoted 2 lines
fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8]'
020001bdc0a801af'Quoted 1 line
Quoted 1 line
[ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS-Cache: O-
Quoted 2 lines
Quoted 2 lines
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.721053] FS-Cache: O-cookie d=00000000173ea0een=00000000243f210c
Quoted 2 lines
Quoted 2 lines
[ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.721994] FS-Cache: N-cookie c=00000000d479c294[p=0000000079ef0365
Quoted 2 lines
Quoted 2 lines
fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8]'
020001bdc0a801af'Quoted 10 lines
Quoted 10 lines
Any ideas what could be going on?
Thanks for any help!
Quoted 4 lines
Quoted 4 lines
DebugData:
CIFS Version 2.23
Features:DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C
Quoted 6 lines
Quoted 6 lines
IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect0x300
1) Name: 192.168.1.175 Uses: 1 Capability:0x300067Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x100080000021
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0
Share Capabilities: None Share Flags:0x30
tid:0x1Maximal Access:0x11f01ff1) \\file1\XXX Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 14 lines
Quoted 14 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175Number of credits: 8190 Dialect
0x300Quoted 1 line
Quoted 1 line
2) Name: 192.168.1.175 Uses: 1 Capability:
0x300067Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x10006c000019
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0
Share Capabilities: None Share Flags:0x30
tid:0x1Maximal Access:0x11f01ff1) \\file1\usenet Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 14 lines
Quoted 14 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175Quoted 1 line
Quoted 1 line
3) Name: 192.168.1.175 Uses: 1 Capability:
0x300067Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId:
0x10007c000061
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0
Share Capabilities: None Share Flags:0x30
tid:0x1Maximal Access:0x11f01ff1) \\file1\YYY Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 16 lines
Quoted 16 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid1211553
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175Quoted 2 lines
Quoted 2 lines
Number of credits: 8190 Dialect
0x300
4) Name: 192.168.1.175 Uses: 1 Capability:0x300067Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x10006c000055
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0
Share Capabilities: None Share Flags:0x30
tid:0x1Maximal Access:0x11f01ff1) \\file1\ZZZ Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 5 lines
Quoted 5 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs:Quoted 7 lines
Quoted 7 lines
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175--
Thanks,Steve
I would expect the close unmatched open to be related to the session being closed (thus all files implicitly closed on the server) but the session drop overlapping the application closing a file.
Quoted 302 lines · Scott M. Lewandowski
I've now got a trace that corresponds to the dmesg output below. I will take a look later, but upon quick glance, this doesn't look like anything that will mean much to me, unfortunately. If someone would like to look at the file I can find a way to get it to you.
[53223.099221] CIFS VFS: Send error in read = -4 [53223.232373] CIFS VFS: Send error in read = -4 [53254.762914] CIFS VFS: Send error in read = -4 [53301.403925] CIFS VFS: Send error in read = -4 [53301.772568] CIFS VFS: Send error in read = -4 [53301.818566] CIFS VFS: Send error in read = -4 [53301.885510] CIFS VFS: Send error in read = -4 [53302.055959] CIFS VFS: Send error in read = -4 [53302.146337] CIFS VFS: Send error in read = -4 [53302.236322] CIFS VFS: Send error in read = -4 [53302.329525] CIFS VFS: Send error in read = -4 [53303.000869] CIFS VFS: Send error in read = -4 [53303.106961] CIFS VFS: Send error in read = -4 [53304.756998] CIFS VFS: Close unmatched open [53670.776832] CIFS VFS: Close unmatched openQuoted 281 lines
-----Original Message-----
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 10:57 AM
To: Scott M. Lewandowski <[email protected]>
Cc: Scott M. Lewandowski <[email protected]>; CIFS <linux- [email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019Are you comfortable with dynamic tracing ("trace-cmd record -e ..." ... the list of trace points can be seen in
/sys/kernel/debug/tracing/events/cifs)?Quoted 27 lines · Scott M. Lewandowski
I’m also seeing entries like the following mixed in with the other messages:
[27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd:5
And
[31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidateinode
00000000b4733e96Quoted 13 lines
-----Original Message-----
From: Scott M. Lewandowski <[email protected]>
Sent: Friday, May 1, 2020 12:20 AM
To: 'Steve French' <[email protected]>
Cc: CIFS <[email protected]>
Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019Quoted 2 lines
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
Not using scp at the moment.
Quoted 1 line
Could an app be sending signals trying to kill the process doing the
reads?
Quoted 18 lines
Quoted 18 lines
Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.
One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105
-scott
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019Quoted 6 lines
Quoted 6 lines
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019server.
Quoted 3 lines
Quoted 3 lines
Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move
the credential file):
Quoted 13 lines
Quoted 13 lines
//file1/usenet
/mnt/usenetcifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserveri no,n
ounix,iocharset=utf8 0 0My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.
Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:
Quoted 11 lines
Quoted 11 lines
[ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS:Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4
[ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFSVFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS:
Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open [ 191.784922]SMB2_read:4 callbacks suppressed [ 191.784925] CIFS VFS:
Quoted 10 lines
Quoted 10 lines
Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4
[ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFSVFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS:
Close unmatched open [ 198.247594] CIFS VFS: Close unmatched openQuoted 3 lines
Quoted 3 lines
Here is the duplicate cookie info:
[ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS-Cache: O-
Quoted 2 lines
Quoted 2 lines
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.714330] FS-Cache: O-cookie d=00000000173ea0een=00000000243f210c
Quoted 2 lines
Quoted 2 lines
[ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c[p=0000000079ef0365
Quoted 2 lines
Quoted 2 lines
fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8]'
020001bdc0a801af'Quoted 1 line
Quoted 1 line
[ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS-Cache: O-
Quoted 2 lines
Quoted 2 lines
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.717679] FS-Cache: O-cookie d=00000000173ea0een=00000000243f210c
Quoted 2 lines
Quoted 2 lines
[ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7[p=0000000079ef0365
Quoted 2 lines
Quoted 2 lines
fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8]'
020001bdc0a801af'Quoted 1 line
Quoted 1 line
[ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS-Cache: O-
Quoted 2 lines
Quoted 2 lines
cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.721053] FS-Cache: O-cookie d=00000000173ea0een=00000000243f210c
Quoted 2 lines
Quoted 2 lines
[ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.721994] FS-Cache: N-cookie c=00000000d479c294[p=0000000079ef0365
Quoted 2 lines
Quoted 2 lines
fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee
n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8]'
020001bdc0a801af'Quoted 10 lines
Quoted 10 lines
Any ideas what could be going on?
Thanks for any help!
Quoted 4 lines
Quoted 4 lines
DebugData:
CIFS Version 2.23
Features:DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C
Quoted 6 lines
Quoted 6 lines
IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect0x300
1) Name: 192.168.1.175 Uses: 1 Capability:0x300067Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x100080000021
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0
Share Capabilities: None Share Flags:0x30
tid:0x1Maximal Access:0x11f01ff1) \\file1\XXX Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 14 lines
Quoted 14 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175Number of credits: 8190 Dialect
0x300Quoted 1 line
Quoted 1 line
2) Name: 192.168.1.175 Uses: 1 Capability:
0x300067Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x10006c000019
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0
Share Capabilities: None Share Flags:0x30
tid:0x1Maximal Access:0x11f01ff1) \\file1\usenet Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 14 lines
Quoted 14 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175Quoted 1 line
Quoted 1 line
3) Name: 192.168.1.175 Uses: 1 Capability:
0x300067Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId:
0x10007c000061
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0
Share Capabilities: None Share Flags:0x30
tid:0x1Maximal Access:0x11f01ff1) \\file1\YYY Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 16 lines
Quoted 16 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid1211553
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175Quoted 2 lines
Quoted 2 lines
Number of credits: 8190 Dialect
0x300
4) Name: 192.168.1.175 Uses: 1 Capability:0x300067Session Status: 1TCP
Quoted 11 lines
Quoted 11 lines
status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId:
0x10006c000055
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo:0x0Attributes:0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number:0x0
Share Capabilities: None Share Flags:0x30
tid:0x1Maximal Access:0x11f01ff1) \\file1\ZZZ Mounts: 1 DevInfo:
0x20020Attributes:0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number:
0xa3c339a0Quoted 5 lines
Quoted 5 lines
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs:Quoted 7 lines
Quoted 7 lines
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175--
Thanks,Steve
Signature · 3 lines
Thanks, Steve
I don't know much about mono but wonder if signal handling problems (in mono) could be related "EINTR" is suspicious on read ...
Don't know if this is related but google search turned up this:
"Mono uses signals on certain platforms to handle NullReferenceExceptions. It’s also becoming more and more common to add third-party libraries that detect and report crashes. The problem is when those third-party libraries install their own signal handlers, overwriting Mono’s signal handlers. When this happens, any NullReferenceException would be detected by such libraries as a fatal SIGSEGV, and the libraries would report it as a crash and terminate the process."
Quoted 199 lines · Scott M. Lewandowski
Quoted 2 lines
EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
Not using scp at the moment.
Quoted 1 line
Could an app be sending signals trying to kill the process doing the reads?
Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it.
One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105
-scott
From: Steve French <[email protected]>
Sent: Friday, May 1, 2020 12:07 AM
To: Scott M. Lewandowski <[email protected]>
Cc: CIFS <[email protected]>
Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads
On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:[email protected]> wrote:
I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file)://file1/usenet
/mnt/usenetcifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,nounix,iocharset=utf8 0 0My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.
Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:
[ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS: Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4 [ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFS VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS: Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open [ 191.784922] SMB2_read: 4 callbacks suppressed [ 191.784925] CIFS VFS: Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4 [ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFS VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS: Close unmatched open [ 198.247594] CIFS VFS: Close unmatched openHere is the duplicate cookie info:
[ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8] '020001bdc0a801af' [ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8] '020001bdc0a801af' [ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8] '020001bdc0a801af'Any ideas what could be going on?
Thanks for any help!
DebugData:
CIFS Version 2.23
Features: DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,CIFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect0x300
1) Name: 192.168.1.175 Uses: 1 Capability:0x300067Session Status: 1 TCP status: 1 Instance: 1Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x100080000021 Shares: 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None Share Flags: 0x30 tid: 0x1 Maximal Access: 0x11f01ff 1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175Number of credits: 8190 Dialect
0x3002) Name: 192.168.1.175 Uses: 1 Capability:
0x300067Session Status: 1 TCP status: 1 Instance: 1Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000019 Shares: 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None Share Flags: 0x30 tid: 0x1 Maximal Access: 0x11f01ff 1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.1753) Name: 192.168.1.175 Uses: 1 Capability:
0x300067Session Status: 1 TCP status: 1 Instance: 1Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: 0x10007c000061 Shares: 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None Share Flags: 0x30 tid: 0x1 Maximal Access: 0x11f01ff 1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553 Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175Number of credits: 8190 Dialect
0x300
4) Name: 192.168.1.175 Uses: 1 Capability:0x300067Session Status: 1 TCP status: 1 Instance: 1Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000055 Shares: 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None Share Flags: 0x30 tid: 0x1 Maximal Access: 0x11f01ff 1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175
Signature · 3 lines
Thanks, Steve
Hi Scott,
If you can reproduce this, could you try making a capture of mounting + waiting for the bug?
https://wiki.samba.org/index.php/Bug_Reporting#cifs.ko
Cheers,
Signature · 4 lines
Aurélien Aptel / SUSE Labs Samba Team GPG: 1839 CB5F 9F5B FB9B AA97 8C99 03C8 A49B 521B D5D3 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)