Skip to contentSkip to conversation

Writes greater than 64k fails with -ENOSPC

DISCUSSION Suresh Jayaraman 3 participants 15 messages Jan 29, 2013 – Feb 7, 2013 12 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi all,

I'm looking into a report on 3.0 based kernel (plus stable fixes) where writes greater than 64k to a NAS (Hitachi NAS) is failing (simple dd). The problem was not seen with a 2.6.32-ish kernel. Also, note that the problem is not seen with other Servers such as Windows 2003 or Windows 8 Servers.

The strace output shows the close() fall fails with -ENOSPC.

The relevant cFYI snip

Jan 23 08:31:45 vsusix02 kernel: [1003552.227274] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
Jan 23 08:31:45 vsusix02 kernel: [1003552.227277] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 127044
Jan 23 08:31:45 vsusix02 kernel: [1003552.345848] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.393906] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssmb.c: async write at 1015808 8192 bytes
Jan 23 08:31:45 vsusix02 kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
Jan 23 08:31:45 vsusix02 kernel: [1003552.393914] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 8260
Jan 23 08:31:45 vsusix02 kernel: [1003552.479378] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.481215] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.481260] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c: Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28

The problem seems to be that during close(), we try to flush the buffers by calling cifs_flush() which in turn will call filemap_write_and_wait() to wait on the pages under writeback to complete. do_writepages() will invoke cifs_writepages() which is perhaps returning -ENOSPC and it is propagated back.

There are no quota restrictions or disk space problems. The tcpdump output doesn't show any errors during the write or close.

I'm not sure what could be causing the problem and would appreciate any clues or debugging suggestions.

Thanks

Signature · 1 line
Suresh Jayaraman

A message that is not in the archive (sent off-list or not yet received): 5107BF75.704-IBi9RG/[email protected]

#2 lore raw
Quoted 29 lines · On Tue, 29 Jan 2013 17:54:21 +0530 Suresh Jayaraman…

Hi all,

I'm looking into a report on 3.0 based kernel (plus stable fixes) where writes greater than 64k to a NAS (Hitachi NAS) is failing (simple dd). The problem was not seen with a 2.6.32-ish kernel. Also, note that the problem is not seen with other Servers such as Windows 2003 or Windows 8 Servers.

The strace output shows the close() fall fails with -ENOSPC.

The relevant cFYI snip

Jan 23 08:31:45 vsusix02 kernel: [1003552.227274] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
Jan 23 08:31:45 vsusix02 kernel: [1003552.227277] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 127044
Jan 23 08:31:45 vsusix02 kernel: [1003552.345848] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.393906] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssmb.c: async write at 1015808 8192 bytes
Jan 23 08:31:45 vsusix02 kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
Jan 23 08:31:45 vsusix02 kernel: [1003552.393914] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 8260
Jan 23 08:31:45 vsusix02 kernel: [1003552.479378] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.481215] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.481260] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c: Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28

The problem seems to be that during close(), we try to flush the buffers by calling cifs_flush() which in turn will call filemap_write_and_wait() to wait on the pages under writeback to complete. do_writepages() will invoke cifs_writepages() which is perhaps returning -ENOSPC and it is propagated back.

Quoted 8 lines

There are no quota restrictions or disk space problems. The tcpdump output doesn't show any errors during the write or close.

I'm not sure what could be causing the problem and would appreciate any clues or debugging suggestions.

Thanks

I'd look for "short" write replies. We have this in cifs_writev_callback:

                if (written < wdata->bytes)
                        wdata->result = -ENOSPC;

IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX responses, but I remember it not being 100% clear...

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20130129192913.4942c635-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#4 lore raw
Quoted 64 lines

-----Original Message-----
From: [email protected] [mailto:linux-cifs- [email protected]] On Behalf Of Jeff Layton
Sent: Tuesday, January 29, 2013 7:29 PM
To: Suresh Jayaraman
Cc: linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 53 lines · On Tue, 29 Jan 2013 17:54:21 +0530 Suresh Jayaraman…

Hi all,

I'm looking into a report on 3.0 based kernel (plus stable fixes) where writes greater than 64k to a NAS (Hitachi NAS) is failing (simple dd). The problem was not seen with a 2.6.32-ish kernel. Also, note that the problem is not seen with other Servers such as Windows 2003 or Windows 8 Servers.

The strace output shows the close() fall fails with -ENOSPC.

The relevant cFYI snip

Jan 23 08:31:45 vsusix02 kernel: [1003552.227274]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
[1003552.227277]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
ort.c: Sending smb:  total_len 127044 Jan 23 08:31:45 vsusix02 kernel:
[1003552.345848]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
[1003552.393906]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssm b.c: async write at 1015808 8192 bytes Jan 23 08:31:45 vsusix02
kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:

[1003552.393914]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
ort.c: Sending smb:  total_len 8260 Jan 23 08:31:45 vsusix02 kernel:
[1003552.479378]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
[1003552.481215]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
[1003552.481260]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c : Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28

The problem seems to be that during close(), we try to flush the buffers by calling cifs_flush() which in turn will call filemap_write_and_wait() to wait on the pages under writeback to complete. do_writepages() will invoke cifs_writepages() which is perhaps returning -ENOSPC and it is propagated back.

There are no quota restrictions or disk space problems. The tcpdump output doesn't show any errors during the write or close.

I'm not sure what could be causing the problem and would appreciate any clues or debugging suggestions.

Quoted 8 lines
Quoted 2 lines

Thanks

I'd look for "short" write replies. We have this in cifs_writev_callback:

                if (written < wdata->bytes)
                        wdata->result = -ENOSPC;

If the server does not return an error, then the client can conclude that at least something was written, based on the value of CountOfBytesWritten in the response. Unless this is zero, I think the client substituting ENOSPC may be incorrect.

Quoted 2 lines

IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX responses, but I remember it not being 100% clear...

File a doc issue! Happy to clarify it if needed.

Tom.

A message that is not in the archive (sent off-list or not yet received): 614F550557B82C44AC27C492ADA391AA044F2484-Jfd81uAzPQsmBPcf4Gwd0Fir+X/St4rqwBk/1ggFUS45P9zcU8sUGwC/[email protected]

#6 lore raw
Quoted 67 lines · On Wed, 30 Jan 2013 14:06:19 +0000 Tom Talpey…
Quoted 67 lines

-----Original Message-----
From: [email protected] [mailto:linux-cifs- [email protected]] On Behalf Of Jeff Layton
Sent: Tuesday, January 29, 2013 7:29 PM
To: Suresh Jayaraman
Cc: linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 55 lines · On Tue, 29 Jan 2013 17:54:21 +0530 Suresh Jayaraman…

Hi all,

I'm looking into a report on 3.0 based kernel (plus stable fixes) where writes greater than 64k to a NAS (Hitachi NAS) is failing (simple dd). The problem was not seen with a 2.6.32-ish kernel. Also, note that the problem is not seen with other Servers such as Windows 2003 or Windows 8 Servers.

The strace output shows the close() fall fails with -ENOSPC.

The relevant cFYI snip

Jan 23 08:31:45 vsusix02 kernel: [1003552.227274]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
[1003552.227277]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
ort.c: Sending smb:  total_len 127044 Jan 23 08:31:45 vsusix02 kernel:
[1003552.345848]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
[1003552.393906]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssm b.c: async write at 1015808 8192 bytes Jan 23 08:31:45 vsusix02
kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:

[1003552.393914]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
ort.c: Sending smb:  total_len 8260 Jan 23 08:31:45 vsusix02 kernel:
[1003552.479378]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
[1003552.481215]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
[1003552.481260]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c : Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28

The problem seems to be that during close(), we try to flush the buffers by calling cifs_flush() which in turn will call filemap_write_and_wait() to wait on the pages under writeback to complete. do_writepages() will invoke cifs_writepages() which is perhaps returning -ENOSPC and it is propagated back.

There are no quota restrictions or disk space problems. The tcpdump output doesn't show any errors during the write or close.

I'm not sure what could be causing the problem and would appreciate any clues or debugging suggestions.

Thanks

Quoted 8 lines
Quoted 5 lines

I'd look for "short" write replies. We have this in cifs_writev_callback:

                if (written < wdata->bytes)
                        wdata->result = -ENOSPC;

If the server does not return an error, then the client can conclude that at least something was written, based on the value of CountOfBytesWritten in the response. Unless this is zero, I think the client substituting ENOSPC may be incorrect.

That's probably the case, but short writes are really tricky to deal with...

Say we're spraying out a bunch of 64k writes, but the server then replies that only 63k of each was written. Now we have to go back and redirty the pages that didn't get fully written. That's fine and not too hard to do, but now you have a bunch of dirty pages sprinkled around the file every 63k.

So, on the next pass through the dirty page radix tree, we go and try to issue another WRITE_ANDX call. I'm not sure if we'd have to go back and restart writepages ourselves at that point for the WB_SYNC_ALL case.

Note that this sort of server behavior was the crux of my argument with Steve a few months ago about defaulting to 64k writes on servers w/o POSIX extensions. We can be reasonably sure that most servers handle 64k writes OK since that's what Windows does.

The spec is not 100% clear on whether servers are *required* to support arbitrarily large writes up to the 128k limit. Clearly there are some that do not, and a larger default is problematic against those servers.

Quoted 5 lines
Quoted 2 lines

IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX responses, but I remember it not being 100% clear...

File a doc issue! Happy to clarify it if needed.

That might be nice to do once Suresh confirms whether that's the case.

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20130130093729.736ad582-4QP7MXygkU+dMjc06nkz3ljfA9RmPOcC@public.gmane.org

#8 lore raw
Quoted 105 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Wednesday, January 30, 2013 9:37 AM
To: Tom Talpey
Cc: Suresh Jayaraman; linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 76 lines · On Wed, 30 Jan 2013 14:06:19 +0000 Tom Talpey…
Quoted 76 lines

-----Original Message-----
From: [email protected] [mailto:linux-cifs- [email protected]] On Behalf Of Jeff Layton
Sent: Tuesday, January 29, 2013 7:29 PM
To: Suresh Jayaraman
Cc: linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 64 lines · On Tue, 29 Jan 2013 17:54:21 +0530 Suresh Jayaraman…

Hi all,

I'm looking into a report on 3.0 based kernel (plus stable fixes) where writes greater than 64k to a NAS (Hitachi NAS) is failing (simple dd). The problem was not seen with a 2.6.32-ish kernel. Also, note that the problem is not seen with other Servers such as Windows 2003 or Windows 8 Servers.

The strace output shows the close() fall fails with -ENOSPC.

The relevant cFYI snip

Jan 23 08:31:45 vsusix02 kernel: [1003552.227274]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
ansp
ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
[1003552.227277]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
ansp
ort.c: Sending smb:  total_len 127044 Jan 23 08:31:45 vsusix02 kernel:
[1003552.345848]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/co
nnec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
[1003552.393906]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/ci fssm
b.c: async write at 1015808 8192 bytes Jan 23 08:31:45 vsusix02
kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr ansp
ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:

[1003552.393914]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
ansp
ort.c: Sending smb:  total_len 8260 Jan 23 08:31:45 vsusix02 kernel:
[1003552.479378]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/co
nnec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
[1003552.481215]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/co
nnec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
[1003552.481260]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/fi le.c
: Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28

The problem seems to be that during close(), we try to flush the buffers by calling cifs_flush() which in turn will call filemap_write_and_wait() to wait on the pages under writeback to complete. do_writepages() will invoke cifs_writepages() which is perhaps returning -ENOSPC and it is propagated back.

There are no quota restrictions or disk space problems. The tcpdump output doesn't show any errors during the write or close.

I'm not sure what could be causing the problem and would appreciate any clues or debugging suggestions.

Thanks

Quoted 7 lines
Quoted 5 lines

I'd look for "short" write replies. We have this in cifs_writev_callback:

                if (written < wdata->bytes)
                        wdata->result = -ENOSPC;

If the server does not return an error, then the client can conclude that at

least something was written, based on the value of CountOfBytesWritten in the response. Unless this is zero, I think the client substituting ENOSPC may be incorrect.

Quoted 1 line

That's probably the case, but short writes are really tricky to deal with...

Say we're spraying out a bunch of 64k writes, but the server then replies that only 63k of each was written. Now we have to go back and redirty the pages that didn't get fully written. That's fine and not too hard to do, but now you have a bunch of dirty pages sprinkled around the file every 63k.

Quoted 8 lines

So, on the next pass through the dirty page radix tree, we go and try to issue another WRITE_ANDX call. I'm not sure if we'd have to go back and restart writepages ourselves at that point for the WB_SYNC_ALL case.

Note that this sort of server behavior was the crux of my argument with Steve a few months ago about defaulting to 64k writes on servers w/o POSIX extensions. We can be reasonably sure that most servers handle 64k writes OK since that's what Windows does.

Yes, I think you can count on the ability to handle up to 64KB from a server, you can certainly count on it from Windows.

But if the server does not indicate CAP_LARGE_WRITEX, then you can be certain it does not support >64KB. Signing has an effect on large writes too. See MS-SMB section 2.2.4.5.2.1 and 2.2.4.3.2, for example.

I still think it's questionable for the client to unconditionally signal a short write as an ENOSPC error. Yes, it's a very unhelpful server, but it did write some data. It seems wrong to ignore that, at this level.

Quoted 4 lines

The spec is not 100% clear on whether servers are *required* to support arbitrarily large writes up to the 128k limit. Clearly there are some that do not, and a larger default is problematic against those servers.

I'd be very interested to see traces of negotiate, large read and large write from such a server.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX responses, but I remember it not being 100% clear...

File a doc issue! Happy to clarify it if needed.

That might be nice to do once Suresh confirms whether that's the case.

Sounds good.

Tom.

A message that is not in the archive (sent off-list or not yet received): 614F550557B82C44AC27C492ADA391AA044F2E62-Jfd81uAzPQsmBPcf4Gwd0Fir+X/St4rqwBk/1ggFUS45P9zcU8sUGwC/[email protected]

#10 lore raw
Quoted 109 lines · On Wed, 30 Jan 2013 16:06:26 +0000 Tom Talpey…
Quoted 109 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Wednesday, January 30, 2013 9:37 AM
To: Tom Talpey
Cc: Suresh Jayaraman; linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 76 lines · On Wed, 30 Jan 2013 14:06:19 +0000 Tom Talpey…
Quoted 76 lines

-----Original Message-----
From: [email protected] [mailto:linux-cifs- [email protected]] On Behalf Of Jeff Layton
Sent: Tuesday, January 29, 2013 7:29 PM
To: Suresh Jayaraman
Cc: linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 64 lines · On Tue, 29 Jan 2013 17:54:21 +0530 Suresh Jayaraman…

Hi all,

I'm looking into a report on 3.0 based kernel (plus stable fixes) where writes greater than 64k to a NAS (Hitachi NAS) is failing (simple dd). The problem was not seen with a 2.6.32-ish kernel. Also, note that the problem is not seen with other Servers such as Windows 2003 or Windows 8 Servers.

The strace output shows the close() fall fails with -ENOSPC.

The relevant cFYI snip

Jan 23 08:31:45 vsusix02 kernel: [1003552.227274]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
ansp
ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
[1003552.227277]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
ansp
ort.c: Sending smb:  total_len 127044 Jan 23 08:31:45 vsusix02 kernel:
[1003552.345848]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/co
nnec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
[1003552.393906]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/ci fssm
b.c: async write at 1015808 8192 bytes Jan 23 08:31:45 vsusix02
kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr ansp
ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:

[1003552.393914]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
ansp
ort.c: Sending smb:  total_len 8260 Jan 23 08:31:45 vsusix02 kernel:
[1003552.479378]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/co
nnec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
[1003552.481215]
/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/co
nnec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
[1003552.481260]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/fi le.c
: Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28

The problem seems to be that during close(), we try to flush the buffers by calling cifs_flush() which in turn will call filemap_write_and_wait() to wait on the pages under writeback to complete. do_writepages() will invoke cifs_writepages() which is perhaps returning -ENOSPC and it is propagated back.

There are no quota restrictions or disk space problems. The tcpdump output doesn't show any errors during the write or close.

I'm not sure what could be causing the problem and would appreciate any clues or debugging suggestions.

Thanks

Quoted 7 lines
Quoted 5 lines

I'd look for "short" write replies. We have this in cifs_writev_callback:

                if (written < wdata->bytes)
                        wdata->result = -ENOSPC;

If the server does not return an error, then the client can conclude that at

least something was written, based on the value of CountOfBytesWritten in the response. Unless this is zero, I think the client substituting ENOSPC may be incorrect.

Quoted 1 line

That's probably the case, but short writes are really tricky to deal with...

Say we're spraying out a bunch of 64k writes, but the server then replies that only 63k of each was written. Now we have to go back and redirty the pages that didn't get fully written. That's fine and not too hard to do, but now you have a bunch of dirty pages sprinkled around the file every 63k.

So, on the next pass through the dirty page radix tree, we go and try to issue another WRITE_ANDX call. I'm not sure if we'd have to go back and restart writepages ourselves at that point for the WB_SYNC_ALL case.

Quoted 9 lines
Quoted 4 lines

Note that this sort of server behavior was the crux of my argument with Steve a few months ago about defaulting to 64k writes on servers w/o POSIX extensions. We can be reasonably sure that most servers handle 64k writes OK since that's what Windows does.

Yes, I think you can count on the ability to handle up to 64KB from a server, you can certainly count on it from Windows.

But if the server does not indicate CAP_LARGE_WRITEX, then you can be certain it does not support >64KB. Signing has an effect on large writes too. See MS-SMB section 2.2.4.5.2.1 and 2.2.4.3.2, for example.

Right, and we do handle that case correctly, AFAIK.

Quoted 2 lines

I still think it's questionable for the client to unconditionally signal a short write as an ENOSPC error. Yes, it's a very unhelpful server, but it did write some data. It seems wrong to ignore that, at this level.

Yes, almost certainly, but when I was reading the spec a couple of years ago, that wasn't completely clear. IIRC, one way to interpret it was that a short write meant the equivalent of an ENOSPC error.

Now that this code has been in the field for a bit, it seems like we see similar responses when a server just can't handle writes larger than some arbitrary size (usu. 64k).

Quoted 7 lines
Quoted 4 lines

The spec is not 100% clear on whether servers are *required* to support arbitrarily large writes up to the 128k limit. Clearly there are some that do not, and a larger default is problematic against those servers.

I'd be very interested to see traces of negotiate, large read and large write from such a server.

It's almost assuredly sending CAP_LARGE_WRITEX or we'd cap this at the MaxBufferSize. The spec says that that allows the client to exceed the MaxBufferSize on a write, but it studiously does not say by how much. :)

It's clear in hindsight that a lot of server implementors just did the bare minimum and that they only handle what Windows clients will send.

Quoted 12 lines
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX responses, but I remember it not being 100% clear...

File a doc issue! Happy to clarify it if needed.

That might be nice to do once Suresh confirms whether that's the case.

Sounds good.

Tom.

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20130130124932.419f2d36-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#12 lore raw
Quoted 168 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Wednesday, January 30, 2013 12:50 PM
To: Tom Talpey
Cc: Suresh Jayaraman; linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 18 lines · On Wed, 30 Jan 2013 16:06:26 +0000 Tom Talpey…
Quoted 18 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Wednesday, January 30, 2013 9:37 AM
To: Tom Talpey
Cc: Suresh Jayaraman; linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 8 lines · On Wed, 30 Jan 2013 14:06:19 +0000 Tom Talpey…
Quoted 8 lines

-----Original Message-----
From: [email protected] [mailto:linux-cifs- [email protected]] On Behalf Of Jeff Layton
Sent: Tuesday, January 29, 2013 7:29 PM
To: Suresh Jayaraman
Cc: linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines · On Tue, 29 Jan 2013 17:54:21 +0530 Suresh Jayaraman…

Hi all,

I'm looking into a report on 3.0 based kernel (plus stable fixes) where writes greater than 64k to a NAS (Hitachi NAS) is failing (simple dd). The problem was not seen with a 2.6.32-ish

kernel.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Also, note that the problem is not seen with other Servers such as Windows 2003 or Windows 8 Servers.

The strace output shows the close() fall fails with -ENOSPC.

The relevant cFYI snip

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Jan 23 08:31:45 vsusix02 kernel: [1003552.227274]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif s/tr
ansp
ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:

[1003552.227277]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif s/tr
ansp
ort.c: Sending smb: total_len 127044 Jan 23 08:31:45 vsusix02

kernel:

Quoted 66 lines
Quoted 66 lines
Quoted 53 lines
Quoted 53 lines
Quoted 52 lines
[1003552.345848]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif s/co
nnec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:

[1003552.393906]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif s/ci
fssm
b.c: async write at 1015808 8192 bytes Jan 23 08:31:45 vsusix02
kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif s/tr
ansp
ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:

[1003552.393914]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif s/tr
ansp
ort.c: Sending smb: total_len 8260 Jan 23 08:31:45 vsusix02 kernel:

[1003552.479378]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif s/co
nnec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:

[1003552.481215]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif s/co
nnec
t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:

[1003552.481260]

/usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif s/fi
le.c
: Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28

The problem seems to be that during close(), we try to flush the buffers by calling cifs_flush() which in turn will call filemap_write_and_wait() to wait on the pages under writeback to complete. do_writepages() will invoke cifs_writepages() which is perhaps returning -ENOSPC and it is propagated back.

There are no quota restrictions or disk space problems. The tcpdump output doesn't show any errors during the write or close.

I'm not sure what could be causing the problem and would appreciate any clues or debugging suggestions.

Thanks

Quoted 8 lines
Quoted 5 lines

I'd look for "short" write replies. We have this in cifs_writev_callback:

                if (written < wdata->bytes)
                        wdata->result = -ENOSPC;

If the server does not return an error, then the client can conclude that at

least something was written, based on the value of CountOfBytesWritten in the response. Unless this is zero, I think the client substituting ENOSPC may be incorrect.

Quoted 1 line
Quoted 6 lines
Quoted 6 lines

That's probably the case, but short writes are really tricky to deal with...

Say we're spraying out a bunch of 64k writes, but the server then replies that only 63k of each was written. Now we have to go back and redirty the pages that didn't get fully written. That's fine and not too hard to do, but now you have a bunch of dirty pages sprinkled

around the file every 63k.

Quoted 4 lines
Quoted 4 lines

So, on the next pass through the dirty page radix tree, we go and try to issue another WRITE_ANDX call. I'm not sure if we'd have to go back and restart writepages ourselves at that point for the

WB_SYNC_ALL case.

Quoted 7 lines
Quoted 5 lines

Note that this sort of server behavior was the crux of my argument with Steve a few months ago about defaulting to 64k writes on servers w/o POSIX extensions. We can be reasonably sure that most servers handle 64k writes OK since that's what Windows does.

Yes, I think you can count on the ability to handle up to 64KB from a server,

you can certainly count on it from Windows.

Quoted 2 lines

But if the server does not indicate CAP_LARGE_WRITEX, then you can be

certain it does not support >64KB. Signing has an effect on large writes too. See MS-SMB section 2.2.4.5.2.1 and 2.2.4.3.2, for example.

Quoted 1 line

Right, and we do handle that case correctly, AFAIK.

Quoted 1 line

I still think it's questionable for the client to unconditionally signal a short

write as an ENOSPC error. Yes, it's a very unhelpful server, but it did write some data. It seems wrong to ignore that, at this level.

Quoted 1 line

Yes, almost certainly, but when I was reading the spec a couple of years ago, that wasn't completely clear. IIRC, one way to interpret it was that a short write meant the equivalent of an ENOSPC error.

Now that this code has been in the field for a bit, it seems like we see similar responses when a server just can't handle writes larger than some arbitrary size (usu. 64k).

Quoted 4 lines
Quoted 4 lines

The spec is not 100% clear on whether servers are *required* to support arbitrarily large writes up to the 128k limit. Clearly there are some that do not, and a larger default is problematic against those

servers.

Quoted 1 line
Quoted 7 lines
Quoted 1 line

I'd be very interested to see traces of negotiate, large read and large write

from such a server.

Quoted 1 line

It's almost assuredly sending CAP_LARGE_WRITEX or we'd cap this at the MaxBufferSize. The spec says that that allows the client to exceed the MaxBufferSize on a write, but it studiously does not say by how much. :)

Well, except that it's not known what this server is advertising for MaxBufferSize. Maybe it advertises a large value but can't actually handle it. The trace would tell.

Tom.

Quoted 20 lines

It's clear in hindsight that a lot of server implementors just did the bare minimum and that they only handle what Windows clients will send.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX responses, but I remember it not being 100% clear...

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

File a doc issue! Happy to clarify it if needed.

That might be nice to do once Suresh confirms whether that's the case.

Sounds good.

Tom.

--
Jeff Layton <[email protected]>

#13 lore raw
In reply to a message not in the archive
Quoted 8 lines · Tom Talpey
Quoted 8 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Wednesday, January 30, 2013 9:37 AM
To: Tom Talpey
Cc: Suresh Jayaraman; linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 5 lines
Quoted 3 lines

The spec is not 100% clear on whether servers are *required* to support arbitrarily large writes up to the 128k limit. Clearly there are some that do not, and a larger default is problematic against those servers.

I'd be very interested to see traces of negotiate, large read and large write from such a server.

I'm not sure whether I can share the complete trace (without customer's permission) but I can get you the specific bits that might be interesting. I have asked for a full trace (including negotiate protocol).

Thanks

Signature · 1 line
Suresh Jayaraman

A message that is not in the archive (sent off-list or not yet received): 510A3ECF.2080505-IBi9RG/[email protected]

#15 lore raw
Quoted 13 lines · On Thu, 31 Jan 2013 15:22:15 +0530 Suresh Jayaraman…
Quoted 12 lines · Tom Talpey
Quoted 11 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Wednesday, January 30, 2013 9:37 AM
To: Tom Talpey
Cc: Suresh Jayaraman; linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

The spec is not 100% clear on whether servers are *required* to support arbitrarily large writes up to the 128k limit. Clearly there are some that do not, and a larger default is problematic against those servers.

Quoted 10 lines
Quoted 1 line

I'd be very interested to see traces of negotiate, large read and large write from such a server.

I'm not sure whether I can share the complete trace (without customer's permission) but I can get you the specific bits that might be interesting. I have asked for a full trace (including negotiate protocol).

Thanks

We probably don't need the whole capture. Just the "Capabilities" and "MaxBufferSize" values from the NEGOTIATE would be enough...

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20130131062400.6aa8fb6d-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#17 lore raw
Quoted 23 lines · Jeff Layton
Quoted 20 lines · On Thu, 31 Jan 2013 15:22:15 +0530 Suresh Jayaraman…
Quoted 8 lines · Tom Talpey
Quoted 8 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Wednesday, January 30, 2013 9:37 AM
To: Tom Talpey
Cc: Suresh Jayaraman; linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 5 lines
Quoted 3 lines

The spec is not 100% clear on whether servers are *required* to support arbitrarily large writes up to the 128k limit. Clearly there are some that do not, and a larger default is problematic against those servers.

I'd be very interested to see traces of negotiate, large read and large write from such a server.

I'm not sure whether I can share the complete trace (without customer's permission) but I can get you the specific bits that might be interesting. I have asked for a full trace (including negotiate protocol).

Quoted 7 lines
Quoted 3 lines

Thanks

We probably don't need the whole capture. Just the "Capabilities" and "MaxBufferSize" values from the NEGOTIATE would be enough...

The commit ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 did fix the problem.

Finally, got the packet capture too. From the NEGOTIATE response, the MaxBufferSize was 65280 and CAP_LARGE_WRITEX (and LARGE_READX) had been set.

The complete Capabilities were

.... .... .... .... .... .... .... ...0 = Raw Mode: Read Raw and Write Raw are not supported .... .... .... .... .... .... .... ..0. = MPX Mode: Read Mpx and Write Mpx are not supported .... .... .... .... .... .... .... .1.. = Unicode: Unicode strings are supported
.... .... .... .... .... .... .... 1... = Large Files: Large files are supported
.... .... .... .... .... .... ...1 .... = NT SMBs: NT SMBs are supported
.... .... .... .... .... .... ..1. .... = RPC Remote APIs: RPC remote APIs are supported
.... .... .... .... .... .... .1.. .... = NT Status Codes: NT status codes are supported
.... .... .... .... .... .... 1... .... = Level 2 Oplocks: Level 2 oplocks are supported
.... .... .... .... .... ...0 .... .... = Lock and Read: Lock and Read is not supported
.... .... .... .... .... ..1. .... .... = NT Find: NT Find is supported
.... .... .... .... ...1 .... .... .... = Dfs: Dfs is supported
.... .... .... .... ..1. .... .... .... = Infolevel Passthru: NT information level request passthrough is supported .... .... .... .... .1.. .... .... .... = Large ReadX: Large Read andX is supported
.... .... .... .... 1... .... .... .... = Large WriteX: Large Write andX is supported
.... .... .... ...0 .... .... .... .... = LWIO: LWIO ioctl/fsctl is not supported
.... .... 0... .... .... .... .... .... = UNIX: UNIX extensions are not supported
.... ..0. .... .... .... .... .... .... = Compressed Data: Compressed data transfer is not supported ..0. .... .... .... .... .... .... .... = Dynamic Reauth: Dynamic Reauth is not supported
0... .... .... .... .... .... .... .... = Extended Security: Extended security exchanges are not supported

I'm not sure whether restricting it to MaxBufferSize even in case CAP_LARGE_WRITEX was set would make sense..

Signature · 1 line
Suresh Jayaraman

A message that is not in the archive (sent off-list or not yet received): 51136B34.30609-IBi9RG/[email protected]

#19 lore raw
Quoted 60 lines · On Thu, 07 Feb 2013 14:22:04 +0530 Suresh Jayaraman…
Quoted 23 lines · Jeff Layton
Quoted 20 lines · On Thu, 31 Jan 2013 15:22:15 +0530 Suresh Jayaraman…
Quoted 8 lines · Tom Talpey
Quoted 8 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Wednesday, January 30, 2013 9:37 AM
To: Tom Talpey
Cc: Suresh Jayaraman; linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 5 lines
Quoted 3 lines

The spec is not 100% clear on whether servers are *required* to support arbitrarily large writes up to the 128k limit. Clearly there are some that do not, and a larger default is problematic against those servers.

I'd be very interested to see traces of negotiate, large read and large write from such a server.

I'm not sure whether I can share the complete trace (without customer's permission) but I can get you the specific bits that might be interesting. I have asked for a full trace (including negotiate protocol).

Quoted 7 lines
Quoted 3 lines

Thanks

We probably don't need the whole capture. Just the "Capabilities" and "MaxBufferSize" values from the NEGOTIATE would be enough...

The commit ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 did fix the problem.

Finally, got the packet capture too. From the NEGOTIATE response, the MaxBufferSize was 65280 and CAP_LARGE_WRITEX (and LARGE_READX) had been set.

The complete Capabilities were

.... .... .... .... .... .... .... ...0 = Raw Mode: Read Raw and Write Raw are not supported .... .... .... .... .... .... .... ..0. = MPX Mode: Read Mpx and Write Mpx are not supported .... .... .... .... .... .... .... .1.. = Unicode: Unicode strings are supported
.... .... .... .... .... .... .... 1... = Large Files: Large files are supported
.... .... .... .... .... .... ...1 .... = NT SMBs: NT SMBs are supported
.... .... .... .... .... .... ..1. .... = RPC Remote APIs: RPC remote APIs are supported
.... .... .... .... .... .... .1.. .... = NT Status Codes: NT status codes are supported
.... .... .... .... .... .... 1... .... = Level 2 Oplocks: Level 2 oplocks are supported
.... .... .... .... .... ...0 .... .... = Lock and Read: Lock and Read is not supported
.... .... .... .... .... ..1. .... .... = NT Find: NT Find is supported
.... .... .... .... ...1 .... .... .... = Dfs: Dfs is supported
.... .... .... .... ..1. .... .... .... = Infolevel Passthru: NT information level request passthrough is supported .... .... .... .... .1.. .... .... .... = Large ReadX: Large Read andX is supported
.... .... .... .... 1... .... .... .... = Large WriteX: Large Write andX is supported
.... .... .... ...0 .... .... .... .... = LWIO: LWIO ioctl/fsctl is not supported
.... .... 0... .... .... .... .... .... = UNIX: UNIX extensions are not supported
.... ..0. .... .... .... .... .... .... = Compressed Data: Compressed data transfer is not supported ..0. .... .... .... .... .... .... .... = Dynamic Reauth: Dynamic Reauth is not supported
0... .... .... .... .... .... .... .... = Extended Security: Extended security exchanges are not supported

Quoted 2 lines

I'm not sure whether restricting it to MaxBufferSize even in case CAP_LARGE_WRITEX was set would make sense..

No, I think that server just happens to have an unusually large MaxBufferSize. Most servers send values that are smaller than that. The whole idea behind CAP_LARGE_WRITEX is that it allows you to send a write that's larger than the MaxBufferSize.

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20130207065018.0c18868e-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#21 lore raw
Quoted 81 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Thursday, February 7, 2013 6:50 AM
To: Suresh Jayaraman
Cc: Tom Talpey; linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 13 lines · On Thu, 07 Feb 2013 14:22:04 +0530 Suresh Jayaraman…
Quoted 12 lines · Jeff Layton
Quoted 9 lines · On Thu, 31 Jan 2013 15:22:15 +0530 Suresh Jayaraman…
Quoted 8 lines · Tom Talpey
Quoted 8 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Wednesday, January 30, 2013 9:37 AM
To: Tom Talpey
Cc: Suresh Jayaraman; linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

The spec is not 100% clear on whether servers are *required* to support arbitrarily large writes up to the 128k limit. Clearly there are some that do not, and a larger default is problematic against

those servers.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I'd be very interested to see traces of negotiate, large read and large

write from such a server.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

I'm not sure whether I can share the complete trace (without customer's
permission) but I can get you the specific bits that might be interesting. I have asked for a full trace (including negotiate protocol).

Thanks

Quoted 5 lines
Quoted 3 lines

We probably don't need the whole capture. Just the "Capabilities" and "MaxBufferSize" values from the NEGOTIATE would be enough...

The commit ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 did fix the

problem.

Quoted 3 lines

Finally, got the packet capture too. From the NEGOTIATE response, the MaxBufferSize was 65280 and CAP_LARGE_WRITEX (and LARGE_READX)

had been set.

Quoted 30 lines

The complete Capabilities were

.... .... .... .... .... .... .... ...0 = Raw Mode: Read Raw and Write Raw are not supported .... .... .... .... .... .... .... ..0. = MPX
Mode: Read Mpx and Write Mpx are not supported .... .... .... .... .... .... .... .1.. = Unicode: Unicode strings are supported .... .... .... .... .... .... .... 1... = Large Files: Large files are supported .... .... .... .... .... .... ...1 .... = NT SMBs: NT SMBs are supported .... .... .... .... .... .... ..1. .... = RPC Remote APIs:
RPC remote APIs are supported .... .... .... .... .... .... .1.. .... = NT Status Codes: NT status codes are supported .... .... .... .... .... .... 1... .... = Level 2 Oplocks: Level 2 oplocks are supported .... .... .... .... .... ...0 .... .... = Lock and Read: Lock and Read is not supported .... .... .... .... .... ..1. .... .... = NT Find: NT Find is supported .... .... .... .... ...1 .... .... .... = Dfs: Dfs is supported .... .... .... .... ..1. .... .... .... = Infolevel
Passthru: NT information level request passthrough is supported .... .... .... .... .1.. .... .... .... = Large ReadX: Large Read andX is supported .... .... .... .... 1... .... .... .... = Large WriteX:
Large Write andX is supported .... .... .... ...0 .... .... .... .... = LWIO: LWIO ioctl/fsctl is not supported .... .... 0... .... .... .... .... .... = UNIX: UNIX extensions are not supported .... ..0. .... .... .... .... .... .... = Compressed Data: Compressed data transfer is not supported ..0. .... .... .... .... .... .... .... = Dynamic Reauth: Dynamic Reauth is not supported 0... .... .... .... .... .... .... .... = Extended Security: Extended security exchanges are not supported

Quoted 8 lines
Quoted 1 line

I'm not sure whether restricting it to MaxBufferSize even in case

CAP_LARGE_WRITEX was set would make sense..

Quoted 1 line

No, I think that server just happens to have an unusually large MaxBufferSize. Most servers send values that are smaller than that. The whole idea behind CAP_LARGE_WRITEX is that it allows you to send a write that's larger than the MaxBufferSize.

I agree, but it's still unhelpful that this server accepts only 64KB in response to the client's 124KB large write. In my opinion the server should not be offering CAP_LARGE_WRITEX, although it's certainly legal for it to do so. Perhaps your client should be ignoring it, when MaxBufferSize permits >64KB.

BTW, we're investigating the large write size behaviors for a related MS-SMB doc bug. I don't have the full picture yet, but it appears that Windows SMB servers will process somewhere between 124KB and 128KB-1 bytes when issued a large WRITE_ANDX in this situation.

Tom.

A message that is not in the archive (sent off-list or not yet received): 614F550557B82C44AC27C492ADA391AA044FC781-Jfd81uAzPQsmBPcf4Gwd0Fir+X/St4rqwBk/1ggFUS45P9zcU8sUGwC/[email protected]

#23 lore raw
Quoted 85 lines · On Thu, 7 Feb 2013 13:37:00 +0000 Tom Talpey…
Quoted 85 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Thursday, February 7, 2013 6:50 AM
To: Suresh Jayaraman
Cc: Tom Talpey; linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 13 lines · On Thu, 07 Feb 2013 14:22:04 +0530 Suresh Jayaraman…
Quoted 12 lines · Jeff Layton
Quoted 9 lines · On Thu, 31 Jan 2013 15:22:15 +0530 Suresh Jayaraman…
Quoted 8 lines · Tom Talpey
Quoted 8 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Wednesday, January 30, 2013 9:37 AM
To: Tom Talpey
Cc: Suresh Jayaraman; linux-cifs
Subject: Re: Writes greater than 64k fails with -ENOSPC

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

The spec is not 100% clear on whether servers are *required* to support arbitrarily large writes up to the 128k limit. Clearly there are some that do not, and a larger default is problematic against

those servers.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I'd be very interested to see traces of negotiate, large read and large

write from such a server.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

I'm not sure whether I can share the complete trace (without customer's
permission) but I can get you the specific bits that might be interesting. I have asked for a full trace (including negotiate protocol).

Thanks

Quoted 5 lines
Quoted 3 lines

We probably don't need the whole capture. Just the "Capabilities" and "MaxBufferSize" values from the NEGOTIATE would be enough...

The commit ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 did fix the

problem.

Quoted 3 lines

Finally, got the packet capture too. From the NEGOTIATE response, the MaxBufferSize was 65280 and CAP_LARGE_WRITEX (and LARGE_READX)

had been set.

Quoted 30 lines

The complete Capabilities were

.... .... .... .... .... .... .... ...0 = Raw Mode: Read Raw and Write Raw are not supported .... .... .... .... .... .... .... ..0. = MPX
Mode: Read Mpx and Write Mpx are not supported .... .... .... .... .... .... .... .1.. = Unicode: Unicode strings are supported .... .... .... .... .... .... .... 1... = Large Files: Large files are supported .... .... .... .... .... .... ...1 .... = NT SMBs: NT SMBs are supported .... .... .... .... .... .... ..1. .... = RPC Remote APIs:
RPC remote APIs are supported .... .... .... .... .... .... .1.. .... = NT Status Codes: NT status codes are supported .... .... .... .... .... .... 1... .... = Level 2 Oplocks: Level 2 oplocks are supported .... .... .... .... .... ...0 .... .... = Lock and Read: Lock and Read is not supported .... .... .... .... .... ..1. .... .... = NT Find: NT Find is supported .... .... .... .... ...1 .... .... .... = Dfs: Dfs is supported .... .... .... .... ..1. .... .... .... = Infolevel
Passthru: NT information level request passthrough is supported .... .... .... .... .1.. .... .... .... = Large ReadX: Large Read andX is supported .... .... .... .... 1... .... .... .... = Large WriteX:
Large Write andX is supported .... .... .... ...0 .... .... .... .... = LWIO: LWIO ioctl/fsctl is not supported .... .... 0... .... .... .... .... .... = UNIX: UNIX extensions are not supported .... ..0. .... .... .... .... .... .... = Compressed Data: Compressed data transfer is not supported ..0. .... .... .... .... .... .... .... = Dynamic Reauth: Dynamic Reauth is not supported 0... .... .... .... .... .... .... .... = Extended Security: Extended security exchanges are not supported

Quoted 1 line

I'm not sure whether restricting it to MaxBufferSize even in case

CAP_LARGE_WRITEX was set would make sense..

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

No, I think that server just happens to have an unusually large MaxBufferSize. Most servers send values that are smaller than that. The whole idea behind CAP_LARGE_WRITEX is that it allows you to send a write that's larger than the MaxBufferSize.

I agree, but it's still unhelpful that this server accepts only 64KB in response to the client's 124KB large write. In my opinion the server should not be offering CAP_LARGE_WRITEX, although it's certainly legal for it to do so. Perhaps your client should be ignoring it, when MaxBufferSize permits >64KB.

Right. That was the crux of my question a few months ago -- the spec doesn't say how much larger than the MaxBufferSize a server is required to support. In practice, it seems a lot of server vendors didn't do any more than they perceived they had to.

Quoted 4 lines

BTW, we're investigating the large write size behaviors for a related MS-SMB doc bug. I don't have the full picture yet, but it appears that Windows SMB servers will process somewhere between 124KB and 128KB-1 bytes when issued a large WRITE_ANDX in this situation.

Tom.

It's been a while since I wrote this code, but I originally did quite a bit of testing vs. windows versions that were current at the time (win2k8?). At that time, I never saw windows return a short write, but we would generally issue writes that were no larger than 124k or so.

We generally use a max of 2^17-1, minus the length of the WRITE_ANDX header (which is small), and then squeeze as many pages as we can fit within it.

Signature · 1 line
Jeff Layton <[email protected]>
#24 lore raw
In reply to a message not in the archive
Quoted 35 lines · Jeff Layton
Quoted 32 lines · On Tue, 29 Jan 2013 17:54:21 +0530 Suresh Jayaraman…

Hi all,

I'm looking into a report on 3.0 based kernel (plus stable fixes) where writes greater than 64k to a NAS (Hitachi NAS) is failing (simple dd). The problem was not seen with a 2.6.32-ish kernel. Also, note that the problem is not seen with other Servers such as Windows 2003 or Windows 8 Servers.

The strace output shows the close() fall fails with -ENOSPC.

The relevant cFYI snip

Jan 23 08:31:45 vsusix02 kernel: [1003552.227274] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
Jan 23 08:31:45 vsusix02 kernel: [1003552.227277] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 127044
Jan 23 08:31:45 vsusix02 kernel: [1003552.345848] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.393906] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssmb.c: async write at 1015808 8192 bytes
Jan 23 08:31:45 vsusix02 kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
Jan 23 08:31:45 vsusix02 kernel: [1003552.393914] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 8260
Jan 23 08:31:45 vsusix02 kernel: [1003552.479378] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.481215] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.481260] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c: Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28

The problem seems to be that during close(), we try to flush the buffers by calling cifs_flush() which in turn will call filemap_write_and_wait() to wait on the pages under writeback to complete. do_writepages() will invoke cifs_writepages() which is perhaps returning -ENOSPC and it is propagated back.

There are no quota restrictions or disk space problems. The tcpdump output doesn't show any errors during the write or close.

Quoted 8 lines
Quoted 5 lines

I'm not sure what could be causing the problem and would appreciate any clues or debugging suggestions.

Thanks

I'd look for "short" write replies. We have this in cifs_writev_callback:

I think you are spot-on. The packet capture looks little odd:

Client is requesting a WRITE of 126976 bytes [DataLengthHigh -> 1 (multiple by 64k), DataLengthLow -> 61440, WordCount -> 14]

Server seems to have written only 65536 bytes [CountHigh -> 1, CountLow -> 0, Remaining -> 65535] However, [File RW Length] is set to 126976

There are many such writes. I don't have the full capture to verify

Sounds similar to the problem fixed by commit ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 (Some server not handling large wsize correctly?

From: Jeff Layton <[email protected]>
Date: Tue, 17 Jan 2012 16:08:51 -0500
Subject: cifs: lower default wsize when unix extensions are not used

Quoted 6 lines
                if (written < wdata->bytes)
                        wdata->result = -ENOSPC;

IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX responses, but I remember it not being 100% clear...

Did you mean that the spec suggests that in case of -ENOSPC, the SMB/NT status code should still be STATUS_SUCCESS and the POSIX error should be set to -ENOSPC?

Thanks

Signature · 1 line
Suresh Jayaraman

A message that is not in the archive (sent off-list or not yet received): 510A3DB0.1040004-IBi9RG/[email protected]

#26 lore raw
Quoted 57 lines · On Thu, 31 Jan 2013 15:17:28 +0530 Suresh Jayaraman…
Quoted 35 lines · Jeff Layton
Quoted 32 lines · On Tue, 29 Jan 2013 17:54:21 +0530 Suresh Jayaraman…

Hi all,

I'm looking into a report on 3.0 based kernel (plus stable fixes) where writes greater than 64k to a NAS (Hitachi NAS) is failing (simple dd). The problem was not seen with a 2.6.32-ish kernel. Also, note that the problem is not seen with other Servers such as Windows 2003 or Windows 8 Servers.

The strace output shows the close() fall fails with -ENOSPC.

The relevant cFYI snip

Jan 23 08:31:45 vsusix02 kernel: [1003552.227274] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
Jan 23 08:31:45 vsusix02 kernel: [1003552.227277] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 127044
Jan 23 08:31:45 vsusix02 kernel: [1003552.345848] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.393906] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssmb.c: async write at 1015808 8192 bytes
Jan 23 08:31:45 vsusix02 kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
Jan 23 08:31:45 vsusix02 kernel: [1003552.393914] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 8260
Jan 23 08:31:45 vsusix02 kernel: [1003552.479378] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.481215] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.481260] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c: Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28

The problem seems to be that during close(), we try to flush the buffers by calling cifs_flush() which in turn will call filemap_write_and_wait() to wait on the pages under writeback to complete. do_writepages() will invoke cifs_writepages() which is perhaps returning -ENOSPC and it is propagated back.

There are no quota restrictions or disk space problems. The tcpdump output doesn't show any errors during the write or close.

Quoted 8 lines
Quoted 5 lines

I'm not sure what could be causing the problem and would appreciate any clues or debugging suggestions.

Thanks

I'd look for "short" write replies. We have this in cifs_writev_callback:

I think you are spot-on. The packet capture looks little odd:

Client is requesting a WRITE of 126976 bytes [DataLengthHigh -> 1 (multiple by 64k), DataLengthLow -> 61440, WordCount -> 14]

Server seems to have written only 65536 bytes [CountHigh -> 1, CountLow -> 0, Remaining -> 65535] However, [File RW Length] is set to 126976

There are many such writes. I don't have the full capture to verify

Quoted 8 lines

Sounds similar to the problem fixed by commit ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 (Some server not handling large wsize correctly?

From: Jeff Layton <[email protected]>
Date: Tue, 17 Jan 2012 16:08:51 -0500
Subject: cifs: lower default wsize when unix extensions are not used

Yep, though it's only "fixed" in that that patch makes the defaults match what Windows does. You can still make the wsize larger (up to the 128k limit), but at that point we assume that you know what you're doing. :)

Quoted 4 lines
Quoted 4 lines
                if (written < wdata->bytes)
                        wdata->result = -ENOSPC;
Quoted 9 lines
Quoted 3 lines

IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX responses, but I remember it not being 100% clear...

Did you mean that the spec suggests that in case of -ENOSPC, the SMB/NT status code should still be STATUS_SUCCESS and the POSIX error should be set to -ENOSPC?

See section 2.2.4.43.2. The "Error Codes" chart shows 2 entries at the top that seem to describe short writes. I'll note that they *do* indicate that the server ought to return a count of 0 in that situation. It does not indicate how the client should interpret a non-zero, but short write count.

As Tom suggests, the current behavior in that situation is clearly wrong. We ought to just go back and redirty_page_for_writepage those pages that didn't get written out.

If the wsize is larger than the server can handle though, it's likely to make performance suck. Doing writeback may take several passes through writepages since the client will have to go back and fill in the gaps. It would be nice to indicate that to the admin somehow, but I'm not sure how best to do so.

Signature · 1 line
Jeff Layton <[email protected]>