Skip to contentSkip to conversation

slow cifs read on 3.12/3.10 kernel

DISCUSSION Rolf Anderegg 3 participants 4 messages Mar 10, 2014 – Mar 11, 2014 3 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

In the course of evaluating realtime kernels for an Intel Atom based setup, I ran across major read-speed problems on CIFS mounts when using a 3.12 kernel (also tried 3.10, same issue). This resulted in transmissions @ <800KB/s compared to >10MB when using 3.4 kernel's CIFS. At first I thought it had to do with either one of these old buffer size related solutions:

https://bugzilla.samba.org/show_bug.cgi?id=7699
https://bugs.launchpad.net/ubuntu/+source/cifs-utils/+bug/810606
http://ubuntuforums.org/showthread.php?t=1578068

However, fiddling around with rsize/wsize, resp. CIFSMaxBufSize, did not change anything.
Increasing the CIFS debug verbosity and dumping TCP traffic showed that in the 3.12 case, smaller packets are negotiated and transmitted which obviously results in lower throughput. Here are my test logs:

Kernel 3.12.10-rt15 CIFS test log (slow speed):
http://7f42b4439bec450b.paste.se

Kernel 3.4.82-rt100 CIFS test log (normal speed):
http://863be082c9262448.paste.se

I'm all out of handles to crank, so before I dive further into the kernel's CIFS code, I thought I'd call out for some expert help to check if this is a known issue.

Thanks in advance,

Rolf Anderegg

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

There are some quick obvious things to check:
1) since server is Samba - check if unix extensions negotiated and check default rsize
(you can simply do cat /proc/mounts to see what was negotiated)

2) with unix extensions enabled, the maximum read size (and write size) is much larger (which usually should help) so check if differences in rsize or wsize can explain performance differences.

3) Similarly, increasing the maximum number of simultaneous requests that the server can support for each client can have an impact on performance ("max mux = 50" is the default in the server's smb.conf but it can be increased if your workload has many requests from one client at the same time).

4) caching behavior changes - we moved to a much stricter caching policy ("cache=strict") on later kernels so mounting with "cache=loose," and allowing more efficient client side write caching which is usually sufficient for most workloads, may also help.

Quoted 33 lines · Rolf Anderegg

In the course of evaluating realtime kernels for an Intel Atom based setup, I ran across major read-speed problems on CIFS mounts when using a 3.12 kernel (also tried 3.10, same issue). This resulted in transmissions @ <800KB/s compared to >10MB when using 3.4 kernel's CIFS. At first I thought it had to do with either one of these old buffer size related solutions:

https://bugzilla.samba.org/show_bug.cgi?id=7699
https://bugs.launchpad.net/ubuntu/+source/cifs-utils/+bug/810606
http://ubuntuforums.org/showthread.php?t=1578068

However, fiddling around with rsize/wsize, resp. CIFSMaxBufSize, did not change anything.
Increasing the CIFS debug verbosity and dumping TCP traffic showed that in the 3.12 case, smaller packets are negotiated and transmitted which obviously results in lower throughput. Here are my test logs:

Kernel 3.12.10-rt15 CIFS test log (slow speed):
http://7f42b4439bec450b.paste.se

Kernel 3.4.82-rt100 CIFS test log (normal speed):
http://863be082c9262448.paste.se

I'm all out of handles to crank, so before I dive further into the kernel's CIFS code, I thought I'd call out for some expert help to check if this is a known issue.

Thanks in advance,

Rolf Anderegg

--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines
Thanks,

Steve

A message that is not in the archive (sent off-list or not yet received): CAH2r5mvuOhJOamHXT8JXTDmQF2e_ORHaU17HoZ8ScfviHts=UQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#4 lore raw

Am 10.03.2014 20:52, schrieb Steve French:

Quoted 9 lines

There are some quick obvious things to check:
1) since server is Samba - check if unix extensions negotiated and check default rsize
(you can simply do cat /proc/mounts to see what was negotiated)

2) with unix extensions enabled, the maximum read size (and write size) is much larger (which usually should help) so check if differences in rsize or wsize can explain performance differences.

Quoted 5 lines

3) Similarly, increasing the maximum number of simultaneous requests that the server can support for each client can have an impact on performance ("max mux = 50" is the default in the server's smb.conf but it can be increased if your workload has many requests from one client at the same time).

Thanks Steve for your helpful pointers.
The first three points had no effect in my particular case.

Quoted 4 lines

4) caching behavior changes - we moved to a much stricter caching policy ("cache=strict") on later kernels so mounting with "cache=loose," and allowing more efficient client side write caching which is usually sufficient for most workloads, may also help.

The cache mode was the nail that hit it. The cache modes had a dramatic impact on the CIFS mount read speed:

~29 MB/s with "cache=loose"
<800 kB/s with "cache=strict"

Having read the discussion that lead to this change in 3.7
(https://lists.samba.org/archive/samba-technical/2012-April/083228.html), I now understand the reason for the default cache coherency strictness; albeit "a little slower" is quite an understatement in my case.

Thanks,

Rolf

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 27 lines · On Tue, 11 Mar 2014 18:42:28 +0100 Rolf Anderegg…

Am 10.03.2014 20:52, schrieb Steve French:

Quoted 9 lines

There are some quick obvious things to check:
1) since server is Samba - check if unix extensions negotiated and check default rsize
(you can simply do cat /proc/mounts to see what was negotiated)

2) with unix extensions enabled, the maximum read size (and write size) is much larger (which usually should help) so check if differences in rsize or wsize can explain performance differences.

Quoted 5 lines

3) Similarly, increasing the maximum number of simultaneous requests that the server can support for each client can have an impact on performance ("max mux = 50" is the default in the server's smb.conf but it can be increased if your workload has many requests from one client at the same time).

Thanks Steve for your helpful pointers.
The first three points had no effect in my particular case.

Quoted 4 lines

4) caching behavior changes - we moved to a much stricter caching policy ("cache=strict") on later kernels so mounting with "cache=loose," and allowing more efficient client side write caching which is usually sufficient for most workloads, may also help.

The cache mode was the nail that hit it. The cache modes had a dramatic impact on the CIFS mount read speed:

Quoted 8 lines

~29 MB/s with "cache=loose"
<800 kB/s with "cache=strict"

Having read the discussion that lead to this change in 3.7
(https://lists.samba.org/archive/samba-technical/2012-April/083228.html), I now understand the reason for the default cache coherency strictness; albeit "a little slower" is quite an understatement in my case.

If cache=loose helps, then that suggests that you aren't getting oplocks when you open files. That may or may not be expected depending on the usage pattern, but that's probably where you should focus your efforts. Do you have multiple machines or processes opening these files at the same time?

Signature · 1 line
Jeff Layton <[email protected]>