cifs: improve read performance for page size 64KB & cache=strict & vers=2.1+
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): CAEUGjKiLPQP9wp0AgLUvHgKBOe9We2a-RQaZ7cd7CvhnarwWiw@mail.gmail.com
пт, 10 апр. 2020 г. в 09:50, Jones Syue via samba-technical <[email protected]>:
Quoted 64 lines
Hello list,
please help review whether the attached patch is correct, thank you.
Found a read performance issue when linux kernel page size is 64KB. If linux kernel page size is 64KB and mount options cache=strict & vers=2.1+,
it does not supportcifs_readpages(). Instead, it is usingcifs_readpage()andcifs_read()with maximum read IO size 16KB, which is much slower than read IO
size 1MB when negotiated SMB 2.1+. Since modern SMB server supported SMB 2.1+
and Max Read Size can reach more than 64KB (for example 1MB ~ 8MB), this patch
do one more check onmax_readto determine whether server supportreadpages()
and improve read performance for page size 64KB & cache=strict & vers=2.1+.The client is a linux box with linux kernel 4.2.8, page size 64KB (
CONFIG_ARM64_64K_PAGES=y),
cpu arm 1.7GHz, and use mount.cifs as smb client. The server is another linux box with linux kernel 4.2.8, share a file '10G.img' with size 10GB,
and use samba-4.7.12 as smb server.The client mount a share from the server with different cache options: cache=strict and cache=none, mount -tcifs //<server_ip>/Public /cache_strict -overs=3.0,cache=strict,username=<xxx>,password=<yyy> mount -tcifs //<server_ip>/Public /cache_none -overs=3.0,cache=none,username=<xxx>,password=<yyy>
The client download a 10GbE file from the server accross 1GbE network, dd if=/cache_strict/10G.img of=/dev/null bs=1M count=10240 dd if=/cache_none/10G.img of=/dev/null bs=1M count=10240
Found that cache=strict (without patch) is slower read throughput and smaller
read IO size than cache=none.
cache=strict (without patch): read throughput 40MB/s, read IO size is 16KB cache=strict (with patch): read throughput 113MB/s, read IO size is 1MB cache=none: read throughput 109MB/s, read IO size is 1MBLooks like if page size is 64KB,
cifs_set_ops()would usecifs_addr_ops_smallbufinstead ofcifs_addr_ops,/* check if server can support readpages */
if (cifs_sb_master_tcon(cifs_sb)->ses->server->maxBuf <PAGE_SIZE+MAX_CIFS_HDR_SIZE)inode->i_data.a_ops=&cifs_addr_ops_smallbuf; elseinode->i_data.a_ops=&cifs_addr_ops;maxBuf is came from 2 places,
SMB2_negotiate()andCIFSSMBNegotiate(), (SMB2_MAX_BUFFER_SIZEis 64KB)SMB2_negotiate():
/* set it to the maximum buffer size value we can send with 1 credit */server->maxBuf= min_t(unsigned int,le32_to_cpu(rsp->MaxTransactSize),SMB2_MAX_BUFFER_SIZE);
CIFSSMBNegotiate():server->maxBuf=le32_to_cpu(pSMBr->MaxBufferSize);
Quoted 9 lines
Page size 64KB and cache=strict lead to
read_pages()usecifs_readpage()instead
ofcifs_readpages(), and thencifs_read()using maximum read IO size 16KB, which is much slower than maximum read IO size 1MB. (CIFSMaxBufSize is 16KB by default)/* FIXME: set up handlers for larger reads and/or convert to async */ rsize =
min_t(unsigned int, cifs_sb->rsize, CIFSMaxBufSize);
Hi Jones,
Thanks for the patch!
It will work although it is probably a little bit cleaner to initialize server->max_read to server->maxBuf for SMB1 and use the server->max_read in the readpages condition check instead.
@Others, thoughts?
--
Best regards,
Pavel Shilovsky
A message that is not in the archive (sent off-list or not yet received): CAEUGjKhSBNQboKOMFMgos9OQfxcLQZsXp8aBrUSFcaSe1saH2Q@mail.gmail.com
Pavel Shilovsky
Reviewed-byLooks good, thanks!
--
Best regards,
Pavel Shilovsky
вс, 12 апр. 2020 г. в 19:23, Jones Syue <[email protected]>:
Quoted 27 lines
Hello Pavel
Thanks for kindly reviewing!
Please find the attached v2.patch.--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.Quoted 14 lines · Pavel Shilovsky
Hi Jones,
Thanks for the patch!
It will work although it is probably a little bit cleaner to initialize
server->max_readtoserver->maxBuffor SMB1 and use theserver->max_readin the readpages condition check instead.@Others, thoughts?
--
Best regards,
Pavel Shilovsky
merged into cifs-2.6.git for-next pending testing
Quoted 38 lines · Pavel Shilovsky via samba-technical
Looks good, thanks!
Reviewed-by Pavel Shilovsky <[email protected]>--
Best regards,
Pavel Shilovskyвс, 12 апр. 2020 г. в 19:23, Jones Syue <[email protected]>:
Quoted 27 lines
Hello Pavel
Thanks for kindly reviewing!
Please find the attached v2.patch.--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.Quoted 14 lines · Pavel Shilovsky
Hi Jones,
Thanks for the patch!
It will work although it is probably a little bit cleaner to initialize
server->max_readtoserver->maxBuffor SMB1 and use theserver->max_readin the readpages condition check instead.@Others, thoughts?
--
Best regards,
Pavel Shilovsky
Signature · 3 lines
Thanks, Steve
Did you also test (at least briefly) with vers=1.0 since some of your code affects that code path too?
And if anyone figures out how to configure an x86_64 Linux to use PAGE_SIZE of 64K or larger let me know...
Quoted 27 lines · Jones Syue via samba-technical
Hello Pavel
Thanks for kindly reviewing!
Please find the attached v2.patch.--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.Quoted 14 lines · Pavel Shilovsky
Hi Jones,
Thanks for the patch!
It will work although it is probably a little bit cleaner to initialize
server->max_readtoserver->maxBuffor SMB1 and use theserver->max_readin the readpages condition check instead.@Others, thoughts?
--
Best regards,
Pavel Shilovsky
Signature · 3 lines
Thanks, Steve
A message that is not in the archive (sent off-list or not yet received): CAEUGjKhpgmhj9RzcGQXPuFUyoqsUnk2d3oCpOYBdR=EwCO21YQ@mail.gmail.com
Hello Steve
Quoted 1 line
Test read performance over 1GbE network with command:
Also test read performance over 10GbE network,
vers=2.1+ can reach over 600 MB/s with v2.patch.
aarch64, page size 64KB (CONFIG_ARM64_64K_PAGES=y), linux-4.2.8, cpu Annapurna Labs Alpine AL324 Quad-core ARM Cortex-A57 CPU @ 1.70GHz, ram 8GB,
with patch,
vers=1.0,cache=strict: read throughput 110MB/s, max read IO size 16KB vers=2.0,cache=strict: read throughput 106MB/s, max read IO size 16KB vers=2.1,cache=strict: read throughput 667MB/s, max read IO size 1MB vers=3.0,cache=strict: read throughput 639MB/s, max read IO size 1MB without patch,
vers=1.0,cache=strict: read throughput 107MB/s, max read IO size 16KB vers=2.0,cache=strict: read throughput 107MB/s, max read IO size 16KB vers=2.1,cache=strict: read throughput 106MB/s, max read IO size 16KB vers=3.0,cache=strict: read throughput 106MB/s, max read IO size 16KB
command:
mount -tcifs //<server_ip>/<share> /remote_strict -overs=<x.y>,cache=strict,username=<uu>,password=<pp> dd if=/remote_strict/10G.img of=/dev/null bs=1M count=10240
--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.
Quoted 56 lines · Jones Syue
Hello Steve
Quoted 2 lines
Did you also test (at least briefly) with vers=1.0 since some of your code affects that code path too?
Yes test v2.patch on 2 platforms aarch64 (page size 64KB) and
x86_64(page size 4KB), vers=1.0 read function works fine on both.Test read performance over 1GbE network with command:
'dd if=/remote_strict/10G.img of=/dev/null bs=1M count=10240'For read performance on aarch64 (page size 64KB), vers=[1.0|2.0] is not as fast as vers=2.1+,
max_readon both SMB 1 (16KB) and SMB 2.0 (64KB) are still smaller then page size 64KB plus packet header size, hence do not support readpages.
aarch64, page size 64KB (CONFIG_ARM64_64K_PAGES=y), linux-4.2.8, cpu Annapurna Labs Alpine AL324 Quad-core ARM Cortex-A57 CPU @ 1.70GHz, ram 8GB,
with patch,
vers=1.0,cache=strict: read throughput 40MB/s, max read IO size 16KB vers=2.0,cache=strict: read throughput 40MB/s, max read IO size 16KB vers=2.1,cache=strict: read throughput 115MB/s, max read IO size 1MB vers=3.0,cache=strict: read throughput 115MB/s, max read IO size 1MB without patch,
vers=1.0,cache=strict: read throughput 40MB/s, max read IO size 16KB vers=2.0,cache=strict: read throughput 40MB/s, max read IO size 16KB vers=2.1,cache=strict: read throughput 40MB/s, max read IO size 16KB vers=3.0,cache=strict: read throughput 40MB/s, max read IO size 16KBFor read performance on
x86_64(page size 4KB), all vers can support readpages becausemax_readis bigger than page size 4KB plus packet header size.x86_64, page size 4KB, linux-4.2.8,
cpu AMD Embedded R-Series RX-421ND 2.10GHz,
ram 4GB,
without patch,
vers=1.0,cache=strict: read throughput 109MB/s, read IO size 60KB vers=2.0,cache=strict: read throughput 115MB/s, read IO size 64KB vers=2.1,cache=strict: read throughput 117MB/s, read IO size 1MB vers=3.0,cache=strict: read throughput 117MB/s, read IO size 1MB with patch,
vers=1.0,cache=strict: read throughput 110MB/s, read IO size 60KB vers=2.0,cache=strict: read throughput 115MB/s, read IO size 64KB vers=2.1,cache=strict: read throughput 117MB/s, read IO size 1MB vers=3.0,cache=strict: read throughput 117MB/s, read IO size 1MBQuoted 2 lines
And if anyone figures out how to configure an
x86_64Linux to usePAGE_SIZEof 64K or larger let me know...I am using physical platform with arm cpu and aarch64 toolchain, perhaps try qemu-system-aarch64 later.
--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.
Quoted 4 lines
Quoted 2 lines
And if anyone figures out how to configure an
x86_64Linux to usePAGE_SIZEof 64K or larger let me know...I am using physical platform with arm cpu and aarch64 toolchain, perhaps try qemu-system-aarch64 later.
For reference using qemu-system-aarch64 + linux-5.6.4 + 64KB page to test cifs read, this patch can improve cifs read performance:
with patch: read throughput 39 MB/s, SMB read IO size 4MB
/ # dd if=/mnt/cifs/1G.img of=/dev/null bs=4M count=256
256+0 records in
256+0 records out
1073741824 bytes (1.0GB) copied, 25.982352 seconds, 39.4MB/s [~] # strace -p 23934sendfile(38, 32, [297795584] => [301989888], 4194304) = 4194304
without patch: read throughput 18 MB/s, SMB read IO size 16KB / # dd if=/mnt/cifs/1G.img of=/dev/null bs=4M count=256G 256+0 records in
256+0 records out
1073741824 bytes (1.0GB) copied, 54.367686 seconds, 18.8MB/s [~] <0> strace -p 15786sendfile(38, 32, [452984832] => [453001216], 16384) = 16384
This link is a easy way to compile aarch64 linux kernel with page size 64KB , a simple rootfs with busybox, and run it on qemu-system-aarch64:
https://docs.google.com/document/d/1NSVd-dib_asugCZHmZgohLZXHxV25ftzYtUDSppY3hA/edit?usp=sharing
--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.