Skip to contentSkip to conversation

cifs: improve read performance for page size 64KB & cache=strict & vers=2.1+

PATCH Pavel Shilovsky 3 participants 6 messages Apr 10, 2020 – Apr 27, 2020 3 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): CAEUGjKiLPQP9wp0AgLUvHgKBOe9We2a-RQaZ7cd7CvhnarwWiw@mail.gmail.com

#1 lore raw

пт, 10 апр. 2020 г. в 09:50, Jones Syue via samba-technical <[email protected]>:

Quoted 64 lines

Hello list,

please help review whether the attached patch is correct, thank you.

Found a read performance issue when linux kernel page size is 64KB. If linux kernel page size is 64KB and mount options cache=strict & vers=2.1+,
it does not support cifs_readpages(). Instead, it is using cifs_readpage() and
cifs_read() with maximum read IO size 16KB, which is much slower than read IO
size 1MB when negotiated SMB 2.1+. Since modern SMB server supported SMB 2.1+
and Max Read Size can reach more than 64KB (for example 1MB ~ 8MB), this patch
do one more check on max_read to determine whether server support readpages()
and improve read performance for page size 64KB & cache=strict & vers=2.1+.

The client is a linux box with linux kernel 4.2.8, page size 64KB (CONFIG_ARM64_64K_PAGES=y),
cpu arm 1.7GHz, and use mount.cifs as smb client. The server is another linux box with linux kernel 4.2.8, share a file '10G.img' with size 10GB,
and use samba-4.7.12 as smb server.

The client mount a share from the server with different cache options: cache=strict and cache=none, mount -tcifs //<server_ip>/Public /cache_strict -overs=3.0,cache=strict,username=<xxx>,password=<yyy> mount -tcifs //<server_ip>/Public /cache_none -overs=3.0,cache=none,username=<xxx>,password=<yyy>

The client download a 10GbE file from the server accross 1GbE network, dd if=/cache_strict/10G.img of=/dev/null bs=1M count=10240 dd if=/cache_none/10G.img of=/dev/null bs=1M count=10240

Found that cache=strict (without patch) is slower read throughput and smaller
read IO size than cache=none.
cache=strict (without patch): read throughput 40MB/s, read IO size is 16KB cache=strict (with patch): read throughput 113MB/s, read IO size is 1MB cache=none: read throughput 109MB/s, read IO size is 1MB

Looks like if page size is 64KB,
cifs_set_ops() would use cifs_addr_ops_smallbuf instead of cifs_addr_ops,

/* check if server can support readpages */
if (cifs_sb_master_tcon(cifs_sb)->ses->server->maxBuf < PAGE_SIZE + MAX_CIFS_HDR_SIZE) inode->i_data.a_ops = &cifs_addr_ops_smallbuf; else
inode->i_data.a_ops = &cifs_addr_ops;

maxBuf is came from 2 places, SMB2_negotiate() and CIFSSMBNegotiate(), (SMB2_MAX_BUFFER_SIZE is 64KB)
SMB2_negotiate():
/* set it to the maximum buffer size value we can send with 1 credit */ server->maxBuf = min_t(unsigned int, le32_to_cpu(rsp->MaxTransactSize),

      SMB2_MAX_BUFFER_SIZE);

CIFSSMBNegotiate():
server->maxBuf = le32_to_cpu(pSMBr->MaxBufferSize);

Quoted 9 lines

Page size 64KB and cache=strict lead to read_pages() use cifs_readpage() instead
of cifs_readpages(), and then cifs_read() using maximum read IO size 16KB, which is much slower than maximum read IO size 1MB. (CIFSMaxBufSize is 16KB by default)

/* FIXME: set up handlers for larger reads and/or convert to async */ rsize = min_t(unsigned int, cifs_sb->rsize, CIFSMaxBufSize);

Hi Jones,

Thanks for the patch!

It will work although it is probably a little bit cleaner to initialize server->max_read to server->maxBuf for SMB1 and use the server->max_read in the readpages condition check instead.

@Others, thoughts?

--
Best regards,
Pavel Shilovsky

A message that is not in the archive (sent off-list or not yet received): CAEUGjKhSBNQboKOMFMgos9OQfxcLQZsXp8aBrUSFcaSe1saH2Q@mail.gmail.com

author Reviewed-by
#3 lore raw

Looks good, thanks!

--
Best regards,
Pavel Shilovsky

вс, 12 апр. 2020 г. в 19:23, Jones Syue <[email protected]>:

Quoted 27 lines

Hello Pavel

Thanks for kindly reviewing!
Please find the attached v2.patch.

--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.

Quoted 14 lines · Pavel Shilovsky

Hi Jones,

Thanks for the patch!

It will work although it is probably a little bit cleaner to initialize server->max_read to server->maxBuf for SMB1 and use the server->max_read in the readpages condition check instead.

@Others, thoughts?

--
Best regards,
Pavel Shilovsky

#4 lore raw

merged into cifs-2.6.git for-next pending testing

Quoted 38 lines · Pavel Shilovsky via samba-technical

Looks good, thanks!

--
Best regards,
Pavel Shilovsky

вс, 12 апр. 2020 г. в 19:23, Jones Syue <[email protected]>:

Quoted 27 lines

Hello Pavel

Thanks for kindly reviewing!
Please find the attached v2.patch.

--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.

Quoted 14 lines · Pavel Shilovsky

Hi Jones,

Thanks for the patch!

It will work although it is probably a little bit cleaner to initialize server->max_read to server->maxBuf for SMB1 and use the server->max_read in the readpages condition check instead.

@Others, thoughts?

--
Best regards,
Pavel Shilovsky

Signature · 3 lines
Thanks,

Steve
#5 lore raw
In reply to a message not in the archive

Did you also test (at least briefly) with vers=1.0 since some of your code affects that code path too?

And if anyone figures out how to configure an x86_64 Linux to use PAGE_SIZE of 64K or larger let me know...

Quoted 27 lines · Jones Syue via samba-technical

Hello Pavel

Thanks for kindly reviewing!
Please find the attached v2.patch.

--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.

Quoted 14 lines · Pavel Shilovsky

Hi Jones,

Thanks for the patch!

It will work although it is probably a little bit cleaner to initialize server->max_read to server->maxBuf for SMB1 and use the server->max_read in the readpages condition check instead.

@Others, thoughts?

--
Best regards,
Pavel Shilovsky

Signature · 3 lines
Thanks,

Steve

A message that is not in the archive (sent off-list or not yet received): CAEUGjKhpgmhj9RzcGQXPuFUyoqsUnk2d3oCpOYBdR=EwCO21YQ@mail.gmail.com

#7 lore raw

Hello Steve

Quoted 1 line

Test read performance over 1GbE network with command:

Also test read performance over 10GbE network,
vers=2.1+ can reach over 600 MB/s with v2.patch.

aarch64, page size 64KB (CONFIG_ARM64_64K_PAGES=y), linux-4.2.8, cpu Annapurna Labs Alpine AL324 Quad-core ARM Cortex-A57 CPU @ 1.70GHz, ram 8GB,
with patch,
vers=1.0,cache=strict: read throughput 110MB/s, max read IO size 16KB vers=2.0,cache=strict: read throughput 106MB/s, max read IO size 16KB vers=2.1,cache=strict: read throughput 667MB/s, max read IO size 1MB vers=3.0,cache=strict: read throughput 639MB/s, max read IO size 1MB without patch,
vers=1.0,cache=strict: read throughput 107MB/s, max read IO size 16KB vers=2.0,cache=strict: read throughput 107MB/s, max read IO size 16KB vers=2.1,cache=strict: read throughput 106MB/s, max read IO size 16KB vers=3.0,cache=strict: read throughput 106MB/s, max read IO size 16KB

command:
mount -tcifs //<server_ip>/<share> /remote_strict -overs=<x.y>,cache=strict,username=<uu>,password=<pp> dd if=/remote_strict/10G.img of=/dev/null bs=1M count=10240

--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.

Quoted 56 lines · Jones Syue

Hello Steve

Quoted 2 lines

Did you also test (at least briefly) with vers=1.0 since some of your code affects that code path too?

Yes test v2.patch on 2 platforms aarch64 (page size 64KB) and x86_64 (page size 4KB), vers=1.0 read function works fine on both.

Test read performance over 1GbE network with command:
'dd if=/remote_strict/10G.img of=/dev/null bs=1M count=10240'

For read performance on aarch64 (page size 64KB), vers=[1.0|2.0] is not as fast as vers=2.1+, max_read on both SMB 1 (16KB) and SMB 2.0 (64KB) are still smaller then page size 64KB plus packet header size, hence do not support readpages.
aarch64, page size 64KB (CONFIG_ARM64_64K_PAGES=y), linux-4.2.8, cpu Annapurna Labs Alpine AL324 Quad-core ARM Cortex-A57 CPU @ 1.70GHz, ram 8GB,
with patch,
vers=1.0,cache=strict: read throughput 40MB/s, max read IO size 16KB vers=2.0,cache=strict: read throughput 40MB/s, max read IO size 16KB vers=2.1,cache=strict: read throughput 115MB/s, max read IO size 1MB vers=3.0,cache=strict: read throughput 115MB/s, max read IO size 1MB without patch,
vers=1.0,cache=strict: read throughput 40MB/s, max read IO size 16KB vers=2.0,cache=strict: read throughput 40MB/s, max read IO size 16KB vers=2.1,cache=strict: read throughput 40MB/s, max read IO size 16KB vers=3.0,cache=strict: read throughput 40MB/s, max read IO size 16KB

For read performance on x86_64 (page size 4KB), all vers can support readpages because max_read is bigger than page size 4KB plus packet header size.
x86_64, page size 4KB, linux-4.2.8,
cpu AMD Embedded R-Series RX-421ND 2.10GHz,
ram 4GB,
without patch,
vers=1.0,cache=strict: read throughput 109MB/s, read IO size 60KB vers=2.0,cache=strict: read throughput 115MB/s, read IO size 64KB vers=2.1,cache=strict: read throughput 117MB/s, read IO size 1MB vers=3.0,cache=strict: read throughput 117MB/s, read IO size 1MB with patch,
vers=1.0,cache=strict: read throughput 110MB/s, read IO size 60KB vers=2.0,cache=strict: read throughput 115MB/s, read IO size 64KB vers=2.1,cache=strict: read throughput 117MB/s, read IO size 1MB vers=3.0,cache=strict: read throughput 117MB/s, read IO size 1MB

Quoted 2 lines

And if anyone figures out how to configure an x86_64 Linux to use PAGE_SIZE of 64K or larger let me know...

I am using physical platform with arm cpu and aarch64 toolchain, perhaps try qemu-system-aarch64 later.

--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.

#8 lore raw
In reply to a message not in the archive
Quoted 4 lines
Quoted 2 lines

And if anyone figures out how to configure an x86_64 Linux to use PAGE_SIZE of 64K or larger let me know...

I am using physical platform with arm cpu and aarch64 toolchain, perhaps try qemu-system-aarch64 later.

For reference using qemu-system-aarch64 + linux-5.6.4 + 64KB page to test cifs read, this patch can improve cifs read performance:
with patch: read throughput 39 MB/s, SMB read IO size 4MB
/ # dd if=/mnt/cifs/1G.img of=/dev/null bs=4M count=256
256+0 records in
256+0 records out
1073741824 bytes (1.0GB) copied, 25.982352 seconds, 39.4MB/s [~] # strace -p 23934
sendfile(38, 32, [297795584] => [301989888], 4194304) = 4194304

without patch: read throughput 18 MB/s, SMB read IO size 16KB / # dd if=/mnt/cifs/1G.img of=/dev/null bs=4M count=256G 256+0 records in
256+0 records out
1073741824 bytes (1.0GB) copied, 54.367686 seconds, 18.8MB/s [~] <0> strace -p 15786
sendfile(38, 32, [452984832] => [453001216], 16384) = 16384

This link is a easy way to compile aarch64 linux kernel with page size 64KB , a simple rootfs with busybox, and run it on qemu-system-aarch64:
https://docs.google.com/document/d/1NSVd-dib_asugCZHmZgohLZXHxV25ftzYtUDSppY3hA/edit?usp=sharing

--
Regards,
Jones Syue | 薛懷宗
QNAP Systems, Inc.