Skip to contentSkip to conversation

virtiofsd: Improve io bandwidth by replacing pwrite with pwritev

RFC PATCH Superseded piaojun 2 participants 3 messages Aug 7, 2019

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

piaojun

author
#0 lore raw

v2 cover virtiofsd: Improve io bandwidth by replacing pwrite with pwritev

Quoted 1 line

From my test, write bandwidth will be improved greatly by replacing

pwrite with pwritev, and the test result as below:

---
pwrite:

# fio -direct=1 -time_based -iodepth=64 -rw=randwrite -ioengine=libaio -bs=1M -size=1G -numjob=16 -runtime=30 -group_reporting -name=file -filename=/mnt/virtiofs/file
file: (g=0): rw=randwrite, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=64
...
fio-2.13
Starting 16 processes
Jobs: 16 (f=16): [w(16)] [100.0% done] [0KB/886.0MB/0KB /s] [0/886/0 iops] [eta 00m:00s]
file: (groupid=0, jobs=16): err= 0: pid=5799: Tue Aug 6 18:48:26 2019
write: io=26881MB, bw=916988KB/s, iops=895, runt= 30018msec

pwritev:

# fio -direct=1 -time_based -iodepth=64 -rw=randwrite -ioengine=libaio -bs=1M -size=1G -numjob=16 -runtime=30 -group_reporting -name=file -filename=/mnt/virtiofs/file
file: (g=0): rw=randwrite, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=64
...
fio-2.13
Starting 16 processes
Jobs: 16 (f=16): [w(16)] [100.0% done] [0KB/1793MB/0KB /s] [0/1793/0 iops] [eta 00m:00s]
file: (groupid=0, jobs=16): err= 0: pid=6328: Tue Aug 6 18:22:17 2019
write: io=52775MB, bw=1758.7MB/s, iops=1758, runt= 30009msec
---

This patch introduces writev and pwritev for lo_write_buf(). I tried my best not doing harm to the origin code construct, and there will be some *useless* branches in fuse_buf_copy_one() which are hard to judge if they will be useful in the future. So I just leave them alone safely. If the cleanup work is necessary, please let me know.

v2
- Split into two patches
- Add the lost flags support, such as FUSE_BUF_PHYS_ADDR

Jun Piao (2):
add definition of fuse_buf_writev().
use fuse_buf_writev to replace fuse_buf_write for better performance

---
 buffer.c         |   37 +++++++++++++++++++++++++++++++++++++
 fuse_common.h    |   14 ++++++++++++++
 passthrough_ll.c |    5 ++++-
 seccomp.c        |    2 +

--

#1 lore raw

On Wed, Aug 07, 2019 at 02:57:05PM +0800, piaojun wrote:

Hi,
Please use git send-email --thread so that each patch is part of the same email thread. This makes it easier to locate the entire patch series in an email client and to apply the patches together.

Quoted 15 lines
Quoted 1 line

From my test, write bandwidth will be improved greatly by replacing

pwrite with pwritev, and the test result as below:

---
pwrite:

# fio -direct=1 -time_based -iodepth=64 -rw=randwrite -ioengine=libaio -bs=1M -size=1G -numjob=16 -runtime=30 -group_reporting -name=file -filename=/mnt/virtiofs/file
file: (g=0): rw=randwrite, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=64
...
fio-2.13
Starting 16 processes
Jobs: 16 (f=16): [w(16)] [100.0% done] [0KB/886.0MB/0KB /s] [0/886/0 iops] [eta 00m:00s]
file: (groupid=0, jobs=16): err= 0: pid=5799: Tue Aug 6 18:48:26 2019
write: io=26881MB, bw=916988KB/s, iops=895, runt= 30018msec

pwritev:

Quoted 8 lines
# fio -direct=1 -time_based -iodepth=64 -rw=randwrite -ioengine=libaio -bs=1M -size=1G -numjob=16 -runtime=30 -group_reporting -name=file -filename=/mnt/virtiofs/file
file: (g=0): rw=randwrite, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=64
...
fio-2.13
Starting 16 processes
Jobs: 16 (f=16): [w(16)] [100.0% done] [0KB/1793MB/0KB /s] [0/1793/0 iops] [eta 00m:00s]
file: (groupid=0, jobs=16): err= 0: pid=6328: Tue Aug 6 18:22:17 2019
write: io=52775MB, bw=1758.7MB/s, iops=1758, runt= 30009msec

Nice performance improvement!

author
#2 lore raw

Hi Stefan,

Quoted 6 lines · Stefan Hajnoczi

On Wed, Aug 07, 2019 at 02:57:05PM +0800, piaojun wrote:

Hi,
Please use git send-email --thread so that each patch is part of the same email thread. This makes it easier to locate the entire patch series in an email client and to apply the patches together.

Good suggestion, I will try using 'git send-email'.

Thanks,
Jun

Quoted 28 lines
Quoted 16 lines
Quoted 2 lines

From my test, write bandwidth will be improved greatly by replacing

pwrite with pwritev, and the test result as below:

---
pwrite:

# fio -direct=1 -time_based -iodepth=64 -rw=randwrite -ioengine=libaio -bs=1M -size=1G -numjob=16 -runtime=30 -group_reporting -name=file -filename=/mnt/virtiofs/file
file: (g=0): rw=randwrite, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=64
...
fio-2.13
Starting 16 processes
Jobs: 16 (f=16): [w(16)] [100.0% done] [0KB/886.0MB/0KB /s] [0/886/0 iops] [eta 00m:00s]
file: (groupid=0, jobs=16): err= 0: pid=5799: Tue Aug 6 18:48:26 2019
write: io=26881MB, bw=916988KB/s, iops=895, runt= 30018msec

pwritev:

Quoted 8 lines
# fio -direct=1 -time_based -iodepth=64 -rw=randwrite -ioengine=libaio -bs=1M -size=1G -numjob=16 -runtime=30 -group_reporting -name=file -filename=/mnt/virtiofs/file
file: (g=0): rw=randwrite, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=64
...
fio-2.13
Starting 16 processes
Jobs: 16 (f=16): [w(16)] [100.0% done] [0KB/1793MB/0KB /s] [0/1793/0 iops] [eta 00m:00s]
file: (groupid=0, jobs=16): err= 0: pid=6328: Tue Aug 6 18:22:17 2019
write: io=52775MB, bw=1758.7MB/s, iops=1758, runt= 30009msec

Nice performance improvement!
.