Skip to contentSkip to conversation

testing results of support async buffered reads feature

Question Hao_Xu 2 participants 2 messages Oct 10, 2020 io_uring

lore
Share & tools
lore

Hao_Xu

author
#0 lore raw

Hi Jens,
I've done some testing for io_uring async buffered reads with fio. But I found something strange to me.
- when readahead is exactly turned off, the async buffered reads feature appears to be worse than the io-wq method in terms of IOPS.
- when readahead is on, async buffered reads works better but the optimization rate seems to be related with the size of readahead. I'm wondering why.

my environment is:

	server: physical server
	kernel: mainline 5.9.0-rc8+ latest commit 6f2f486d57c4d562cdf4
	fs: ext4
	device: nvme
	fio: 3.20

I did the tests by setting and commenting the code:

	filp->f_mode |= FMODE_BUF_RASYNC;

in fs/ext4/file.c ext4_file_open()

the IOPS in different condition is below:

when blockdev setra 0 /mnt/nvme0n1
QD/Test FMODE_BUF_RASYNC set FMODE_BUF_RASYNC not set
1 12.9k 11.0k
2 32.4k 29.7k
4 65.8k 62.1k
8 123k 116k
16 211k 208k
32 235k 296k
64 241k 328k
128 229k 312k

the async buffered reads feature has a smaller IOPS.

when blockdev setra 64 /mnt/nvme0n1
QD/Test FMODE_BUF_RASYNC set FMODE_BUF_RASYNC not set
1 11.4k 12.2k
2 23.8k 30.0k
4 52.7k 61.7k
8 122k 114k
16 208k 181k
32 237k 199k
64 260k 185k
128 231k 201k

for QD=64 (260-185)/185 = 40.5%

when blockdev setra 128 /mnt/nvme0n1
QD/Test FMODE_BUF_RASYNC set FMODE_BUF_RASYNC not set
1 11.4k 10.8k
2 23.9k 22.7k
4 53.1k 46.5k
8 122k 106k
16 204k 182k
32 212k 200k
64 242k 202k
128 229k 188k

for QD=64 (242-202)/202 = 20.0%

when blockdev setra 256 /mnt/nvme0n1
QD/Test FMODE_BUF_RASYNC set FMODE_BUF_RASYNC not set
1 11.5k 12.2k
2 23.8k 29.7k
4 52.9k 61.9k
8 121k 117k
16 207k 186k
32 229k 204k
64 230k 211k
128 240k 203k

for QD=64 (230-211)/211 = 9.0%

the arguments of fio I use are:
fio_test.sh:
blockdev --setra $2 /dev/nvme0n1

fio -filename=/mnt/nvme0n1/fio_read_test.txt \
    -buffered=1 \
    -iodepth $1 \
    -rw=randread \
    -ioengine=io_uring \
    -randseed=89 \
    -runtime=10s \
    -norandommap \
    -direct=0 \
    -bs=4k \
    -size=4G \
    -name=rand_read_4k
maintainer
#1 lore raw
Quoted 8 lines · Hao_Xu

Hi Jens,
I've done some testing for io_uring async buffered reads with fio. But I found something strange to me.
- when readahead is exactly turned off, the async buffered reads feature appears to be worse than the io-wq method in terms of IOPS.
- when readahead is on, async buffered reads works better but the optimization rate seems to be related with the size of readahead. I'm wondering why.

I don't think these are necessarily unexpected. By and large, the async buffered reads are faster, have lower latencies, and are a lot more efficient in terms of CPU usage. But there are cases where the old thread offload will be quicker, as you're essentially spreading the copying over more cores and can get higher bandwidth that way.

If you're utilizing a single ring for your application, then there might be gains to be had at the higher end of the IOPS or bandwidth spectrum by selectively using IOSQE_ASYNC for a (small) subset of the issued reads.

Signature · 1 line
Jens Axboe