Skip to contentSkip to conversation

blog post on fio write zeroes performance

DISCUSSION Vincent Fu 4 participants 6 messages Nov 5, 2024

lore
Share & tools
lore
#1 lore raw

Nice writeup Vincent. It's been a while since I've looked at the NVMe spec so perhaps I'm asking the wrong question but why is fio managing a data buffer for its handling of the Write Zero command? I would've assumed there's no data payload associated with that NVMe command, thus nothing that should even be subject to internal buffer scrambling inside fio.

Regards,

Adam

________________________________________
From: Vincent Fu <[email protected]>
Sent: Tuesday, November 5, 2024 2:41 PM
To: [email protected]
Cc: [email protected]
Subject: blog post on fio write zeroes performance

I am pleased to share a blog post about fio write zeroes performance:

https://github.com/vincentkfu/fio-blog/wiki/How-could-write-zeroes-performance-be-worse-at-higher-queue-depths

Read about the unexpected performance issue we encountered. Feedback is welcome.

Vincent

author
#2 lore raw
Quoted 14 lines · Adam Horshack

Nice writeup Vincent. It's been a while since I've looked at the NVMe spec so perhaps I'm asking the wrong question but why is fio managing a data buffer for its handling of the Write Zero command? I would've assumed there's no data payload associated with that NVMe command, thus nothing that should even be subject to internal buffer scrambling inside fio.

Regards,

Adam

________________________________________
From: Vincent Fu <[email protected]>
Sent: Tuesday, November 5, 2024 2:41 PM
To: [email protected]
Cc: [email protected]
Subject: blog post on fio write zeroes performance

Quoted 10 lines

I am pleased to share a blog post about fio write zeroes performance:

https://github.com/vincentkfu/fio-blog/wiki/How-could-write-zeroes-performance-be-worse-at-higher-queue-depths

Read about the unexpected performance issue we encountered. Feedback is welcome.

Vincent

When we added write zeroes support for the io_uring_cmd ioengine it was not on our mind that the default behavior for fio is to scramble buffers. Basically these patches just changed the opcode from a regular write to write zeroes.

You do have a good point that perhaps fio should be smart enough to avoid touching the buffers when doing write zeroes.

Thanks for the feedback!

Vincent

#3 lore raw

Internal Use - Confidential

Quoted 10 lines

From: Vincent Fu <[email protected]>
Sent: Tuesday, November 5, 2024 3:37 PM
To: Adam Horshack <[email protected]>; [email protected]
Subject: Re: blog post on fio write zeroes performance

When we added write zeroes support for the io_uring_cmd ioengine it was not on our mind that the default behavior for fio is to scramble buffers. Basically these patches just changed the opcode from a regular write to write zeroes.

Quoted 6 lines

You do have a good point that perhaps fio should be smart enough to avoid touching the buffers when doing write zeroes.

Thanks for the feedback!

Vincent

Does it mean that in the regular case (not write zeroes) fio spends significant time scrambling buffers and IO performance is lower than it could be?

Regards,
Mikhail

author
#4 lore raw
Quoted 15 lines · Mikhail

Internal Use - Confidential

Quoted 13 lines

From: Vincent Fu <[email protected]>
Sent: Tuesday, November 5, 2024 3:37 PM
To: Adam Horshack <[email protected]>; [email protected]
Subject: Re: blog post on fio write zeroes performance

When we added write zeroes support for the io_uring_cmd ioengine it was not on our mind that the default behavior for fio is to scramble buffers. Basically these patches just changed the opcode from a regular write to write zeroes.

You do have a good point that perhaps fio should be smart enough to avoid touching the buffers when doing write zeroes.

Quoted 8 lines
Quoted 3 lines

Thanks for the feedback!

Vincent

Does it mean that in the regular case (not write zeroes) fio spends significant time scrambling buffers and IO performance is lower than it could be?

Regards,
Mikhail

For regular writes on contemporary devices the time that fio spends scrambling buffers is small compared to the device response time.

However, if you want the absolute highest write performance or are really pushing the limits with a device, it might be worth running the job with zero_buffers or scramble_buffers=0 to see what difference it makes.

Vincent

#5 lore raw
Quoted 8 lines · Vincent Fu
Quoted 5 lines · Mikhail

Does it mean that in the regular case (not write zeroes) fio spends significant time scrambling buffers and IO performance is lower than it could be?

Regards,
Mikhail

Quoted 6 lines

For regular writes on contemporary devices the time that fio spends scrambling buffers is small compared to the device response time.

However, if you want the absolute highest write performance or are really pushing the limits with a device, it might be worth running the job with zero_buffers or scramble_buffers=0 to see what difference it makes.

It's worth noting that even in the "not write zeros dedicated command" case some devices (or things in the I/O path like filesystems etc.) can detect or compress buffers of zeros and thus are able to achieve (unrealistic) higher speeds that they cannot reach when "real" data is used...

--
Sitsofe