blog post on fio write zeroes performance
I am pleased to share a blog post about fio write zeroes performance:
Read about the unexpected performance issue we encountered. Feedback is welcome.
Vincent
Nice writeup Vincent. It's been a while since I've looked at the NVMe spec so perhaps I'm asking the wrong question but why is fio managing a data buffer for its handling of the Write Zero command? I would've assumed there's no data payload associated with that NVMe command, thus nothing that should even be subject to internal buffer scrambling inside fio.
Regards,
Adam
________________________________________
From: Vincent Fu <[email protected]>
Sent: Tuesday, November 5, 2024 2:41 PM
To: [email protected]
Cc: [email protected]
Subject: blog post on fio write zeroes performance
I am pleased to share a blog post about fio write zeroes performance:
Read about the unexpected performance issue we encountered. Feedback is welcome.
Vincent
Quoted 14 lines · Adam Horshack
Nice writeup Vincent. It's been a while since I've looked at the NVMe spec so perhaps I'm asking the wrong question but why is fio managing a data buffer for its handling of the Write Zero command? I would've assumed there's no data payload associated with that NVMe command, thus nothing that should even be subject to internal buffer scrambling inside fio.
Regards,
Adam
________________________________________
From: Vincent Fu <[email protected]>
Sent: Tuesday, November 5, 2024 2:41 PM
To: [email protected]
Cc: [email protected]
Subject: blog post on fio write zeroes performance
Quoted 10 lines
I am pleased to share a blog post about fio write zeroes performance:
Read about the unexpected performance issue we encountered. Feedback is welcome.
Vincent
When we added write zeroes support for the io_uring_cmd ioengine it was not on our mind that the default behavior for fio is to scramble buffers. Basically these patches just changed the opcode from a regular write to write zeroes.
You do have a good point that perhaps fio should be smart enough to avoid touching the buffers when doing write zeroes.
Thanks for the feedback!
Vincent
Internal Use - Confidential
Quoted 10 lines
From: Vincent Fu <[email protected]>
Sent: Tuesday, November 5, 2024 3:37 PM
To: Adam Horshack <[email protected]>; [email protected]
Subject: Re: blog post on fio write zeroes performanceWhen we added write zeroes support for the
io_uring_cmdioengine it was not on our mind that the default behavior for fio is to scramble buffers. Basically these patches just changed the opcode from a regular write to write zeroes.
Quoted 6 lines
You do have a good point that perhaps fio should be smart enough to avoid touching the buffers when doing write zeroes.
Thanks for the feedback!
Vincent
Does it mean that in the regular case (not write zeroes) fio spends significant time scrambling buffers and IO performance is lower than it could be?
Regards,
Mikhail
Quoted 15 lines · Mikhail
Internal Use - Confidential
Quoted 13 lines
From: Vincent Fu <[email protected]>
Sent: Tuesday, November 5, 2024 3:37 PM
To: Adam Horshack <[email protected]>; [email protected]
Subject: Re: blog post on fio write zeroes performanceWhen we added write zeroes support for the
io_uring_cmdioengine it was not on our mind that the default behavior for fio is to scramble buffers. Basically these patches just changed the opcode from a regular write to write zeroes.You do have a good point that perhaps fio should be smart enough to avoid touching the buffers when doing write zeroes.
Quoted 8 lines
Quoted 3 lines
Thanks for the feedback!
Vincent
Does it mean that in the regular case (not write zeroes) fio spends significant time scrambling buffers and IO performance is lower than it could be?
Regards,
Mikhail
For regular writes on contemporary devices the time that fio spends scrambling buffers is small compared to the device response time.
However, if you want the absolute highest write performance or are really pushing the limits with a device, it might be worth running the job with zero_buffers or scramble_buffers=0 to see what difference it makes.
Vincent
Quoted 8 lines · Vincent Fu
Quoted 5 lines · Mikhail
Does it mean that in the regular case (not write zeroes) fio spends significant time scrambling buffers and IO performance is lower than it could be?
Regards,
Mikhail
Quoted 6 lines
For regular writes on contemporary devices the time that fio spends scrambling buffers is small compared to the device response time.
However, if you want the absolute highest write performance or are really pushing the limits with a device, it might be worth running the job with
zero_buffersor scramble_buffers=0 to see what difference it makes.
It's worth noting that even in the "not write zeros dedicated command" case some devices (or things in the I/O path like filesystems etc.) can detect or compress buffers of zeros and thus are able to achieve (unrealistic) higher speeds that they cannot reach when "real" data is used...
--
Sitsofe