Multipath bio vs. request
Subsystems (2)
I will be presenting on this topic at LSF/MM/BPF this year, in the IO track.
Here's an introduction for my talk.
DMMP currently supports two different kernel IO interfaces: the BIO interface[1] (struct bio) and the Request interface[2] (struct request). By default DMMP uses the Request interface and over the years much work has been done test and improve the performance of the DMMP Request interface. DMMP can also be manually configured to use the BIO interface. The DMMP BIO interface is supported but little work has been done to test and improve its performance. DMMP is currently the only upstream component which continues to use the Request interface for submitting IO.
At the ALPSS 2024 conference last October we discussed the possibility of deprecating and eventually removing support the Request interface as kernel API. Such a change could impact DMMP so I was asked if Red Hat would be willing to support the effort by measuring the performance of DMMP's BIO interface[3] and comparing it to its Request based performance. Having such a comparative performance analysis would be very helpful in determining what further changes might be needed to move DMMP away from using the Request interface. This would help with the overall effort to improve BIO interface performance and eventually remove support for Request based IO as a kernel API.
In this presentation I will share the preliminary results of Red Hat's DMMP BIO vs Request performance tests[4] and discuss what the next possible steps could be for moving forward.
The tests and performance graphs in this presentation were developed and run by Samuel Petrovic <[email protected]>. Credit goes to Samuel for creating these performance tests and many thanks to Benjamin Marzinski <[email protected]>, Mikulas Patocka <[email protected]> and others on the Red Hat DMMP and Performance teams who contributed to this work.
[1] https://lwn.net/Articles/736534/
[2] https://lwn.net/Articles/738449/
[3] https://lore.kernel.org/linux-scsi/[email protected]
[4] https://people.redhat.com/jmeneghi/LSFMM_2025/DMMP_BIOvsRequest/
Signature · 4 lines
John A. Meneghini Senior Principal Platform Storage Engineer RHEL SST - Platform Storage Group [email protected]
Mike Snitzer
maintainerHi John,
Quoted 8 lines · John Meneghini
I will be presenting on this topic at LSF/MM/BPF this year, in the IO track.
Here's an introduction for my talk.
DMMP currently supports two different kernel IO interfaces: the BIO interface[1] (struct bio) and the Request interface[2] (struct request). By default DMMP uses the Request interface and over the years much work has been done test and improve the performance of the DMMP Request interface. DMMP can also be manually configured to use the BIO interface. The DMMP BIO interface is supported but little work has been done to test and improve its performance. DMMP is currently the only upstream component which continues to use the Request interface for submitting IO.
As I clarified at lunch today, your "DMMP is currently the only upstream component which continues to use the Request interface for submitting IO." makes no sense to me. The request-based DM multipath target is a blk-mq driver. It just acts like most blk-mq drivers.
What is different is DM core's request-based code will clone each request that gets submitted to the request-based DMMP device. And then when the request is submitted to an underlying path it gets directly inserted in the unlering blk-mq request-queue for that path.
So in those aspects request-based DM core and DM multipath are unique and they do require block interfaces that only benefit DMMP -- but that has _always_ been the case (nothing else ever needed to clone requests before submitting them).
Quoted 9 lines
At the ALPSS 2024 conference last October we discussed the possibility of deprecating and eventually removing support the Request interface as kernel API. Such a change could impact DMMP so I was asked if Red Hat would be willing to support the effort by measuring the performance of DMMP's BIO interface[3] and comparing it to its Request based performance. Having such a comparative performance analysis would be very helpful in determining what further changes might be needed to move DMMP away from using the Request interface. This would help with the overall effort to improve BIO interface performance and eventually remove support for Request based IO as a kernel API.
In this presentation I will share the preliminary results of Red Hat's DMMP BIO vs Request performance tests[4] and discuss what the next possible steps could be for moving forward.
Quoted 8 lines
The tests and performance graphs in this presentation were developed and run by Samuel Petrovic <[email protected]>. Credit goes to Samuel for creating these performance tests and many thanks to Benjamin Marzinski <[email protected]>, Mikulas Patocka <[email protected]> and others on the Red Hat DMMP and Performance teams who contributed to this work.
[1] https://lwn.net/Articles/736534/
[2] https://lwn.net/Articles/738449/
[3] https://lore.kernel.org/linux-scsi/[email protected]
[4] https://people.redhat.com/jmeneghi/LSFMM_2025/DMMP_BIOvsRequest/
Other useful context is the 2007 paper that provides an overview of why dm-multipath was switched from bio-based to request-based:
https://www.kernel.org/doc/ols/2007/ols2007v2-pages-235-244.pdf
Mike Snitzer
maintainerQuoted 17 lines · Mike Snitzer
Hi John,
Quoted 8 lines · John Meneghini
I will be presenting on this topic at LSF/MM/BPF this year, in the IO track.
Here's an introduction for my talk.
DMMP currently supports two different kernel IO interfaces: the BIO interface[1] (struct bio) and the Request interface[2] (struct request). By default DMMP uses the Request interface and over the years much work has been done test and improve the performance of the DMMP Request interface. DMMP can also be manually configured to use the BIO interface. The DMMP BIO interface is supported but little work has been done to test and improve its performance. DMMP is currently the only upstream component which continues to use the Request interface for submitting IO.
As I clarified at lunch today, your "DMMP is currently the only upstream component which continues to use the Request interface for submitting IO." makes no sense to me. The request-based DM multipath target is a blk-mq driver. It just acts like most blk-mq drivers.
Quoted 4 lines
What is different is DM core's request-based code will clone each request that gets submitted to the request-based DMMP device. And then when the request is submitted to an underlying path it gets directly inserted in the unlering blk-mq request-queue for that path.
Sorry for typoe: s/unlering/underlying/
Quoted 27 lines
So in those aspects request-based DM core and DM multipath are unique and they do require block interfaces that only benefit DMMP -- but that has _always_ been the case (nothing else ever needed to clone requests before submitting them).
Quoted 9 lines
At the ALPSS 2024 conference last October we discussed the possibility of deprecating and eventually removing support the Request interface as kernel API. Such a change could impact DMMP so I was asked if Red Hat would be willing to support the effort by measuring the performance of DMMP's BIO interface[3] and comparing it to its Request based performance. Having such a comparative performance analysis would be very helpful in determining what further changes might be needed to move DMMP away from using the Request interface. This would help with the overall effort to improve BIO interface performance and eventually remove support for Request based IO as a kernel API.
In this presentation I will share the preliminary results of Red Hat's DMMP BIO vs Request performance tests[4] and discuss what the next possible steps could be for moving forward.
Quoted 8 lines
The tests and performance graphs in this presentation were developed and run by Samuel Petrovic <[email protected]>. Credit goes to Samuel for creating these performance tests and many thanks to Benjamin Marzinski <[email protected]>, Mikulas Patocka <[email protected]> and others on the Red Hat DMMP and Performance teams who contributed to this work.
[1] https://lwn.net/Articles/736534/
[2] https://lwn.net/Articles/738449/
[3] https://lore.kernel.org/linux-scsi/[email protected]
[4] https://people.redhat.com/jmeneghi/LSFMM_2025/DMMP_BIOvsRequest/Other useful context is the 2007 paper that provides an overview of why dm-multipath was switched from bio-based to request-based:
https://www.kernel.org/doc/ols/2007/ols2007v2-pages-235-244.pdf
Quoted 1 line · John Meneghini
[4] https://people.redhat.com/jmeneghi/LSFMM_2025/DMMP_BIOvsRequest/
For the tests run in the link, what did you use for dm_mq_nr_hw_queues and dm_mq_queue_depth?