Rust block layer abstractions and benchmark strategies
I All,
I would like to propose that we have a session on Rust in the block layer again this year. Specifically I would like to discuss some rather puzzling results I observe when I benchmark the C and Rust null block drivers. I did a write up of the challenges I face at [1]. The observations are not tied to rust, they also manifest in the C driver.
Being able to consistently benchmark performance of the null block driver is rather important in terms of validating Rust for use in the block layer. I would hope to be able to collect some feedback on these issues during a session.
If time permits, I would like to give a status update on the efforts of building a feature complete null block driver in Rust. I will send additional patches that enable memory backed devices and timer completions before the Summit. Most of the patches have been ready for a while, but they are pending merge of dependencies (xarray, hrtimer, module_params).
If anyone is interested, I would make myself available for deep dives on the Rust block layer code base, 1:1 or tutorial style. We (the Rust kernel hackers) have had some good experiences with these kinds of sessions in other subsystems.
Best regards,
Andreas Hindborg
[1] https://metaspace.github.io/2024/12/02/problems-in-benchmark-land.html
Hi!
Quoted 5 lines · Andreas Hindborg via Lsf-pc
I would like to propose that we have a session on Rust in the block layer again this year. Specifically I would like to discuss some rather puzzling results I observe when I benchmark the C and Rust null block drivers. I did a write up of the challenges I face at [1]. The observations are not tied to rust, they also manifest in the C driver.
The results are indeed somewhat curious. One factor I didn't see addressed in your blog is CPU scheduling. I've seen in the past cases where IO tasks were getting migrated across cores leading to jumps in perfomance. Did you try binding fio jobs to one CPU each?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Hi Jan,
Quoted 9 lines · Jan Kara
Hi!
Quoted 5 lines · Andreas Hindborg via Lsf-pc
I would like to propose that we have a session on Rust in the block layer again this year. Specifically I would like to discuss some rather puzzling results I observe when I benchmark the C and Rust null block drivers. I did a write up of the challenges I face at [1]. The observations are not tied to rust, they also manifest in the C driver.
Quoted 4 lines
The results are indeed somewhat curious. One factor I didn't see addressed in your blog is CPU scheduling. I've seen in the past cases where IO tasks were getting migrated across cores leading to jumps in perfomance. Did you try binding fio jobs to one CPU each?
Yes, I am pinning the io jobs to cores with fio options cpus_allowed=0-<jobs> and --cpus_allowed_policy=split so I get 1 job per core.
The kernel is configured with PREEMPT_NONE=y.
Best regards,
Andreas Hindborg
Quoted 13 lines · Andreas Hindborg
Quoted 11 lines · Jan Kara
Quoted 5 lines · Andreas Hindborg via Lsf-pc
I would like to propose that we have a session on Rust in the block layer again this year. Specifically I would like to discuss some rather puzzling results I observe when I benchmark the C and Rust null block drivers. I did a write up of the challenges I face at [1]. The observations are not tied to rust, they also manifest in the C driver.
The results are indeed somewhat curious. One factor I didn't see addressed in your blog is CPU scheduling. I've seen in the past cases where IO tasks were getting migrated across cores leading to jumps in perfomance. Did you try binding fio jobs to one CPU each?
Quoted 4 lines
Yes, I am pinning the io jobs to cores with fio options
cpus_allowed=0-<jobs>and--cpus_allowed_policy=splitso I get 1 job per core.The kernel is configured with PREEMPT_NONE=y.
Ah, OK. In that case no great ideas from me. Since you've mentioned that when you get to slow / fast case the performance tends to stay there, perhaps you could use perf to profile the slow / fast case and see where's the difference?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Hello Andreas,
Quoted 18 lines · Andreas Hindborg
Hi Jan,
Quoted 9 lines · Jan Kara
Hi!
Quoted 5 lines · Andreas Hindborg via Lsf-pc
I would like to propose that we have a session on Rust in the block layer again this year. Specifically I would like to discuss some rather puzzling results I observe when I benchmark the C and Rust null block drivers. I did a write up of the challenges I face at [1]. The observations are not tied to rust, they also manifest in the C driver.
Quoted 4 lines
The results are indeed somewhat curious. One factor I didn't see addressed in your blog is CPU scheduling. I've seen in the past cases where IO tasks were getting migrated across cores leading to jumps in perfomance. Did you try binding fio jobs to one CPU each?
Quoted 4 lines
Yes, I am pinning the io jobs to cores with fio options
cpus_allowed=0-<jobs>and--cpus_allowed_policy=splitso I get 1 job per core.The kernel is configured with PREEMPT_NONE=y.
"I also cover a problem with the benchmark results that manifested during testing for v6.12-rc2."
I assume that all the results on:
https://metaspace.github.io/2024/12/02/problems-in-benchmark-land.html
are with kernel v6.12-rc2 ?
It would be interesting to test an older kernel version, and see if it is e.g. a scheduler bug.
You might also want to test with this series applied (which landed last minute before v6.13 was tagged):
https://lore.kernel.org/lkml/20250119110410.GAZ4zcKkx5sCjD5XvH@fat_crate.local/T/#u
It fixes bugs that were introduced in v6.12-rc1 and v6.7-rc2 respectively.
Kind regards,
Niklas
Quoted 26 lines · Niklas Cassel
Hello Andreas,
Quoted 18 lines · Andreas Hindborg
Hi Jan,
Quoted 9 lines · Jan Kara
Hi!
Quoted 5 lines · Andreas Hindborg via Lsf-pc
I would like to propose that we have a session on Rust in the block layer again this year. Specifically I would like to discuss some rather puzzling results I observe when I benchmark the C and Rust null block drivers. I did a write up of the challenges I face at [1]. The observations are not tied to rust, they also manifest in the C driver.
Quoted 4 lines
The results are indeed somewhat curious. One factor I didn't see addressed in your blog is CPU scheduling. I've seen in the past cases where IO tasks were getting migrated across cores leading to jumps in perfomance. Did you try binding fio jobs to one CPU each?
Quoted 4 lines
Yes, I am pinning the io jobs to cores with fio options
cpus_allowed=0-<jobs>and--cpus_allowed_policy=splitso I get 1 job per core.The kernel is configured with PREEMPT_NONE=y.
Quoted 7 lines
"I also cover a problem with the benchmark results that manifested during testing for v6.12-rc2."
I assume that all the results on:
https://metaspace.github.io/2024/12/02/problems-in-benchmark-land.htmlare with kernel v6.12-rc2 ?
Yes.
Quoted 3 lines
It would be interesting to test an older kernel version, and see if it is e.g. a scheduler bug.
Yes, I did not do that. I should collect some more detailed data for past kernels.
Quoted 7 lines
You might also want to test with this series applied (which landed last minute before v6.13 was tagged):
https://lore.kernel.org/lkml/20250119110410.GAZ4zcKkx5sCjD5XvH@fat_crate.local/T/#uIt fixes bugs that were introduced in v6.12-rc1 and v6.7-rc2 respectively.
I'll try that, thanks.
Best regards,
Andreas Hindborg