performance regression between 6.1.x and 5.15.x
Subsystems (6)
Hi,
I noticed a performance regression of xfs 6.1.27/6.1.23, with the compare to xfs 5.15.110.
It is yet not clear whether it is a problem of xfs or lvm2.
any guide to troubleshoot it?
test case:
disk: NVMe PCIe3 SSD *4
LVM: raid0 default strip size 64K.
fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30
-iodepth 1 -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=4 -directory=/mnt/test
6.1.27/6.1.23
fio bw=2623MiB/s (2750MB/s)
perf report:
Samples: 330K of event 'cycles', Event count (approx.): 120739812790
Overhead Command Shared Object Symbol 31.07% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_string 5.11% fio [kernel.kallsyms] [k] iomap_set_range_uptodate.part.24 3.36% fio [kernel.kallsyms] [k] asm_exc_nmi 3.29% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 2.27% fio [kernel.kallsyms] [k] iomap_write_begin 2.18% fio [kernel.kallsyms] [k] get_page_from_freelist 2.11% fio [kernel.kallsyms] [k] xas_load 2.10% fio [kernel.kallsyms] [k] xas_descend
5.15.110
fio bw=6796MiB/s (7126MB/s)
perf report:
Samples: 267K of event 'cycles', Event count (approx.): 186688803871
Overhead Command Shared Object Symbol 38.09% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_string 6.76% fio [kernel.kallsyms] [k] iomap_set_range_uptodate 4.40% fio [kernel.kallsyms] [k] xas_load 3.94% fio [kernel.kallsyms] [k] get_page_from_freelist 3.04% fio [kernel.kallsyms] [k] asm_exc_nmi 1.97% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 1.88% fio [kernel.kallsyms] [k] __pagevec_lru_add 1.53% fio [kernel.kallsyms] [k] iomap_write_begin 1.53% fio [kernel.kallsyms] [k] __add_to_page_cache_locked 1.41% fio [kernel.kallsyms] [k] xas_start
Best Regards
Wang Yugui ([email protected])
2023/05/08
Hi,
Quoted 35 lines
Hi,
I noticed a performance regression of xfs 6.1.27/6.1.23, with the compare to xfs 5.15.110.
It is yet not clear whether it is a problem of xfs or lvm2.
any guide to troubleshoot it?
test case:
disk: NVMe PCIe3 SSD *4
LVM: raid0 default strip size 64K.
fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30
-iodepth 1 -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=4 -directory=/mnt/test6.1.27/6.1.23
fio bw=2623MiB/s (2750MB/s)
perf report:
Samples: 330K of event 'cycles', Event count (approx.): 120739812790Overhead Command Shared Object Symbol 31.07% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_string 5.11% fio [kernel.kallsyms] [k] iomap_set_range_uptodate.part.24 3.36% fio [kernel.kallsyms] [k] asm_exc_nmi 3.29% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 2.27% fio [kernel.kallsyms] [k] iomap_write_begin 2.18% fio [kernel.kallsyms] [k] get_page_from_freelist 2.11% fio [kernel.kallsyms] [k] xas_load 2.10% fio [kernel.kallsyms] [k] xas_descend5.15.110
fio bw=6796MiB/s (7126MB/s)
perf report:
Samples: 267K of event 'cycles', Event count (approx.): 186688803871
Quoted 12 lines
Overhead Command Shared Object Symbol 38.09% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_string 6.76% fio [kernel.kallsyms] [k] iomap_set_range_uptodate 4.40% fio [kernel.kallsyms] [k] xas_load 3.94% fio [kernel.kallsyms] [k] get_page_from_freelist 3.04% fio [kernel.kallsyms] [k] asm_exc_nmi 1.97% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 1.88% fio [kernel.kallsyms] [k] __pagevec_lru_add 1.53% fio [kernel.kallsyms] [k] iomap_write_begin 1.53% fio [kernel.kallsyms] [k] __add_to_page_cache_locked 1.41% fio [kernel.kallsyms] [k] xas_start
more info:
1, 6.2.14 have same performance as 6.1.x
2, 6.1.x fio performance detail:
Jobs: 4 (f=4): [W(4)][16.7%][w=10.2GiB/s][w=10.4k IOPS][eta 00m:15s]
Jobs: 4 (f=4): [W(4)][25.0%][w=9949MiB/s][w=9949 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][31.2%][w=9618MiB/s][w=9618 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][37.5%][w=7970MiB/s][w=7970 IOPS][eta 00m:10s]
Jobs: 4 (f=4): [W(4)][41.2%][w=5048MiB/s][w=5047 IOPS][eta 00m:10s]
Jobs: 4 (f=4): [W(4)][42.1%][w=2489MiB/s][w=2488 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][42.9%][w=3227MiB/s][w=3226 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][45.5%][w=3622MiB/s][w=3622 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][47.8%][w=3651MiB/s][w=3650 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][52.2%][w=3435MiB/s][w=3435 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][52.0%][w=2464MiB/s][w=2463 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][53.8%][w=2438MiB/s][w=2438 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][55.6%][w=2435MiB/s][w=2434 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][57.1%][w=2449MiB/s][w=2448 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][60.7%][w=2422MiB/s][w=2421 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][62.1%][w=2457MiB/s][w=2457 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][63.3%][w=2436MiB/s][w=2436 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][64.5%][w=2432MiB/s][w=2431 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][67.7%][w=2440MiB/s][w=2440 IOPS][eta 00m:10s]
Jobs: 4 (f=4): [W(4)][71.0%][w=2437MiB/s][w=2437 IOPS][eta 00m:09s]
Jobs: 4 (f=4): [W(4)][74.2%][w=2442MiB/s][w=2442 IOPS][eta 00m:08s]
Jobs: 4 (f=4): [W(4)][77.4%][w=2425MiB/s][w=2424 IOPS][eta 00m:07s]
Jobs: 4 (f=4): [W(4)][80.6%][w=2459MiB/s][w=2459 IOPS][eta 00m:06s]
Jobs: 4 (f=4): [W(4)][86.7%][w=2428MiB/s][w=2427 IOPS][eta 00m:04s]
Jobs: 4 (f=4): [W(4)][90.0%][w=2441MiB/s][w=2440 IOPS][eta 00m:03s]
Jobs: 4 (f=4): [W(4)][93.3%][w=2438MiB/s][w=2437 IOPS][eta 00m:02s]
Jobs: 4 (f=4): [W(4)][96.7%][w=2450MiB/s][w=2449 IOPS][eta 00m:01s]
Jobs: 4 (f=4): [W(4)][100.0%][w=2430MiB/s][w=2430 IOPS][eta 00m:00s]
Jobs: 4 (f=4): [F(4)][100.0%][w=2372MiB/s][w=2372 IOPS][eta 00m:00s]
5.15 fio performance detail:
Jobs: 4 (f=4): [W(4)][14.3%][w=8563MiB/s][w=8563 IOPS][eta 00m:18s]
Jobs: 4 (f=4): [W(4)][18.2%][w=6376MiB/s][w=6375 IOPS][eta 00m:18s]
Jobs: 4 (f=4): [W(4)][20.8%][w=4566MiB/s][w=4565 IOPS][eta 00m:19s]
Jobs: 4 (f=4): [W(4)][23.1%][w=3947MiB/s][w=3947 IOPS][eta 00m:20s]
Jobs: 4 (f=4): [W(4)][25.9%][w=4601MiB/s][w=4601 IOPS][eta 00m:20s]
Jobs: 4 (f=4): [W(4)][28.6%][w=5797MiB/s][w=5796 IOPS][eta 00m:20s]
Jobs: 4 (f=4): [W(4)][32.1%][w=6802MiB/s][w=6801 IOPS][eta 00m:19s]
Jobs: 4 (f=4): [W(4)][35.7%][w=7411MiB/s][w=7411 IOPS][eta 00m:18s]
Jobs: 4 (f=4): [W(4)][40.7%][w=8445MiB/s][w=8444 IOPS][eta 00m:16s]
Jobs: 4 (f=4): [W(4)][46.2%][w=7992MiB/s][w=7992 IOPS][eta 00m:14s]
Jobs: 4 (f=4): [W(4)][52.0%][w=8118MiB/s][w=8117 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][56.0%][w=7742MiB/s][w=7741 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][62.5%][w=7497MiB/s][w=7496 IOPS][eta 00m:09s]
Jobs: 4 (f=4): [W(4)][66.7%][w=7248MiB/s][w=7248 IOPS][eta 00m:08s]
Jobs: 4 (f=4): [W(4)][70.8%][w=7461MiB/s][w=7460 IOPS][eta 00m:07s]
Jobs: 4 (f=4): [W(4)][75.0%][w=7959MiB/s][w=7959 IOPS][eta 00m:06s]
Jobs: 4 (f=4): [W(3),F(1)][79.2%][w=6982MiB/s][w=6982 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [_(2),W(1),_(1)][87.0%][w=2809MiB/s][w=2808 IOPS][eta 00m:03s]
Jobs: 1 (f=1): [_(2),W(1),_(1)][95.5%][w=2669MiB/s][w=2668 IOPS][eta 00m:01s]
Jobs: 1 (f=1): [_(2),F(1),_(1)][100.0%][w=2552MiB/s][w=2552 IOPS][eta 00m:00s]
3, 'sysctl -a |grep dirty' of 6.1.x and 5.15.x
vm.dirty_background_bytes = 1073741824
vm.dirty_background_ratio = 0
vm.dirty_bytes = 8589934592
vm.dirty_expire_centisecs = 600
vm.dirty_ratio = 0
vm.dirty_writeback_centisecs = 200
vm.dirtytime_expire_seconds = 43200
Best Regards
Wang Yugui ([email protected])
2023/05/08
Quoted 48 lines · Wang Yugui
Hi,
Quoted 46 lines
Hi,
I noticed a performance regression of xfs 6.1.27/6.1.23, with the compare to xfs 5.15.110.
It is yet not clear whether it is a problem of xfs or lvm2.
any guide to troubleshoot it?
test case:
disk: NVMe PCIe3 SSD *4
LVM: raid0 default strip size 64K.
fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30
-iodepth 1 -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=4 -directory=/mnt/test6.1.27/6.1.23
fio bw=2623MiB/s (2750MB/s)
perf report:
Samples: 330K of event 'cycles', Event count (approx.): 120739812790Overhead Command Shared Object Symbol 31.07% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_string 5.11% fio [kernel.kallsyms] [k] iomap_set_range_uptodate.part.24 3.36% fio [kernel.kallsyms] [k] asm_exc_nmi 3.29% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 2.27% fio [kernel.kallsyms] [k] iomap_write_begin 2.18% fio [kernel.kallsyms] [k] get_page_from_freelist 2.11% fio [kernel.kallsyms] [k] xas_load 2.10% fio [kernel.kallsyms] [k] xas_descend5.15.110
fio bw=6796MiB/s (7126MB/s)
perf report:
Samples: 267K of event 'cycles', Event count (approx.): 186688803871Overhead Command Shared Object Symbol 38.09% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_string 6.76% fio [kernel.kallsyms] [k] iomap_set_range_uptodate 4.40% fio [kernel.kallsyms] [k] xas_load 3.94% fio [kernel.kallsyms] [k] get_page_from_freelist 3.04% fio [kernel.kallsyms] [k] asm_exc_nmi 1.97% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 1.88% fio [kernel.kallsyms] [k] __pagevec_lru_add 1.53% fio [kernel.kallsyms] [k] iomap_write_begin 1.53% fio [kernel.kallsyms] [k] __add_to_page_cache_locked 1.41% fio [kernel.kallsyms] [k] xas_start
Because you are testing buffered IO, you need to run perf across all CPUs and tasks, not just the fio process so that it captures the profile of memory reclaim and writeback that is being performed by the kernel.
Quoted 4 lines
more info:
1, 6.2.14 have same performance as 6.1.x
Quoted 13 lines
2, 6.1.x fio performance detail:
Jobs: 4 (f=4): [W(4)][16.7%][w=10.2GiB/s][w=10.4k IOPS][eta 00m:15s]
Jobs: 4 (f=4): [W(4)][25.0%][w=9949MiB/s][w=9949 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][31.2%][w=9618MiB/s][w=9618 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][37.5%][w=7970MiB/s][w=7970 IOPS][eta 00m:10s]
Jobs: 4 (f=4): [W(4)][41.2%][w=5048MiB/s][w=5047 IOPS][eta 00m:10s]
Jobs: 4 (f=4): [W(4)][42.1%][w=2489MiB/s][w=2488 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][42.9%][w=3227MiB/s][w=3226 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][45.5%][w=3622MiB/s][w=3622 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][47.8%][w=3651MiB/s][w=3650 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][52.2%][w=3435MiB/s][w=3435 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][52.0%][w=2464MiB/s][w=2463 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][53.8%][w=2438MiB/s][w=2438 IOPS][eta 00m:12s]
Looks like it's throttled on dirty pages at this point.
How much memory does your test system have, and what does changing the writeback throttling thresholds do? What's the numa layout?
Watching the stats in /proc/meminfo would be useful. I tend to use Performance Co-Pilot (PCP) to collect these sorts of stats and plot them in real time so I can see how the state of the machine is changing as the test is running....
Quoted 9 lines
Jobs: 4 (f=4): [W(4)][55.6%][w=2435MiB/s][w=2434 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][57.1%][w=2449MiB/s][w=2448 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][60.7%][w=2422MiB/s][w=2421 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][62.1%][w=2457MiB/s][w=2457 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][63.3%][w=2436MiB/s][w=2436 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][64.5%][w=2432MiB/s][w=2431 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][67.7%][w=2440MiB/s][w=2440 IOPS][eta 00m:10s]
Jobs: 4 (f=4): [W(4)][71.0%][w=2437MiB/s][w=2437 IOPS][eta 00m:09s]
Jobs: 4 (f=4): [W(4)][74.2%][w=2442MiB/s][w=2442 IOPS][eta 00m:08s]
Quoted 8 lines
Jobs: 4 (f=4): [W(4)][77.4%][w=2425MiB/s][w=2424 IOPS][eta 00m:07s]
Jobs: 4 (f=4): [W(4)][80.6%][w=2459MiB/s][w=2459 IOPS][eta 00m:06s]
Jobs: 4 (f=4): [W(4)][86.7%][w=2428MiB/s][w=2427 IOPS][eta 00m:04s]
Jobs: 4 (f=4): [W(4)][90.0%][w=2441MiB/s][w=2440 IOPS][eta 00m:03s]
Jobs: 4 (f=4): [W(4)][93.3%][w=2438MiB/s][w=2437 IOPS][eta 00m:02s]
Jobs: 4 (f=4): [W(4)][96.7%][w=2450MiB/s][w=2449 IOPS][eta 00m:01s]
Jobs: 4 (f=4): [W(4)][100.0%][w=2430MiB/s][w=2430 IOPS][eta 00m:00s]
Jobs: 4 (f=4): [F(4)][100.0%][w=2372MiB/s][w=2372 IOPS][eta 00m:00s]
fsync at the end is instant, which indicates that writing into the kernel is almost certainly being throttled.
Quoted 5 lines
5.15 fio performance detail:
Jobs: 4 (f=4): [W(4)][14.3%][w=8563MiB/s][w=8563 IOPS][eta 00m:18s]
Jobs: 4 (f=4): [W(4)][18.2%][w=6376MiB/s][w=6375 IOPS][eta 00m:18s]
Jobs: 4 (f=4): [W(4)][20.8%][w=4566MiB/s][w=4565 IOPS][eta 00m:19s]
Jobs: 4 (f=4): [W(4)][23.1%][w=3947MiB/s][w=3947 IOPS][eta 00m:20s]
So throttling starts and perf drops...
Quoted 5 lines
Jobs: 4 (f=4): [W(4)][25.9%][w=4601MiB/s][w=4601 IOPS][eta 00m:20s]
Jobs: 4 (f=4): [W(4)][28.6%][w=5797MiB/s][w=5796 IOPS][eta 00m:20s]
Jobs: 4 (f=4): [W(4)][32.1%][w=6802MiB/s][w=6801 IOPS][eta 00m:19s]
Jobs: 4 (f=4): [W(4)][35.7%][w=7411MiB/s][w=7411 IOPS][eta 00m:18s]
Jobs: 4 (f=4): [W(4)][40.7%][w=8445MiB/s][w=8444 IOPS][eta 00m:16s]
.... and then it picks back up....
Quoted 11 lines
Jobs: 4 (f=4): [W(4)][46.2%][w=7992MiB/s][w=7992 IOPS][eta 00m:14s]
Jobs: 4 (f=4): [W(4)][52.0%][w=8118MiB/s][w=8117 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][56.0%][w=7742MiB/s][w=7741 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][62.5%][w=7497MiB/s][w=7496 IOPS][eta 00m:09s]
Jobs: 4 (f=4): [W(4)][66.7%][w=7248MiB/s][w=7248 IOPS][eta 00m:08s]
Jobs: 4 (f=4): [W(4)][70.8%][w=7461MiB/s][w=7460 IOPS][eta 00m:07s]
Jobs: 4 (f=4): [W(4)][75.0%][w=7959MiB/s][w=7959 IOPS][eta 00m:06s]
Jobs: 4 (f=4): [W(3),F(1)][79.2%][w=6982MiB/s][w=6982 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [_(2),W(1),_(1)][87.0%][w=2809MiB/s][w=2808 IOPS][eta 00m:03s]
Jobs: 1 (f=1): [_(2),W(1),_(1)][95.5%][w=2669MiB/s][w=2668 IOPS][eta 00m:01s]
Jobs: 1 (f=1): [_(2),F(1),_(1)][100.0%][w=2552MiB/s][w=2552 IOPS][eta 00m:00s]
I suspect that the likely culprit is mm-level changes - the page reclaim algorithm was completely replaced in 6.1 with a multi-generation LRU that will have different cache footprint behaviour in exactly this sort of "repeatedly over-write same files in a set that are significantly larger than memory" micro-benchmark.
i.e. these commits:
07017acb0601 mm: multi-gen LRU: admin guide d6c3af7d8a2b mm: multi-gen LRU: debugfs interface 1332a809d95a mm: multi-gen LRU: thrashing prevention 354ed5974429 mm: multi-gen LRU: kill switch f76c83378851 mm: multi-gen LRU: optimize multiple memcgs bd74fdaea146 mm: multi-gen LRU: support page table walks 018ee47f1489 mm: multi-gen LRU: exploit locality in rmap ac35a4902374 mm: multi-gen LRU: minimal implementation ec1c86b25f4b mm: multi-gen LRU: groundwork
If that's the case, I'd expect kernels up to 6.0 to demonstrate the same behaviour as 5.15, and 6.1+ to demonstrate the same behaviour as you've reported.
So I'm thinking that the best thing to do is confirm that the change of behaviour is a result of the multi-gen LRU changes. If it is, then it's up to the multi-gen LRU developers to determine how to fix it. If it's not the multi-gen LRU, we'll have to kep digging.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Hi,
Quoted 50 lines · Wang Yugui
Quoted 48 lines
Hi,
Quoted 46 lines
Hi,
I noticed a performance regression of xfs 6.1.27/6.1.23, with the compare to xfs 5.15.110.
It is yet not clear whether it is a problem of xfs or lvm2.
any guide to troubleshoot it?
test case:
disk: NVMe PCIe3 SSD *4
LVM: raid0 default strip size 64K.
fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30
-iodepth 1 -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=4 -directory=/mnt/test6.1.27/6.1.23
fio bw=2623MiB/s (2750MB/s)
perf report:
Samples: 330K of event 'cycles', Event count (approx.): 120739812790Overhead Command Shared Object Symbol 31.07% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_string 5.11% fio [kernel.kallsyms] [k] iomap_set_range_uptodate.part.24 3.36% fio [kernel.kallsyms] [k] asm_exc_nmi 3.29% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 2.27% fio [kernel.kallsyms] [k] iomap_write_begin 2.18% fio [kernel.kallsyms] [k] get_page_from_freelist 2.11% fio [kernel.kallsyms] [k] xas_load 2.10% fio [kernel.kallsyms] [k] xas_descend5.15.110
fio bw=6796MiB/s (7126MB/s)
perf report:
Samples: 267K of event 'cycles', Event count (approx.): 186688803871Overhead Command Shared Object Symbol 38.09% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_string 6.76% fio [kernel.kallsyms] [k] iomap_set_range_uptodate 4.40% fio [kernel.kallsyms] [k] xas_load 3.94% fio [kernel.kallsyms] [k] get_page_from_freelist 3.04% fio [kernel.kallsyms] [k] asm_exc_nmi 1.97% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 1.88% fio [kernel.kallsyms] [k] __pagevec_lru_add 1.53% fio [kernel.kallsyms] [k] iomap_write_begin 1.53% fio [kernel.kallsyms] [k] __add_to_page_cache_locked 1.41% fio [kernel.kallsyms] [k] xas_start
Quoted 4 lines
Because you are testing buffered IO, you need to run perf across all CPUs and tasks, not just the fio process so that it captures the profile of memory reclaim and writeback that is being performed by the kernel.
'perf report' of all CPU.
Samples: 211K of event 'cycles', Event count (approx.): 56590727219
Overhead Command Shared Object Symbol 16.29% fio [kernel.kallsyms] [k] rep_movs_alternative 3.38% kworker/u98:1+f [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 3.11% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 3.05% swapper [kernel.kallsyms] [k] intel_idle 2.63% fio [kernel.kallsyms] [k] get_page_from_freelist 2.33% fio [kernel.kallsyms] [k] asm_exc_nmi 2.26% kworker/u98:1+f [kernel.kallsyms] [k] __folio_start_writeback 1.40% fio [kernel.kallsyms] [k] __filemap_add_folio 1.37% fio [kernel.kallsyms] [k] lru_add_fn 1.35% fio [kernel.kallsyms] [k] xas_load 1.33% fio [kernel.kallsyms] [k] iomap_write_begin 1.31% fio [kernel.kallsyms] [k] xas_descend 1.19% kworker/u98:1+f [kernel.kallsyms] [k] folio_clear_dirty_for_io 1.07% fio [kernel.kallsyms] [k] folio_add_lru 1.01% fio [kernel.kallsyms] [k] __folio_mark_dirty 1.00% kworker/u98:1+f [kernel.kallsyms] [k] _raw_spin_lock_irqsave
and 'top' show that 'kworker/u98:1' have over 80% CPU usage.
Quoted 18 lines
Quoted 4 lines
more info:
1, 6.2.14 have same performance as 6.1.x
Quoted 13 lines
2, 6.1.x fio performance detail:
Jobs: 4 (f=4): [W(4)][16.7%][w=10.2GiB/s][w=10.4k IOPS][eta 00m:15s]
Jobs: 4 (f=4): [W(4)][25.0%][w=9949MiB/s][w=9949 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][31.2%][w=9618MiB/s][w=9618 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][37.5%][w=7970MiB/s][w=7970 IOPS][eta 00m:10s]
Jobs: 4 (f=4): [W(4)][41.2%][w=5048MiB/s][w=5047 IOPS][eta 00m:10s]
Jobs: 4 (f=4): [W(4)][42.1%][w=2489MiB/s][w=2488 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][42.9%][w=3227MiB/s][w=3226 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][45.5%][w=3622MiB/s][w=3622 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][47.8%][w=3651MiB/s][w=3650 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][52.2%][w=3435MiB/s][w=3435 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][52.0%][w=2464MiB/s][w=2463 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][53.8%][w=2438MiB/s][w=2438 IOPS][eta 00m:12s]
Quoted 4 lines
Looks like it's throttled on dirty pages at this point.
How much memory does your test system have, and what does changing the writeback throttling thresholds do? What's the numa layout?
This is a DELL server with two E5-2680 v2 configured as NUMA.
There is 192G memory in this server,
128G memory is installed in CPU1,
64G memory is installed in CPU2.
and sysctl config:
vm.dirty_background_bytes = 1073741824 # 1G
vm.dirty_background_ratio = 0
vm.dirty_bytes = 8589934592 # 8G
vm.dirty_ratio = 0
and NVMe SSD *4 are connected to one NVMe hba / CPU1.
Quoted 73 lines
Watching the stats in
/proc/meminfowould be useful. I tend to use Performance Co-Pilot (PCP) to collect these sorts of stats and plot them in real time so I can see how the state of the machine is changing as the test is running....Quoted 9 lines
Jobs: 4 (f=4): [W(4)][55.6%][w=2435MiB/s][w=2434 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][57.1%][w=2449MiB/s][w=2448 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][60.7%][w=2422MiB/s][w=2421 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][62.1%][w=2457MiB/s][w=2457 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][63.3%][w=2436MiB/s][w=2436 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][64.5%][w=2432MiB/s][w=2431 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][67.7%][w=2440MiB/s][w=2440 IOPS][eta 00m:10s]
Jobs: 4 (f=4): [W(4)][71.0%][w=2437MiB/s][w=2437 IOPS][eta 00m:09s]
Jobs: 4 (f=4): [W(4)][74.2%][w=2442MiB/s][w=2442 IOPS][eta 00m:08s]Quoted 8 lines
Jobs: 4 (f=4): [W(4)][77.4%][w=2425MiB/s][w=2424 IOPS][eta 00m:07s]
Jobs: 4 (f=4): [W(4)][80.6%][w=2459MiB/s][w=2459 IOPS][eta 00m:06s]
Jobs: 4 (f=4): [W(4)][86.7%][w=2428MiB/s][w=2427 IOPS][eta 00m:04s]
Jobs: 4 (f=4): [W(4)][90.0%][w=2441MiB/s][w=2440 IOPS][eta 00m:03s]
Jobs: 4 (f=4): [W(4)][93.3%][w=2438MiB/s][w=2437 IOPS][eta 00m:02s]
Jobs: 4 (f=4): [W(4)][96.7%][w=2450MiB/s][w=2449 IOPS][eta 00m:01s]
Jobs: 4 (f=4): [W(4)][100.0%][w=2430MiB/s][w=2430 IOPS][eta 00m:00s]
Jobs: 4 (f=4): [F(4)][100.0%][w=2372MiB/s][w=2372 IOPS][eta 00m:00s]fsync at the end is instant, which indicates that writing into the kernel is almost certainly being throttled.
Quoted 5 lines
5.15 fio performance detail:
Jobs: 4 (f=4): [W(4)][14.3%][w=8563MiB/s][w=8563 IOPS][eta 00m:18s]
Jobs: 4 (f=4): [W(4)][18.2%][w=6376MiB/s][w=6375 IOPS][eta 00m:18s]
Jobs: 4 (f=4): [W(4)][20.8%][w=4566MiB/s][w=4565 IOPS][eta 00m:19s]
Jobs: 4 (f=4): [W(4)][23.1%][w=3947MiB/s][w=3947 IOPS][eta 00m:20s]So throttling starts and perf drops...
Quoted 5 lines
Jobs: 4 (f=4): [W(4)][25.9%][w=4601MiB/s][w=4601 IOPS][eta 00m:20s]
Jobs: 4 (f=4): [W(4)][28.6%][w=5797MiB/s][w=5796 IOPS][eta 00m:20s]
Jobs: 4 (f=4): [W(4)][32.1%][w=6802MiB/s][w=6801 IOPS][eta 00m:19s]
Jobs: 4 (f=4): [W(4)][35.7%][w=7411MiB/s][w=7411 IOPS][eta 00m:18s]
Jobs: 4 (f=4): [W(4)][40.7%][w=8445MiB/s][w=8444 IOPS][eta 00m:16s].... and then it picks back up....
Quoted 11 lines
Jobs: 4 (f=4): [W(4)][46.2%][w=7992MiB/s][w=7992 IOPS][eta 00m:14s]
Jobs: 4 (f=4): [W(4)][52.0%][w=8118MiB/s][w=8117 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [W(4)][56.0%][w=7742MiB/s][w=7741 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [W(4)][62.5%][w=7497MiB/s][w=7496 IOPS][eta 00m:09s]
Jobs: 4 (f=4): [W(4)][66.7%][w=7248MiB/s][w=7248 IOPS][eta 00m:08s]
Jobs: 4 (f=4): [W(4)][70.8%][w=7461MiB/s][w=7460 IOPS][eta 00m:07s]
Jobs: 4 (f=4): [W(4)][75.0%][w=7959MiB/s][w=7959 IOPS][eta 00m:06s]
Jobs: 4 (f=4): [W(3),F(1)][79.2%][w=6982MiB/s][w=6982 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [_(2),W(1),_(1)][87.0%][w=2809MiB/s][w=2808 IOPS][eta 00m:03s]
Jobs: 1 (f=1): [_(2),W(1),_(1)][95.5%][w=2669MiB/s][w=2668 IOPS][eta 00m:01s]
Jobs: 1 (f=1): [_(2),F(1),_(1)][100.0%][w=2552MiB/s][w=2552 IOPS][eta 00m:00s]I suspect that the likely culprit is mm-level changes - the page reclaim algorithm was completely replaced in 6.1 with a multi-generation LRU that will have different cache footprint behaviour in exactly this sort of "repeatedly over-write same files in a set that are significantly larger than memory" micro-benchmark.
i.e. these commits:
07017acb0601mm: multi-gen LRU: admin guided6c3af7d8a2bmm: multi-gen LRU: debugfs interface1332a809d95amm: multi-gen LRU: thrashing prevention354ed5974429mm: multi-gen LRU: kill switchf76c83378851mm: multi-gen LRU: optimize multiple memcgsbd74fdaea146mm: multi-gen LRU: support page table walks018ee47f1489mm: multi-gen LRU: exploit locality in rmapac35a4902374mm: multi-gen LRU: minimal implementationec1c86b25f4bmm: multi-gen LRU: groundwork
Quoted 3 lines
If that's the case, I'd expect kernels up to 6.0 to demonstrate the same behaviour as 5.15, and 6.1+ to demonstrate the same behaviour as you've reported.
I tested 6.4.0-rc1. the performance become a little worse.
Best Regards
Wang Yugui ([email protected])
2023/05/09
Quoted 11 lines
So I'm thinking that the best thing to do is confirm that the change of behaviour is a result of the multi-gen LRU changes. If it is, then it's up to the multi-gen LRU developers to determine how to fix it. If it's not the multi-gen LRU, we'll have to kep digging.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 77 lines · Wang Yugui
Hi,
Quoted 50 lines · Wang Yugui
Quoted 48 lines
Hi,
Quoted 46 lines
Hi,
I noticed a performance regression of xfs 6.1.27/6.1.23, with the compare to xfs 5.15.110.
It is yet not clear whether it is a problem of xfs or lvm2.
any guide to troubleshoot it?
test case:
disk: NVMe PCIe3 SSD *4
LVM: raid0 default strip size 64K.
fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30
-iodepth 1 -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=4 -directory=/mnt/test6.1.27/6.1.23
fio bw=2623MiB/s (2750MB/s)
perf report:
Samples: 330K of event 'cycles', Event count (approx.): 120739812790Overhead Command Shared Object Symbol 31.07% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_string 5.11% fio [kernel.kallsyms] [k] iomap_set_range_uptodate.part.24 3.36% fio [kernel.kallsyms] [k] asm_exc_nmi 3.29% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 2.27% fio [kernel.kallsyms] [k] iomap_write_begin 2.18% fio [kernel.kallsyms] [k] get_page_from_freelist 2.11% fio [kernel.kallsyms] [k] xas_load 2.10% fio [kernel.kallsyms] [k] xas_descend5.15.110
fio bw=6796MiB/s (7126MB/s)
perf report:
Samples: 267K of event 'cycles', Event count (approx.): 186688803871Overhead Command Shared Object Symbol 38.09% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_string 6.76% fio [kernel.kallsyms] [k] iomap_set_range_uptodate 4.40% fio [kernel.kallsyms] [k] xas_load 3.94% fio [kernel.kallsyms] [k] get_page_from_freelist 3.04% fio [kernel.kallsyms] [k] asm_exc_nmi 1.97% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 1.88% fio [kernel.kallsyms] [k] __pagevec_lru_add 1.53% fio [kernel.kallsyms] [k] iomap_write_begin 1.53% fio [kernel.kallsyms] [k] __add_to_page_cache_locked 1.41% fio [kernel.kallsyms] [k] xas_startQuoted 4 lines
Because you are testing buffered IO, you need to run perf across all CPUs and tasks, not just the fio process so that it captures the profile of memory reclaim and writeback that is being performed by the kernel.
'perf report' of all CPU.
Samples: 211K of event 'cycles', Event count (approx.): 56590727219Overhead Command Shared Object Symbol 16.29% fio [kernel.kallsyms] [k] rep_movs_alternative 3.38% kworker/u98:1+f [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 3.11% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 3.05% swapper [kernel.kallsyms] [k] intel_idle 2.63% fio [kernel.kallsyms] [k] get_page_from_freelist 2.33% fio [kernel.kallsyms] [k] asm_exc_nmi 2.26% kworker/u98:1+f [kernel.kallsyms] [k] __folio_start_writeback 1.40% fio [kernel.kallsyms] [k] __filemap_add_folio 1.37% fio [kernel.kallsyms] [k] lru_add_fn 1.35% fio [kernel.kallsyms] [k] xas_load 1.33% fio [kernel.kallsyms] [k] iomap_write_begin 1.31% fio [kernel.kallsyms] [k] xas_descend 1.19% kworker/u98:1+f [kernel.kallsyms] [k] folio_clear_dirty_for_io 1.07% fio [kernel.kallsyms] [k] folio_add_lru 1.01% fio [kernel.kallsyms] [k] __folio_mark_dirty 1.00% kworker/u98:1+f [kernel.kallsyms] [k] _raw_spin_lock_irqsave
Quoted 1 line
and 'top' show that 'kworker/u98:1' have over 80% CPU usage.
Can you provide an expanded callgraph profile for both the good and bad kernels showing the CPU used in the fio write() path and the kworker-based writeback path?
[ The test machine I have that I could reproduce this sort of performance anomoly went bad a month ago, so I have no hardware available to me right now to reproduce this behaviour locally. Hence I'll need you to do the profiling I need to understand the regression for me. ]
Quoted 18 lines
Quoted 18 lines
I suspect that the likely culprit is mm-level changes - the page reclaim algorithm was completely replaced in 6.1 with a multi-generation LRU that will have different cache footprint behaviour in exactly this sort of "repeatedly over-write same files in a set that are significantly larger than memory" micro-benchmark.
i.e. these commits:
07017acb0601mm: multi-gen LRU: admin guided6c3af7d8a2bmm: multi-gen LRU: debugfs interface1332a809d95amm: multi-gen LRU: thrashing prevention354ed5974429mm: multi-gen LRU: kill switchf76c83378851mm: multi-gen LRU: optimize multiple memcgsbd74fdaea146mm: multi-gen LRU: support page table walks018ee47f1489mm: multi-gen LRU: exploit locality in rmapac35a4902374mm: multi-gen LRU: minimal implementationec1c86b25f4bmm: multi-gen LRU: groundwork
Quoted 5 lines
Quoted 3 lines
If that's the case, I'd expect kernels up to 6.0 to demonstrate the same behaviour as 5.15, and 6.1+ to demonstrate the same behaviour as you've reported.
I tested 6.4.0-rc1. the performance become a little worse.
Thanks, that's as I expected.
WHich means that the interesting kernel versions to check now are a 6.0.x kernel, and then if it has the same perf as 5.15.x, then the commit before the multi-gen LRU was introduced vs the commit after the multi-gen LRU was introduced to see if that is the functionality that introduced the regression....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Hi,
Quoted 78 lines · Wang Yugui
Quoted 77 lines
Hi,
Quoted 50 lines · Wang Yugui
Quoted 48 lines
Hi,
Quoted 46 lines
Hi,
I noticed a performance regression of xfs 6.1.27/6.1.23, with the compare to xfs 5.15.110.
It is yet not clear whether it is a problem of xfs or lvm2.
any guide to troubleshoot it?
test case:
disk: NVMe PCIe3 SSD *4
LVM: raid0 default strip size 64K.
fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30
-iodepth 1 -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=4 -directory=/mnt/test6.1.27/6.1.23
fio bw=2623MiB/s (2750MB/s)
perf report:
Samples: 330K of event 'cycles', Event count (approx.): 120739812790Overhead Command Shared Object Symbol 31.07% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_string 5.11% fio [kernel.kallsyms] [k] iomap_set_range_uptodate.part.24 3.36% fio [kernel.kallsyms] [k] asm_exc_nmi 3.29% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 2.27% fio [kernel.kallsyms] [k] iomap_write_begin 2.18% fio [kernel.kallsyms] [k] get_page_from_freelist 2.11% fio [kernel.kallsyms] [k] xas_load 2.10% fio [kernel.kallsyms] [k] xas_descend5.15.110
fio bw=6796MiB/s (7126MB/s)
perf report:
Samples: 267K of event 'cycles', Event count (approx.): 186688803871Overhead Command Shared Object Symbol 38.09% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_string 6.76% fio [kernel.kallsyms] [k] iomap_set_range_uptodate 4.40% fio [kernel.kallsyms] [k] xas_load 3.94% fio [kernel.kallsyms] [k] get_page_from_freelist 3.04% fio [kernel.kallsyms] [k] asm_exc_nmi 1.97% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 1.88% fio [kernel.kallsyms] [k] __pagevec_lru_add 1.53% fio [kernel.kallsyms] [k] iomap_write_begin 1.53% fio [kernel.kallsyms] [k] __add_to_page_cache_locked 1.41% fio [kernel.kallsyms] [k] xas_startQuoted 4 lines
Because you are testing buffered IO, you need to run perf across all CPUs and tasks, not just the fio process so that it captures the profile of memory reclaim and writeback that is being performed by the kernel.
'perf report' of all CPU.
Samples: 211K of event 'cycles', Event count (approx.): 56590727219Overhead Command Shared Object Symbol 16.29% fio [kernel.kallsyms] [k] rep_movs_alternative 3.38% kworker/u98:1+f [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 3.11% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 3.05% swapper [kernel.kallsyms] [k] intel_idle 2.63% fio [kernel.kallsyms] [k] get_page_from_freelist 2.33% fio [kernel.kallsyms] [k] asm_exc_nmi 2.26% kworker/u98:1+f [kernel.kallsyms] [k] __folio_start_writeback 1.40% fio [kernel.kallsyms] [k] __filemap_add_folio 1.37% fio [kernel.kallsyms] [k] lru_add_fn 1.35% fio [kernel.kallsyms] [k] xas_load 1.33% fio [kernel.kallsyms] [k] iomap_write_begin 1.31% fio [kernel.kallsyms] [k] xas_descend 1.19% kworker/u98:1+f [kernel.kallsyms] [k] folio_clear_dirty_for_io 1.07% fio [kernel.kallsyms] [k] folio_add_lru 1.01% fio [kernel.kallsyms] [k] __folio_mark_dirty 1.00% kworker/u98:1+f [kernel.kallsyms] [k] _raw_spin_lock_irqsave
Quoted 5 lines
Quoted 1 line
and 'top' show that 'kworker/u98:1' have over 80% CPU usage.
Can you provide an expanded callgraph profile for both the good and bad kernels showing the CPU used in the fio
write()path and the kworker-based writeback path?
I'm sorry that some detail guide for info gather of this test please.
the test machine here is already reserved.
Quoted 29 lines
[ The test machine I have that I could reproduce this sort of performance anomoly went bad a month ago, so I have no hardware available to me right now to reproduce this behaviour locally. Hence I'll need you to do the profiling I need to understand the regression for me. ]
Quoted 18 lines
Quoted 18 lines
I suspect that the likely culprit is mm-level changes - the page reclaim algorithm was completely replaced in 6.1 with a multi-generation LRU that will have different cache footprint behaviour in exactly this sort of "repeatedly over-write same files in a set that are significantly larger than memory" micro-benchmark.
i.e. these commits:
07017acb0601mm: multi-gen LRU: admin guided6c3af7d8a2bmm: multi-gen LRU: debugfs interface1332a809d95amm: multi-gen LRU: thrashing prevention354ed5974429mm: multi-gen LRU: kill switchf76c83378851mm: multi-gen LRU: optimize multiple memcgsbd74fdaea146mm: multi-gen LRU: support page table walks018ee47f1489mm: multi-gen LRU: exploit locality in rmapac35a4902374mm: multi-gen LRU: minimal implementationec1c86b25f4bmm: multi-gen LRU: groundworkQuoted 4 lines
Quoted 3 lines
If that's the case, I'd expect kernels up to 6.0 to demonstrate the same behaviour as 5.15, and 6.1+ to demonstrate the same behaviour as you've reported.
I tested 6.4.0-rc1. the performance become a little worse.
Quoted 7 lines
Thanks, that's as I expected.
WHich means that the interesting kernel versions to check now are a 6.0.x kernel, and then if it has the same perf as 5.15.x, then the commit before the multi-gen LRU was introduced vs the commit after the multi-gen LRU was introduced to see if that is the functionality that introduced the regression....
more performance test result:
linux 6.0.18
fio WRITE: bw=2565MiB/s (2689MB/s) linux 5.17.0 fio WRITE: bw=2602MiB/s (2729MB/s) linux 5.16.20 fio WRITE: bw=7666MiB/s (8039MB/s),
so it is a problem between 5.16.20 and 5.17.0?
Best Regards
Wang Yugui ([email protected])
2023/05/09
Quoted 17 lines · Wang Yugui
Quoted 17 lines · Wang Yugui
Quoted 16 lines
Quoted 16 lines · Wang Yugui
Quoted 15 lines
Quoted 15 lines
Hi,
I noticed a performance regression of xfs 6.1.27/6.1.23, with the compare to xfs 5.15.110.
It is yet not clear whether it is a problem of xfs or lvm2.
any guide to troubleshoot it?
test case:
disk: NVMe PCIe3 SSD *4
LVM: raid0 default strip size 64K.
fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30
-iodepth 1 -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=4 -directory=/mnt/test
.....
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 4 lines
Because you are testing buffered IO, you need to run perf across all CPUs and tasks, not just the fio process so that it captures the profile of memory reclaim and writeback that is being performed by the kernel.
'perf report' of all CPU.
Samples: 211K of event 'cycles', Event count (approx.): 56590727219Overhead Command Shared Object Symbol 16.29% fio [kernel.kallsyms] [k] rep_movs_alternative 3.38% kworker/u98:1+f [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 3.11% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 3.05% swapper [kernel.kallsyms] [k] intel_idle 2.63% fio [kernel.kallsyms] [k] get_page_from_freelist 2.33% fio [kernel.kallsyms] [k] asm_exc_nmi 2.26% kworker/u98:1+f [kernel.kallsyms] [k] __folio_start_writeback 1.40% fio [kernel.kallsyms] [k] __filemap_add_folio 1.37% fio [kernel.kallsyms] [k] lru_add_fn 1.35% fio [kernel.kallsyms] [k] xas_load 1.33% fio [kernel.kallsyms] [k] iomap_write_begin 1.31% fio [kernel.kallsyms] [k] xas_descend 1.19% kworker/u98:1+f [kernel.kallsyms] [k] folio_clear_dirty_for_io 1.07% fio [kernel.kallsyms] [k] folio_add_lru 1.01% fio [kernel.kallsyms] [k] __folio_mark_dirty 1.00% kworker/u98:1+f [kernel.kallsyms] [k] _raw_spin_lock_irqsave
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
and 'top' show that 'kworker/u98:1' have over 80% CPU usage.
Can you provide an expanded callgraph profile for both the good and bad kernels showing the CPU used in the fio
write()path and the kworker-based writeback path?I'm sorry that some detail guide for info gather of this test please.
'perf record -g' and 'perf report -g' should enable callgraph profiling and reporting. See the perf-record man page for '--callgraph' to make sure you have the right kernel config for this to work efficiently.
You can do quick snapshots in time via 'perf top -U -g' and then after a few seconds type 'E' then immediately type 'P' and the fully expanded callgraph profile will get written to a perf.hist.N file in the current working directory...
Quoted 12 lines
Quoted 9 lines
Quoted 1 line
I tested 6.4.0-rc1. the performance become a little worse.
Thanks, that's as I expected.
WHich means that the interesting kernel versions to check now are a 6.0.x kernel, and then if it has the same perf as 5.15.x, then the commit before the multi-gen LRU was introduced vs the commit after the multi-gen LRU was introduced to see if that is the functionality that introduced the regression....
more performance test result:
Quoted 8 lines
linux 6.0.18
fio WRITE: bw=2565MiB/s (2689MB/s) linux 5.17.0 fio WRITE: bw=2602MiB/s (2729MB/s) linux 5.16.20 fio WRITE: bw=7666MiB/s (8039MB/s),so it is a problem between 5.16.20 and 5.17.0?
Ok, that is further back in time than I expected. In terms of XFS, there are only two commits between 5.16..5.17 that might impact performance:
ebb7fb1557b1 ("xfs, iomap: limit individual ioend chain lengths in writeback")
and
6795801366da ("xfs: Support large folios")
To test whether ebb7fb1557b1 is the cause, go tofs/iomap/buffered-io.c and change:
-#define IOEND_BATCH_SIZE 4096
+#define IOEND_BATCH_SIZE 1048576
This will increase the IO submission chain lengths to at least 4GB from the 16MB bound that was placed on 5.17 and newer kernels.
To test whether 6795801366da is the cause, go to fs/xfs/xfs_icache.c and comment out both calls to mapping_set_large_folios(). This will ensure the page cache only instantiates single page folios the same as 5.16 would have.
If neither of them change behaviour, then I think you're going to need to do a bisect between 5.16..5.17 to find the commit that introduced the regression. I know kernel bisects are slow and painful, but it's exactly what I'd be doing right now if my performance test machine wasn't broken....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Hi,
Quoted 9 lines
'perf record -g' and 'perf report -g' should enable callgraph profiling and reporting. See the perf-record man page for '--callgraph' to make sure you have the right kernel config for this to work efficiently.
You can do quick snapshots in time via 'perf top -U -g' and then after a few seconds type 'E' then immediately type 'P' and the fully expanded callgraph profile will get written to a perf.hist.N file in the current working directory...
'perf report -g ' of BAD kernel 6.1.y
Samples: 439K of event 'cycles', Event count (approx.): 148701386235
Children Self Command Shared Object Symbol + 61.82% 0.01% fio [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwfra + 61.81% 0.01% fio [kernel.kallsyms] [k] do_syscall_64 + 61.71% 0.00% fio libpthread-2.17.so [.] 0x00007f4e7a40e6fd + 61.66% 0.00% fio [kernel.kallsyms] [k] ksys_write + 61.64% 0.02% fio [kernel.kallsyms] [k] vfs_write + 61.60% 0.02% fio [kernel.kallsyms] [k] xfs_file_buffered_write + 61.20% 0.56% fio [kernel.kallsyms] [k] iomap_file_buffered_write + 25.25% 1.44% fio [kernel.kallsyms] [k] iomap_write_begin + 23.19% 0.75% fio [kernel.kallsyms] [k] __filemap_get_folio + 21.57% 0.00% kworker/u98:4+f [kernel.kallsyms] [k] ret_from_fork + 21.57% 0.00% kworker/u98:4+f [kernel.kallsyms] [k] kthread + 21.57% 0.00% kworker/u98:4+f [kernel.kallsyms] [k] worker_thread + 21.57% 0.00% kworker/u98:4+f [kernel.kallsyms] [k] process_one_work + 21.57% 0.00% kworker/u98:4+f [kernel.kallsyms] [k] wb_workfn + 21.57% 0.00% kworker/u98:4+f [kernel.kallsyms] [k] wb_writeback + 21.57% 0.00% kworker/u98:4+f [kernel.kallsyms] [k] __writeback_inodes_wb + 21.57% 0.00% kworker/u98:4+f [kernel.kallsyms] [k] writeback_sb_inodes + 21.57% 0.00% kworker/u98:4+f [kernel.kallsyms] [k] __writeback_single_inode + 21.57% 0.00% kworker/u98:4+f [kernel.kallsyms] [k] do_writepages + 21.57% 0.00% kworker/u98:4+f [kernel.kallsyms] [k] xfs_vm_writepages + 21.55% 0.00% kworker/u98:4+f [kernel.kallsyms] [k] iomap_writepages + 21.55% 0.85% kworker/u98:4+f [kernel.kallsyms] [k] write_cache_pages + 20.23% 0.22% fio [kernel.kallsyms] [k] copy_page_from_iter_atomic + 19.99% 0.04% fio [kernel.kallsyms] [k] copyin + 19.94% 19.77% fio [kernel.kallsyms] [k] copy_user_enhanced_fast_stri + 16.47% 0.00% fio [unknown] [k] 0x00000000024803f0 + 16.47% 0.00% fio [unknown] [k] 0x0000000002480440 + 16.47% 0.00% fio [unknown] [k] 0x00000000024804b0 + 16.47% 0.00% fio [unknown] [k] 0x0000000002480520 + 16.47% 0.00% fio [unknown] [k] 0x0000000002480590 + 16.47% 0.00% fio [unknown] [k] 0x0000000002480670 + 16.47% 0.00% fio [unknown] [k] 0x00000000024806e0 + 16.47% 0.00% fio [unknown] [k] 0x0000000002480750 + 16.47% 0.00% fio [unknown] [k] 0x00000000024807c0 + 16.47% 0.00% fio [unknown] [k] 0x0000000002480830 + 16.47% 0.00% fio [unknown] [k] 0x00000000024808a0 + 16.47% 0.00% fio [unknown] [k] 0x0000000002480910 + 16.47% 0.00% fio [unknown] [k] 0x00007f4e21070430 + 15.73% 0.00% fio [unknown] [k] 0x0000000002480290 + 15.73% 0.00% fio [unknown] [k] 0x000000000247f790 + 15.73% 0.00% fio [unknown] [k] 0x000000000247fdf0 + 15.73% 0.00% fio [unknown] [k] 0x000000000247fe60 + 15.73% 0.00% fio [unknown] [k] 0x000000000247fed0 + 15.73% 0.00% fio [unknown] [k] 0x000000000247ff40 + 15.73% 0.00% fio [unknown] [k] 0x000000000247ffb0 + 15.73% 0.00% fio [unknown] [k] 0x0000000002480020 + 15.73% 0.00% fio [unknown] [k] 0x0000000002480090 + 15.73% 0.00% fio [unknown] [k] 0x0000000002480180 + 15.73% 0.00% fio [unknown] [k] 0x00000000024801b0 + 15.73% 0.00% fio [unknown] [k] 0x0000000002480220 + 15.73% 0.00% fio [unknown] [k] 0x00007f4e2102ba18 + 15.07% 0.74% kworker/u98:4+f [kernel.kallsyms] [k] iomap_writepage_map + 15.01% 0.00% fio [unknown] [k] 0x0000000002480a70
'perf report -g ' of GOOD kernel 5.15.y
Samples: 335K of event 'cycles', Event count (approx.): 229260446285
Children Self Command Shared Object Symbol + 66.35% 0.01% fio [kernel.kallsyms] [k] entry_SYSCALL_64_aft + 66.34% 0.01% fio [kernel.kallsyms] [k] do_syscall_64 + 65.49% 0.00% fio libpthread-2.17.so [.] 0x00007ff968de56fd + 65.45% 0.00% fio [kernel.kallsyms] [k] ksys_write + 65.44% 0.02% fio [kernel.kallsyms] [k] vfs_write + 65.41% 0.00% fio [kernel.kallsyms] [k] new_sync_write + 65.40% 0.03% fio [kernel.kallsyms] [k] xfs_file_buffered_wr + 65.13% 0.53% fio [kernel.kallsyms] [k] iomap_file_buffered_ + 27.17% 0.36% fio [kernel.kallsyms] [k] copy_page_from_iter_ + 26.76% 0.05% fio [kernel.kallsyms] [k] copyin + 26.73% 26.52% fio [kernel.kallsyms] [k] copy_user_enhanced_f + 23.12% 1.05% fio [kernel.kallsyms] [k] iomap_write_begin + 21.59% 0.34% fio [kernel.kallsyms] [k] grab_cache_page_writ + 21.13% 0.65% fio [kernel.kallsyms] [k] pagecache_get_page + 18.73% 0.00% kworker/u97:11+ [kernel.kallsyms] [k] ret_from_fork + 18.73% 0.00% kworker/u97:11+ [kernel.kallsyms] [k] kthread + 18.73% 0.00% kworker/u97:11+ [kernel.kallsyms] [k] worker_thread + 18.73% 0.00% kworker/u97:11+ [kernel.kallsyms] [k] process_one_work + 18.73% 0.00% kworker/u97:11+ [kernel.kallsyms] [k] wb_workfn + 18.73% 0.00% kworker/u97:11+ [kernel.kallsyms] [k] wb_writeback + 18.73% 0.00% kworker/u97:11+ [kernel.kallsyms] [k] __writeback_inodes_w + 18.73% 0.00% kworker/u97:11+ [kernel.kallsyms] [k] writeback_sb_inodes + 18.73% 0.00% kworker/u97:11+ [kernel.kallsyms] [k] __writeback_single_i + 18.73% 0.00% kworker/u97:11+ [kernel.kallsyms] [k] do_writepages + 18.73% 0.00% kworker/u97:11+ [kernel.kallsyms] [k] xfs_vm_writepages + 18.66% 0.00% kworker/u97:11+ [kernel.kallsyms] [k] iomap_writepages + 18.66% 0.48% kworker/u97:11+ [kernel.kallsyms] [k] write_cache_pages + 16.75% 0.00% fio [unknown] [k] 0x0000000001e613f0 + 16.75% 0.00% fio [unknown] [k] 0x0000000001e61440 + 16.75% 0.00% fio [unknown] [k] 0x0000000001e614b0 + 16.75% 0.00% fio [unknown] [k] 0x0000000001e61520 + 16.75% 0.00% fio [unknown] [k] 0x0000000001e61590 + 16.75% 0.00% fio [unknown] [k] 0x0000000001e61670 + 16.75% 0.00% fio [unknown] [k] 0x0000000001e616e0 + 16.75% 0.00% fio [unknown] [k] 0x0000000001e61750 + 16.75% 0.00% fio [unknown] [k] 0x0000000001e617c0 + 16.75% 0.00% fio [unknown] [k] 0x0000000001e61830 + 16.75% 0.00% fio [unknown] [k] 0x0000000001e618a0 + 16.75% 0.00% fio [unknown] [k] 0x0000000001e61910 + 16.75% 0.00% fio [unknown] [k] 0x00007ff9151e0430 + 16.70% 0.00% fio [unknown] [k] 0x0000000001e60790 + 16.70% 0.00% fio [unknown] [k] 0x0000000001e60df0 + 16.70% 0.00% fio [unknown] [k] 0x0000000001e60e60 + 16.70% 0.00% fio [unknown] [k] 0x0000000001e60ed0 + 16.70% 0.00% fio [unknown] [k] 0x0000000001e60f40 + 16.70% 0.00% fio [unknown] [k] 0x0000000001e60fb0 + 16.70% 0.00% fio [unknown] [k] 0x0000000001e61020 + 16.70% 0.00% fio [unknown] [k] 0x0000000001e61090 + 16.70% 0.00% fio [unknown] [k] 0x0000000001e61180 + 16.70% 0.00% fio [unknown] [k] 0x0000000001e611b0 + 16.70% 0.00% fio [unknown] [k] 0x0000000001e61220 + 16.70% 0.00% fio [unknown] [k] 0x0000000001e61290 + 16.70% 0.00% fio [unknown] [k] 0x00007ff91519ba18
Quoted 18 lines
Ok, that is further back in time than I expected. In terms of XFS, there are only two commits between 5.16..5.17 that might impact performance:
ebb7fb1557b1("xfs, iomap: limit individual ioend chain lengths in writeback")and
6795801366da("xfs: Support large folios")To test whether
ebb7fb1557b1is the cause, go tofs/iomap/buffered-io.cand change:(quoted diff) +1
-1-#define IOEND_BATCH_SIZE 4096 +#define IOEND_BATCH_SIZE 1048576This will increase the IO submission chain lengths to at least 4GB from the 16MB bound that was placed on 5.17 and newer kernels.
Quoted 4 lines
To test whether
6795801366dais the cause, go tofs/xfs/xfs_icache.cand comment out both calls tomapping_set_large_folios(). This will ensure the page cache only instantiates single page folios the same as 5.16 would have.
6.1.x with 'mapping_set_large_folios remove' and 'IOEND_BATCH_SIZE=1048576'
fio WRITE: bw=6451MiB/s (6764MB/s)
still performance regression when compare to linux 5.16.20
fio WRITE: bw=7666MiB/s (8039MB/s),
but the performance regression is not too big, then difficult to bisect. We noticed samle level performance regression on btrfs too. so maby some problem of some code that is used by both btrfs and xfs such as iomap and mm/folio.
6.1.x with 'mapping_set_large_folios remove' only'
fio WRITE: bw=2676MiB/s (2806MB/s)
6.1.x with 'IOEND_BATCH_SIZE=1048576' only'
fio WRITE: bw=5092MiB/s (5339MB/s), fio WRITE: bw=6076MiB/s (6371MB/s)
maybe we need more fix or ' ebb7fb1557b1 ("xfs, iomap: limit individual ioend chain lengths in writeback")'.
Best Regards
Wang Yugui ([email protected])
2023/05/10
Quoted 13 lines
If neither of them change behaviour, then I think you're going to need to do a bisect between 5.16..5.17 to find the commit that introduced the regression. I know kernel bisects are slow and painful, but it's exactly what I'd be doing right now if my performance test machine wasn't broken....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 26 lines · Wang Yugui
Quoted 18 lines
Ok, that is further back in time than I expected. In terms of XFS, there are only two commits between 5.16..5.17 that might impact performance:
ebb7fb1557b1("xfs, iomap: limit individual ioend chain lengths in writeback")and
6795801366da("xfs: Support large folios")To test whether
ebb7fb1557b1is the cause, go tofs/iomap/buffered-io.cand change:(quoted diff) +1
-1-#define IOEND_BATCH_SIZE 4096 +#define IOEND_BATCH_SIZE 1048576This will increase the IO submission chain lengths to at least 4GB from the 16MB bound that was placed on 5.17 and newer kernels.
Quoted 4 lines
To test whether
6795801366dais the cause, go tofs/xfs/xfs_icache.cand comment out both calls tomapping_set_large_folios(). This will ensure the page cache only instantiates single page folios the same as 5.16 would have.6.1.x with '
mapping_set_large_foliosremove' and 'IOEND_BATCH_SIZE=1048576'fio WRITE: bw=6451MiB/s (6764MB/s)
Quoted 7 lines
still performance regression when compare to linux 5.16.20
fio WRITE: bw=7666MiB/s (8039MB/s),but the performance regression is not too big, then difficult to bisect. We noticed samle level performance regression on btrfs too. so maby some problem of some code that is used by both btrfs and xfs such as iomap and mm/folio.
Yup, that's quite possibly something like the multi-gen LRU changes, but that's not the regression we need to find. :/
Quoted 9 lines
6.1.x with '
mapping_set_large_foliosremove' only'fio WRITE: bw=2676MiB/s (2806MB/s)6.1.x with 'IOEND_BATCH_SIZE=1048576' only'
fio WRITE: bw=5092MiB/s (5339MB/s), fio WRITE: bw=6076MiB/s (6371MB/s)maybe we need more fix or '
ebb7fb1557b1("xfs, iomap: limit individual ioend chain lengths in writeback")'.
OK, can you re-run the two 6.1.x kernels above (the slow and the fast) and record the output of iostat -dxm 1 whilst the fio test is running? I want to see what the overall differences in the IO load on the devices are between the two runs. This will tell us how the IO sizes and queue depths change between the two kernels, etc.
Right now I'm suspecting a contention interaction between write(), do_writepages() and folio_end_writeback()...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Hi,
Quoted 48 lines · Wang Yugui
Quoted 26 lines
Quoted 18 lines
Ok, that is further back in time than I expected. In terms of XFS, there are only two commits between 5.16..5.17 that might impact performance:
ebb7fb1557b1("xfs, iomap: limit individual ioend chain lengths in writeback")and
6795801366da("xfs: Support large folios")To test whether
ebb7fb1557b1is the cause, go tofs/iomap/buffered-io.cand change:(quoted diff) +1
-1-#define IOEND_BATCH_SIZE 4096 +#define IOEND_BATCH_SIZE 1048576This will increase the IO submission chain lengths to at least 4GB from the 16MB bound that was placed on 5.17 and newer kernels.
Quoted 4 lines
To test whether
6795801366dais the cause, go tofs/xfs/xfs_icache.cand comment out both calls tomapping_set_large_folios(). This will ensure the page cache only instantiates single page folios the same as 5.16 would have.6.1.x with '
mapping_set_large_foliosremove' and 'IOEND_BATCH_SIZE=1048576'fio WRITE: bw=6451MiB/s (6764MB/s)Quoted 7 lines
still performance regression when compare to linux 5.16.20
fio WRITE: bw=7666MiB/s (8039MB/s),but the performance regression is not too big, then difficult to bisect. We noticed samle level performance regression on btrfs too. so maby some problem of some code that is used by both btrfs and xfs such as iomap and mm/folio.
Yup, that's quite possibly something like the multi-gen LRU changes, but that's not the regression we need to find. :/
Quoted 9 lines
6.1.x with '
mapping_set_large_foliosremove' only'fio WRITE: bw=2676MiB/s (2806MB/s)6.1.x with 'IOEND_BATCH_SIZE=1048576' only'
fio WRITE: bw=5092MiB/s (5339MB/s), fio WRITE: bw=6076MiB/s (6371MB/s)maybe we need more fix or '
ebb7fb1557b1("xfs, iomap: limit individual ioend chain lengths in writeback")'.
Quoted 6 lines
OK, can you re-run the two 6.1.x kernels above (the slow and the fast) and record the output of
iostat -dxm 1whilst the fio test is running? I want to see what the overall differences in the IO load on the devices are between the two runs. This will tell us how the IO sizes and queue depths change between the two kernels, etc.
iostat -dxm 1 result saved in attachment file.
good.txt good performance
bad.txt bad performance
Best Regards
Wang Yugui ([email protected])
2023/05/10
Quoted 10 lines
Right now I'm suspecting a contention interaction between
write(),do_writepages()andfolio_end_writeback()...Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
[root@T620 ~]# iostat -dxm 1
Linux 6.1.28-0.6.el7.x86_64 (T620) 05/10/2023 _x86_64_ (40 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 203.46 0.17 20.83 0.00 13.92 1357.81 0.21 9.62 0.09 9.70 0.61 1.29 nvme2c2n1 0.00 203.40 0.18 20.87 0.00 13.93 1355.38 0.09 4.37 0.08 4.40 0.27 0.57 nvme1n1 0.00 203.40 0.18 20.89 0.00 13.93 1353.59 0.08 3.53 0.03 3.56 0.29 0.61 nvme3n1 0.00 203.61 0.21 20.87 0.00 13.92 1352.33 0.15 7.20 0.05 7.27 0.38 0.79 sda 0.00 0.02 0.85 0.13 0.03 0.00 60.13 0.00 0.72 0.61 1.42 0.78 0.08 dm-0 0.00 0.00 0.05 42.88 0.00 2.67 127.43 1.04 24.19 0.07 24.22 0.04 0.18 dm-1 0.00 0.00 0.03 42.77 0.00 2.67 127.81 1.37 31.79 0.11 31.81 0.09 0.40 dm-2 0.00 0.00 0.03 42.77 0.00 2.67 127.82 0.89 20.79 0.09 20.80 0.03 0.12 dm-3 0.00 0.00 0.03 42.77 0.00 2.67 127.82 0.70 16.30 0.02 16.31 0.03 0.14 dm-4 0.00 0.00 0.14 171.18 0.00 10.68 127.72 3.99 23.27 0.07 23.29 0.02 0.41
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 25582.00 0.00 1368.00 0.00 1705.69 2553.55 15.84 11.58 0.00 11.58 0.62 85.00 nvme2c2n1 0.00 25583.00 0.00 1347.00 0.00 1680.68 2555.33 5.53 4.10 0.00 4.10 0.45 60.50 nvme1n1 0.00 25590.00 0.00 1348.00 0.00 1680.75 2553.54 3.85 2.86 0.00 2.86 0.45 61.10 nvme3n1 0.00 25585.00 0.00 1371.00 0.00 1709.50 2553.65 14.62 10.66 0.00 10.66 0.65 89.00 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 26940.00 0.00 1682.44 127.90 283.90 10.54 0.00 10.54 0.03 89.20 dm-1 0.00 0.00 0.00 26952.00 0.00 1682.57 127.85 308.07 11.43 0.00 11.43 0.03 85.30 dm-2 0.00 0.00 0.00 26948.00 0.00 1682.62 127.88 102.20 3.79 0.00 3.79 0.02 62.30 dm-3 0.00 0.00 0.00 26951.00 0.00 1682.77 127.87 68.75 2.55 0.00 2.55 0.02 62.80 dm-4 0.00 0.00 0.00 107893.00 0.00 6736.76 127.88 762.46 7.07 0.00 7.07 0.01 91.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 26957.00 0.00 1421.00 0.00 1771.55 2553.23 14.10 9.92 0.00 9.92 0.63 89.20 nvme2c2n1 0.00 26955.00 0.00 1421.00 0.00 1771.60 2553.29 5.67 3.99 0.00 3.99 0.44 63.20 nvme1n1 0.00 26953.00 0.00 1421.00 0.00 1771.60 2553.30 3.80 2.67 0.00 2.67 0.44 63.20 nvme3n1 0.00 26955.00 0.00 1421.00 0.00 1771.56 2553.24 15.01 10.56 0.00 10.56 0.64 91.00 sda 0.00 0.00 0.00 5.00 0.00 0.04 18.20 0.01 1.20 0.00 1.20 0.60 0.30 dm-0 0.00 0.00 0.00 28370.00 0.00 1771.02 127.85 292.33 10.30 0.00 10.30 0.03 91.40 dm-1 0.00 0.00 0.00 28357.00 0.00 1770.87 127.90 273.60 9.65 0.00 9.65 0.03 89.30 dm-2 0.00 0.00 0.00 28359.00 0.00 1770.85 127.89 105.64 3.73 0.00 3.73 0.02 64.50 dm-3 0.00 0.00 0.00 28361.00 0.00 1770.73 127.87 68.33 2.41 0.00 2.41 0.02 64.40 dm-4 0.00 0.00 0.00 113345.00 0.00 7077.08 127.87 739.98 6.53 0.00 6.53 0.01 94.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 29349.00 0.00 1499.00 0.00 1869.08 2553.62 29.05 19.38 0.00 19.38 0.60 90.60 nvme2c2n1 0.00 29355.00 0.00 1529.00 0.00 1906.56 2553.72 6.26 4.10 0.00 4.10 0.45 68.60 nvme1n1 0.00 29352.00 0.00 1537.00 0.00 1915.27 2552.04 4.21 2.74 0.00 2.74 0.45 68.50 nvme3n1 0.00 29353.00 0.00 1512.00 0.00 1884.06 2551.96 24.56 16.24 0.00 16.24 0.63 94.90 sda 0.00 0.00 0.00 1.00 0.00 0.00 8.00 0.00 0.00 0.00 0.00 1.00 0.10 dm-0 0.00 0.00 0.00 30902.00 0.00 1929.52 127.88 481.12 15.57 0.00 15.57 0.03 95.10 dm-1 0.00 0.00 0.00 30897.00 0.00 1929.46 127.89 571.80 18.51 0.00 18.51 0.03 90.40 dm-2 0.00 0.00 0.00 30903.00 0.00 1929.44 127.87 116.15 3.76 0.00 3.76 0.02 69.30 dm-3 0.00 0.00 0.00 30901.00 0.00 1929.46 127.88 74.80 2.42 0.00 2.42 0.02 69.20 dm-4 0.00 0.00 0.00 123603.00 0.00 7717.87 127.88 1243.94 10.06 0.00 10.06 0.01 96.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 25950.00 0.00 1392.00 0.00 1734.20 2551.47 23.06 16.56 0.00 16.56 0.60 83.20 nvme2c2n1 0.00 25954.00 0.00 1387.00 0.00 1729.16 2553.22 5.69 4.10 0.00 4.10 0.44 61.40 nvme1n1 0.00 25947.00 0.00 1380.00 0.00 1720.43 2553.22 4.02 2.91 0.00 2.91 0.45 61.80 nvme3n1 0.00 25954.00 0.00 1378.00 0.00 1717.91 2553.17 18.68 13.55 0.00 13.55 0.63 87.20 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 27322.00 0.00 1705.69 127.85 364.35 13.34 0.00 13.34 0.03 87.40 dm-1 0.00 0.00 0.00 27319.00 0.00 1705.76 127.87 451.39 16.52 0.00 16.52 0.03 83.50 dm-2 0.00 0.00 0.00 27322.00 0.00 1705.72 127.86 104.82 3.84 0.00 3.84 0.02 62.40 dm-3 0.00 0.00 0.00 27315.00 0.00 1705.68 127.89 71.30 2.61 0.00 2.61 0.02 62.50 dm-4 0.00 0.00 0.00 109278.00 0.00 6822.85 127.87 992.64 9.08 0.00 9.08 0.01 89.60
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 26937.00 0.00 1416.00 0.00 1765.16 2552.99 35.40 25.00 0.00 25.00 0.63 89.90 nvme2c2n1 0.00 26944.00 0.00 1420.00 0.00 1770.19 2553.06 5.75 4.05 0.00 4.05 0.45 63.80 nvme1n1 0.00 26938.00 0.00 1420.00 0.00 1770.18 2553.05 3.86 2.72 0.00 2.72 0.45 63.30 nvme3n1 0.00 26936.00 0.00 1448.00 0.00 1803.89 2551.35 6.91 4.77 0.00 4.77 0.52 75.40 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 28357.00 0.00 1770.76 127.89 129.60 4.57 0.00 4.57 0.03 76.20 dm-1 0.00 0.00 0.00 28357.00 0.00 1770.77 127.89 698.41 24.63 0.00 24.63 0.03 89.90 dm-2 0.00 0.00 0.00 28364.00 0.00 1770.81 127.86 106.04 3.74 0.00 3.74 0.02 65.30 dm-3 0.00 0.00 0.00 28358.00 0.00 1770.81 127.89 68.45 2.41 0.00 2.41 0.02 64.80 dm-4 0.00 0.00 0.00 113436.00 0.00 7083.15 127.88 1002.26 8.84 0.00 8.84 0.01 93.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 24627.00 0.00 1318.00 0.00 1645.44 2556.80 11.61 8.81 0.00 8.81 0.63 83.40 nvme2c2n1 0.00 24631.00 0.00 1298.00 0.00 1619.25 2554.87 5.13 3.95 0.00 3.95 0.44 57.60 nvme1n1 0.00 24634.00 0.00 1298.00 0.00 1619.25 2554.88 3.47 2.68 0.00 2.68 0.44 56.90 nvme3n1 0.00 24634.00 0.00 1298.00 0.00 1619.23 2554.84 4.19 3.23 0.00 3.23 0.52 67.70 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 25932.00 0.00 1619.05 127.87 75.80 2.92 0.00 2.92 0.03 69.40 dm-1 0.00 0.00 0.00 25925.00 0.00 1619.02 127.90 224.33 8.65 0.00 8.65 0.03 84.10 dm-2 0.00 0.00 0.00 25929.00 0.00 1619.06 127.88 94.71 3.65 0.00 3.65 0.02 59.30 dm-3 0.00 0.00 0.00 25932.00 0.00 1619.07 127.87 61.35 2.37 0.00 2.37 0.02 58.90 dm-4 0.00 0.00 0.00 103718.00 0.00 6476.21 127.88 455.83 4.39 0.00 4.39 0.01 84.90
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 27534.00 0.00 1462.00 0.00 1821.66 2551.82 25.20 17.24 0.00 17.24 0.60 88.40 nvme2c2n1 0.00 27537.00 0.00 1452.00 0.00 1810.38 2553.48 6.29 4.33 0.00 4.33 0.44 64.00 nvme1n1 0.00 27535.00 0.00 1452.00 0.00 1810.38 2553.48 4.08 2.81 0.00 2.81 0.44 63.90 nvme3n1 0.00 27532.00 0.00 1452.00 0.00 1810.40 2553.51 6.85 4.72 0.00 4.72 0.52 75.00 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 28992.00 0.00 1810.63 127.90 128.65 4.44 0.00 4.44 0.03 75.50 dm-1 0.00 0.00 0.00 28998.00 0.00 1810.60 127.87 494.88 17.07 0.00 17.07 0.03 88.60 dm-2 0.00 0.00 0.00 29001.00 0.00 1810.56 127.86 117.12 4.04 0.00 4.04 0.02 65.00 dm-3 0.00 0.00 0.00 29000.00 0.00 1810.56 127.86 73.24 2.53 0.00 2.53 0.02 64.90 dm-4 0.00 0.00 0.00 116103.00 0.00 7249.36 127.88 813.95 7.01 0.00 7.01 0.01 90.80
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 26478.00 0.00 1391.00 0.00 1735.71 2555.53 22.99 16.53 0.00 16.53 0.63 87.00 nvme2c2n1 0.00 26471.00 0.00 1395.00 0.00 1740.75 2555.60 5.75 4.12 0.00 4.12 0.44 61.60 nvme1n1 0.00 26477.00 0.00 1395.00 0.00 1740.75 2555.60 3.82 2.74 0.00 2.74 0.44 61.20 nvme3n1 0.00 26480.00 0.00 1395.00 0.00 1740.78 2555.64 6.88 4.93 0.00 4.93 0.52 72.60 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 27867.00 0.00 1739.59 127.85 129.73 4.66 0.00 4.66 0.03 73.40 dm-1 0.00 0.00 0.00 27861.00 0.00 1739.65 127.88 452.05 16.23 0.00 16.23 0.03 87.40 dm-2 0.00 0.00 0.00 27854.00 0.00 1739.65 127.91 107.19 3.85 0.00 3.85 0.02 63.00 dm-3 0.00 0.00 0.00 27859.00 0.00 1739.62 127.89 68.79 2.47 0.00 2.47 0.02 62.70 dm-4 0.00 0.00 0.00 111329.00 0.00 6951.52 127.88 757.81 6.81 0.00 6.81 0.01 89.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 26664.00 0.00 1399.00 0.00 1741.75 2549.75 34.17 24.42 0.00 24.42 0.62 87.40 nvme2c2n1 0.00 26664.00 0.00 1395.00 0.00 1736.70 2549.64 5.63 4.03 0.00 4.03 0.45 63.10 nvme1n1 0.00 26670.00 0.00 1400.00 0.00 1742.94 2549.67 3.90 2.78 0.00 2.78 0.45 63.40 nvme3n1 0.00 26664.00 0.00 1399.00 0.00 1741.72 2549.71 7.68 5.49 0.00 5.49 0.52 72.40 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 28071.00 0.00 1752.78 127.88 144.22 5.14 0.00 5.14 0.03 73.40 dm-1 0.00 0.00 0.00 28071.00 0.00 1752.81 127.88 673.15 23.98 0.00 23.98 0.03 87.90 dm-2 0.00 0.00 0.00 28071.00 0.00 1752.76 127.88 102.83 3.66 0.00 3.66 0.02 64.10 dm-3 0.00 0.00 0.00 28077.00 0.00 1752.75 127.85 67.94 2.42 0.00 2.42 0.02 64.90 dm-4 0.00 0.00 0.00 112290.00 0.00 7011.10 127.87 988.33 8.80 0.00 8.80 0.01 90.20
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 28862.00 0.00 1524.00 0.00 1900.19 2553.53 38.41 25.20 0.00 25.20 0.61 92.40 nvme2c2n1 0.00 28857.00 0.00 1525.00 0.00 1901.43 2553.53 6.28 4.12 0.00 4.12 0.45 68.50 nvme1n1 0.00 28853.00 0.00 1524.00 0.00 1900.17 2553.51 4.21 2.76 0.00 2.76 0.45 68.40 nvme3n1 0.00 28847.00 0.00 1524.00 0.00 1900.12 2553.45 9.62 6.31 0.00 6.31 0.52 79.70 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 30369.00 0.00 1896.88 127.92 183.75 6.05 0.00 6.05 0.03 80.30 dm-1 0.00 0.00 0.00 30384.00 0.00 1896.86 127.86 758.64 24.97 0.00 24.97 0.03 92.60 dm-2 0.00 0.00 0.00 30379.00 0.00 1896.91 127.88 117.22 3.86 0.00 3.86 0.02 69.80 dm-3 0.00 0.00 0.00 30375.00 0.00 1896.92 127.90 75.70 2.49 0.00 2.49 0.02 69.10 dm-4 0.00 0.00 0.00 121507.00 0.00 7587.55 127.89 1135.30 9.34 0.00 9.34 0.01 94.20
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 26847.00 0.00 1411.00 0.00 1757.56 2551.02 19.05 13.50 0.00 13.50 0.61 86.00 nvme2c2n1 0.00 26836.00 0.00 1425.00 0.00 1775.09 2551.14 5.94 4.17 0.00 4.17 0.44 63.00 nvme1n1 0.00 26847.00 0.00 1421.00 0.00 1770.14 2551.20 3.93 2.77 0.00 2.77 0.44 62.40 nvme3n1 0.00 26843.00 0.00 1422.00 0.00 1771.41 2551.23 5.43 3.82 0.00 3.82 0.52 74.00 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 28259.00 0.00 1764.54 127.88 98.97 3.50 0.00 3.50 0.03 74.60 dm-1 0.00 0.00 0.00 28263.00 0.00 1764.42 127.85 370.40 13.11 0.00 13.11 0.03 86.30 dm-2 0.00 0.00 0.00 28252.00 0.00 1764.46 127.91 108.94 3.86 0.00 3.86 0.02 64.20 dm-3 0.00 0.00 0.00 28263.00 0.00 1764.52 127.86 69.18 2.45 0.00 2.45 0.02 63.60 dm-4 0.00 0.00 0.00 113037.00 0.00 7057.93 127.88 647.52 5.73 0.00 5.73 0.01 88.30
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 24345.00 0.00 1300.00 0.00 1619.89 2551.95 12.49 9.61 0.00 9.61 0.63 82.20 nvme2c2n1 0.00 24347.00 0.00 1285.00 0.00 1601.16 2551.89 5.07 3.94 0.00 3.94 0.45 58.20 nvme1n1 0.00 24349.00 0.00 1285.00 0.00 1601.19 2551.93 3.46 2.69 0.00 2.69 0.46 58.60 nvme3n1 0.00 24350.00 0.00 1285.00 0.00 1601.23 2552.01 4.56 3.55 0.00 3.55 0.53 68.00 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 25635.00 0.00 1600.61 127.87 81.69 3.19 0.00 3.19 0.03 71.80 dm-1 0.00 0.00 0.00 25630.00 0.00 1600.55 127.89 240.19 9.37 0.00 9.37 0.03 85.10 dm-2 0.00 0.00 0.00 25632.00 0.00 1600.54 127.88 91.86 3.58 0.00 3.58 0.02 62.90 dm-3 0.00 0.00 0.00 25634.00 0.00 1600.56 127.88 59.96 2.34 0.00 2.34 0.02 63.40 dm-4 0.00 0.00 0.00 102531.00 0.00 6402.26 127.88 473.77 4.62 0.00 4.62 0.01 86.70
^C
[root@T620 ~]#
[root@T620 ~]# iostat -dxm 1
Linux 6.1.28-0.5.el7.x86_64 (T620) 05/10/2023 _x86_64_ (40 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 779.13 1.45 62.83 0.03 52.31 1667.62 0.36 3.89 0.07 3.97 1.17 7.50 nvme2c2n1 0.00 775.15 1.49 62.80 0.03 52.33 1667.93 0.07 1.06 0.09 1.09 0.47 3.04 nvme1n1 0.00 775.15 1.54 62.86 0.03 52.33 1665.16 0.07 0.92 0.03 0.94 0.52 3.33 nvme3n1 0.00 777.00 1.77 62.83 0.03 52.33 1659.94 0.11 1.53 0.05 1.57 0.60 3.86 sda 0.02 0.10 12.71 0.76 0.43 0.02 67.86 0.00 0.32 0.26 1.45 0.40 0.55 dm-0 0.00 0.00 0.85 839.78 0.01 52.33 127.50 1.41 1.66 0.05 1.66 0.05 3.86 dm-1 0.00 0.00 0.45 841.92 0.01 52.33 127.24 2.74 3.13 0.09 3.13 0.09 7.50 dm-2 0.00 0.00 0.48 837.90 0.00 52.33 127.84 0.94 1.12 0.10 1.12 0.04 3.05 dm-3 0.00 0.00 0.53 837.95 0.00 52.33 127.83 0.76 0.90 0.01 0.90 0.04 3.34 dm-4 0.00 0.00 2.32 3357.55 0.02 209.32 127.60 5.83 1.70 0.06 1.70 0.02 7.72
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 10015.00 0.00 542.00 0.00 663.12 2505.68 1.71 3.15 0.00 3.15 0.96 51.90 nvme2c2n1 0.00 10015.00 0.00 539.00 0.00 659.34 2505.25 1.20 2.22 0.00 2.22 0.76 40.90 nvme1n1 0.00 10014.00 0.00 539.00 0.00 659.31 2505.14 1.11 2.07 0.00 2.07 0.74 39.70 nvme3n1 0.00 10014.00 0.00 539.00 0.00 659.31 2505.14 1.37 2.54 0.00 2.54 0.94 50.70 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 10553.00 0.00 659.56 128.00 18.83 1.78 0.00 1.78 0.05 52.10 dm-1 0.00 0.00 0.00 10554.00 0.00 659.62 128.00 25.53 2.42 0.00 2.42 0.05 53.30 dm-2 0.00 0.00 0.00 10554.00 0.00 659.59 127.99 15.50 1.47 0.00 1.47 0.04 42.80 dm-3 0.00 0.00 0.00 10553.00 0.00 659.56 128.00 13.93 1.32 0.00 1.32 0.04 41.50 dm-4 0.00 0.00 0.00 42214.00 0.00 2638.34 128.00 73.85 1.75 0.00 1.75 0.01 54.70
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 9258.00 0.00 502.00 0.00 610.50 2490.65 1.58 3.15 0.00 3.15 0.96 48.00 nvme2c2n1 0.00 9258.00 0.00 502.00 0.00 610.50 2490.65 1.12 2.24 0.00 2.24 0.75 37.60 nvme1n1 0.00 9258.00 0.00 502.00 0.00 610.50 2490.65 1.05 2.09 0.00 2.09 0.74 37.10 nvme3n1 0.00 9258.00 0.00 503.00 0.00 610.51 2485.73 1.27 2.53 0.00 2.53 0.91 45.60 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 9761.00 0.00 610.01 127.99 16.79 1.72 0.00 1.72 0.05 48.30 dm-1 0.00 0.00 0.00 9760.00 0.00 610.00 128.00 22.74 2.33 0.00 2.33 0.05 50.40 dm-2 0.00 0.00 0.00 9760.00 0.00 610.00 128.00 13.89 1.42 0.00 1.42 0.04 40.40 dm-3 0.00 0.00 0.00 9760.00 0.00 610.00 128.00 12.56 1.29 0.00 1.29 0.04 40.00 dm-4 0.00 0.00 0.00 39041.00 0.00 2440.01 128.00 66.05 1.69 0.00 1.69 0.01 51.30
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 7498.00 0.00 407.00 0.00 494.00 2485.78 1.30 3.19 0.00 3.19 0.94 38.30 nvme2c2n1 0.00 7533.00 0.00 407.00 0.00 493.97 2485.64 0.99 2.43 0.00 2.43 0.75 30.50 nvme1n1 0.00 7497.00 0.00 407.00 0.00 494.00 2485.78 0.92 2.26 0.00 2.26 0.73 29.60 nvme3n1 0.00 7497.00 0.00 407.00 0.00 494.00 2485.78 1.04 2.55 0.00 2.55 0.85 34.50 sda 0.00 0.00 0.00 1.00 0.00 0.00 8.00 0.00 1.00 0.00 1.00 1.00 0.10 dm-0 0.00 0.00 0.00 7904.00 0.00 494.00 128.00 12.50 1.58 0.00 1.58 0.05 37.10 dm-1 0.00 0.00 0.00 7905.00 0.00 494.00 127.98 17.70 2.24 0.00 2.24 0.05 40.60 dm-2 0.00 0.00 0.00 7940.00 0.00 494.00 127.42 11.46 1.44 0.00 1.44 0.04 33.10 dm-3 0.00 0.00 0.00 7904.00 0.00 494.00 128.00 10.23 1.29 0.00 1.29 0.04 32.30 dm-4 0.00 0.00 0.00 31653.00 0.00 1976.00 127.85 51.94 1.64 0.00 1.64 0.01 40.90
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 9697.00 0.00 528.00 0.00 639.25 2479.52 1.65 3.12 0.00 3.12 0.99 52.50 nvme2c2n1 0.00 9725.00 0.00 529.00 0.00 639.28 2474.93 1.11 2.09 0.00 2.09 0.76 40.30 nvme1n1 0.00 9697.00 0.00 528.00 0.00 639.25 2479.52 1.03 1.95 0.00 1.95 0.74 39.30 nvme3n1 0.00 9697.00 0.00 528.00 0.00 639.25 2479.52 1.07 2.02 0.00 2.02 0.77 40.80 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 10224.00 0.00 639.00 128.00 12.52 1.22 0.00 1.22 0.04 42.70 dm-1 0.00 0.00 0.00 10224.00 0.00 639.00 128.00 23.98 2.35 0.00 2.35 0.05 54.20 dm-2 0.00 0.00 0.00 10253.00 0.00 639.00 127.64 13.46 1.31 0.00 1.31 0.04 42.10 dm-3 0.00 0.00 0.00 10224.00 0.00 639.00 128.00 11.66 1.14 0.00 1.14 0.04 41.20 dm-4 0.00 0.00 0.00 40925.00 0.00 2556.00 127.91 61.66 1.51 0.00 1.51 0.01 54.70
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 6690.00 0.00 365.00 0.00 440.75 2473.03 1.25 3.42 0.00 3.42 0.94 34.40 nvme2c2n1 0.00 6737.00 0.00 367.00 0.00 440.72 2459.40 0.95 2.59 0.00 2.59 0.75 27.70 nvme1n1 0.00 6690.00 0.00 365.00 0.00 440.75 2473.03 0.88 2.42 0.00 2.42 0.75 27.20 nvme3n1 0.00 6690.00 0.00 365.00 0.00 440.75 2473.03 0.90 2.46 0.00 2.46 0.75 27.40 sda 0.00 0.00 0.00 1.00 0.00 0.00 8.00 0.00 1.00 0.00 1.00 1.00 0.10 dm-0 0.00 0.00 0.00 7056.00 0.00 441.00 128.00 9.82 1.39 0.00 1.39 0.04 28.80 dm-1 0.00 0.00 0.00 7056.00 0.00 441.00 128.00 16.67 2.36 0.00 2.36 0.05 35.70 dm-2 0.00 0.00 0.00 7105.00 0.00 441.00 127.12 10.70 1.51 0.00 1.51 0.04 29.10 dm-3 0.00 0.00 0.00 7056.00 0.00 441.00 128.00 9.60 1.36 0.00 1.36 0.04 28.50 dm-4 0.00 0.00 0.00 28273.00 0.00 1764.00 127.78 46.83 1.66 0.00 1.66 0.01 36.00
^C
[root@T620 ~]#
Quoted 58 lines · Wang Yugui
Hi,
Quoted 48 lines · Wang Yugui
Quoted 26 lines
Quoted 18 lines
Ok, that is further back in time than I expected. In terms of XFS, there are only two commits between 5.16..5.17 that might impact performance:
ebb7fb1557b1("xfs, iomap: limit individual ioend chain lengths in writeback")and
6795801366da("xfs: Support large folios")To test whether
ebb7fb1557b1is the cause, go tofs/iomap/buffered-io.cand change:(quoted diff) +1
-1-#define IOEND_BATCH_SIZE 4096 +#define IOEND_BATCH_SIZE 1048576This will increase the IO submission chain lengths to at least 4GB from the 16MB bound that was placed on 5.17 and newer kernels.
Quoted 4 lines
To test whether
6795801366dais the cause, go tofs/xfs/xfs_icache.cand comment out both calls tomapping_set_large_folios(). This will ensure the page cache only instantiates single page folios the same as 5.16 would have.6.1.x with '
mapping_set_large_foliosremove' and 'IOEND_BATCH_SIZE=1048576'fio WRITE: bw=6451MiB/s (6764MB/s)Quoted 7 lines
still performance regression when compare to linux 5.16.20
fio WRITE: bw=7666MiB/s (8039MB/s),but the performance regression is not too big, then difficult to bisect. We noticed samle level performance regression on btrfs too. so maby some problem of some code that is used by both btrfs and xfs such as iomap and mm/folio.
Yup, that's quite possibly something like the multi-gen LRU changes, but that's not the regression we need to find. :/
Quoted 9 lines
6.1.x with '
mapping_set_large_foliosremove' only'fio WRITE: bw=2676MiB/s (2806MB/s)6.1.x with 'IOEND_BATCH_SIZE=1048576' only'
fio WRITE: bw=5092MiB/s (5339MB/s), fio WRITE: bw=6076MiB/s (6371MB/s)maybe we need more fix or '
ebb7fb1557b1("xfs, iomap: limit individual ioend chain lengths in writeback")'.Quoted 6 lines
OK, can you re-run the two 6.1.x kernels above (the slow and the fast) and record the output of
iostat -dxm 1whilst the fio test is running? I want to see what the overall differences in the IO load on the devices are between the two runs. This will tell us how the IO sizes and queue depths change between the two kernels, etc.
Quoted 3 lines
iostat -dxm 1result saved in attachment file.
good.txt good performance
bad.txt bad performance
Thanks!
What I see here is that neither the good or the bad config are able to drive the hardware to 100% utilisation, but the way the IO stack is behaving is identical. The only difference is that the good config is driving much more IO to the devices, such that the top level RAID0 stripe reports ~90% utilisation vs 50% utilisation.
What this says to me is that the limitation in throughput is the single threaded background IO submission (the bdi-flush thread) is CPU bound in both cases, and that the difference is in how much CPU each IO submission is consuming.
From some tests here at lower bandwidth (1-2GB/s) with a batch size of 4096, I'm seeing the vast majority of submission CPU time being spent in folio_start_writeback(), and the vast majority of CPU time in IO completion being spent in folio_end_writeback. There's an order of magnitude more CPU time in these functions than in any of the XFS or iomap writeback functions.
A typical 5 second expanded snapshot profile (from perf top -g -U) of the bdi-flusher thread looks like this:
99.22% 3.68% [kernel] [k] write_cache_pages
- 65.13% write_cache_pages
- 46.84% iomap_do_writepage
- 35.50% __folio_start_writeback
- 7.94% _raw_spin_lock_irqsave
- 11.35% do_raw_spin_lock
__pv_queued_spin_lock_slowpath
- 5.37% _raw_spin_unlock_irqrestore
- 5.32% do_raw_spin_unlock
__raw_callee_save___pv_queued_spin_unlock
- 0.92% asm_common_interrupt
common_interrupt
__common_interrupt
handle_edge_irq
handle_irq_event
__handle_irq_event_percpu
vring_interrupt
virtblk_done
- 4.18% __mod_lruvec_page_state
- 2.18% __mod_lruvec_state
1.16% __mod_node_page_state
0.68% __mod_memcg_lruvec_state
0.90% __mod_memcg_lruvec_state
2.88% xas_descend
1.63% percpu_counter_add_batch
1.63% mod_zone_page_state
1.15% xas_load
1.11% xas_start
0.93% __rcu_read_unlock
- 0.89% folio_memcg_lock
0.63% asm_common_interrupt
common_interrupt
__common_interrupt
handle_edge_irq
handle_irq_event
__handle_irq_event_percpu
vring_interrupt
virtblk_done
virtblk_complete_batch
blk_mq_end_request_batch
bio_endio
iomap_writepage_end_bio
iomap_finish_ioend
- 2.75% xfs_map_blocks
- 1.55% __might_sleep
1.26% __might_resched
- 1.90% bio_add_folio
1.13% __bio_try_merge_page
- 1.82% submit_bio
- submit_bio_noacct
- 1.82% submit_bio_noacct_nocheck
- __submit_bio
1.77% blk_mq_submit_bio
1.27% inode_to_bdi
1.19% xas_clear_mark
0.65% xas_set_mark
0.57% iomap_page_create.isra.0
- 12.91% folio_clear_dirty_for_io
- 2.72% __mod_lruvec_page_state
- 1.84% __mod_lruvec_state
0.98% __mod_node_page_state
0.58% __mod_memcg_lruvec_state
1.55% mod_zone_page_state
1.49% percpu_counter_add_batch
- 0.72% asm_common_interrupt
common_interrupt
__common_interrupt
handle_edge_irq
handle_irq_event
__handle_irq_event_percpu
vring_interrupt
virtblk_done
virtblk_complete_batch
blk_mq_end_request_batch
bio_endio
iomap_writepage_end_bio
iomap_finish_ioend
0.55% folio_mkclean
- 8.08% filemap_get_folios_tag
1.84% xas_find_marked
- 1.89% __pagevec_release
1.87% release_pages
- 1.65% __might_sleep
1.33% __might_resched
1.22% folio_unlock - 3.68% ret_from_fork
kthread
worker_thread
process_one_work
wb_workfn
wb_writeback
__writeback_inodes_wb
writeback_sb_inodes
__writeback_single_inode
do_writepages
xfs_vm_writepages
iomap_writepages
write_cache_pages
This indicates that 35% of writeback submission CPU is in __folio_start_writeback(), 13% is in folio_clear_dirty_for_io(), 8% is in filemap_get_folios_tag() and only ~8% of CPU time is in the rest of the iomap/XFS code building and submitting bios from the folios passed to it. i.e. it looks a lot like writeback is is contending with the incoming write(), IO completion and memory reclaim contexts for access to the page cache mapping and mm accounting structures.
Unfortunately, I don't have access to hardware that I can use to confirm this is the cause, but it doesn't look like it's directly an XFS/iomap issue at this point. The larger batch sizes reduce both memory reclaim and IO completion competition with submission, so it kinda points in this direction.
I suspect we need to start using high order folios in the write path where we have large user IOs for streaming writes, but I also wonder if there isn't some sort of batched accounting/mapping tree updates we could do for all the adjacent folios in a single bio....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Hi,
Quoted 201 lines · Wang Yugui
Quoted 58 lines
Hi,
Quoted 48 lines · Wang Yugui
Quoted 26 lines
Quoted 18 lines
Ok, that is further back in time than I expected. In terms of XFS, there are only two commits between 5.16..5.17 that might impact performance:
ebb7fb1557b1("xfs, iomap: limit individual ioend chain lengths in writeback")and
6795801366da("xfs: Support large folios")To test whether
ebb7fb1557b1is the cause, go tofs/iomap/buffered-io.cand change:(quoted diff) +1
-1-#define IOEND_BATCH_SIZE 4096 +#define IOEND_BATCH_SIZE 1048576This will increase the IO submission chain lengths to at least 4GB from the 16MB bound that was placed on 5.17 and newer kernels.
Quoted 4 lines
To test whether
6795801366dais the cause, go tofs/xfs/xfs_icache.cand comment out both calls tomapping_set_large_folios(). This will ensure the page cache only instantiates single page folios the same as 5.16 would have.6.1.x with '
mapping_set_large_foliosremove' and 'IOEND_BATCH_SIZE=1048576'fio WRITE: bw=6451MiB/s (6764MB/s)Quoted 7 lines
still performance regression when compare to linux 5.16.20
fio WRITE: bw=7666MiB/s (8039MB/s),but the performance regression is not too big, then difficult to bisect. We noticed samle level performance regression on btrfs too. so maby some problem of some code that is used by both btrfs and xfs such as iomap and mm/folio.
Yup, that's quite possibly something like the multi-gen LRU changes, but that's not the regression we need to find. :/
Quoted 9 lines
6.1.x with '
mapping_set_large_foliosremove' only'fio WRITE: bw=2676MiB/s (2806MB/s)6.1.x with 'IOEND_BATCH_SIZE=1048576' only'
fio WRITE: bw=5092MiB/s (5339MB/s), fio WRITE: bw=6076MiB/s (6371MB/s)maybe we need more fix or '
ebb7fb1557b1("xfs, iomap: limit individual ioend chain lengths in writeback")'.Quoted 6 lines
OK, can you re-run the two 6.1.x kernels above (the slow and the fast) and record the output of
iostat -dxm 1whilst the fio test is running? I want to see what the overall differences in the IO load on the devices are between the two runs. This will tell us how the IO sizes and queue depths change between the two kernels, etc.Quoted 3 lines
iostat -dxm 1result saved in attachment file.
good.txt good performance
bad.txt bad performanceThanks!
What I see here is that neither the good or the bad config are able to drive the hardware to 100% utilisation, but the way the IO stack is behaving is identical. The only difference is that the good config is driving much more IO to the devices, such that the top level RAID0 stripe reports ~90% utilisation vs 50% utilisation.
What this says to me is that the limitation in throughput is the single threaded background IO submission (the bdi-flush thread) is CPU bound in both cases, and that the difference is in how much CPU each IO submission is consuming.
From some tests here at lower bandwidth (1-2GB/s) with a batch size of 4096, I'm seeing the vast majority of submission CPU time being spent in
folio_start_writeback(), and the vast majority of CPU time in IO completion being spent infolio_end_writeback. There's an order of magnitude more CPU time in these functions than in any of the XFS or iomap writeback functions.A typical 5 second expanded snapshot profile (from
perf top -g -U) of the bdi-flusher thread looks like this:99.22% 3.68% [kernel] [k] write_cache_pages - 65.13% write_cache_pages - 46.84% iomap_do_writepage - 35.50% __folio_start_writeback - 7.94% _raw_spin_lock_irqsave - 11.35% do_raw_spin_lock __pv_queued_spin_lock_slowpath - 5.37% _raw_spin_unlock_irqrestore - 5.32% do_raw_spin_unlock __raw_callee_save___pv_queued_spin_unlock - 0.92% asm_common_interrupt common_interrupt __common_interrupt handle_edge_irq handle_irq_event __handle_irq_event_percpu vring_interrupt virtblk_done - 4.18% __mod_lruvec_page_state - 2.18% __mod_lruvec_state 1.16% __mod_node_page_state 0.68% __mod_memcg_lruvec_state 0.90% __mod_memcg_lruvec_state 2.88% xas_descend 1.63% percpu_counter_add_batch 1.63% mod_zone_page_state 1.15% xas_load 1.11% xas_start 0.93% __rcu_read_unlock - 0.89% folio_memcg_lock 0.63% asm_common_interrupt common_interrupt __common_interrupt handle_edge_irq handle_irq_event __handle_irq_event_percpu vring_interrupt virtblk_done virtblk_complete_batch blk_mq_end_request_batch bio_endio iomap_writepage_end_bio iomap_finish_ioend - 2.75% xfs_map_blocks - 1.55% __might_sleep 1.26% __might_resched - 1.90% bio_add_folio 1.13% __bio_try_merge_page - 1.82% submit_bio - submit_bio_noacct - 1.82% submit_bio_noacct_nocheck - __submit_bio 1.77% blk_mq_submit_bio 1.27% inode_to_bdi 1.19% xas_clear_mark 0.65% xas_set_mark 0.57% iomap_page_create.isra.0 - 12.91% folio_clear_dirty_for_io - 2.72% __mod_lruvec_page_state - 1.84% __mod_lruvec_state 0.98% __mod_node_page_state 0.58% __mod_memcg_lruvec_state 1.55% mod_zone_page_state 1.49% percpu_counter_add_batch - 0.72% asm_common_interrupt common_interrupt __common_interrupt handle_edge_irq handle_irq_event __handle_irq_event_percpu vring_interrupt virtblk_done virtblk_complete_batch blk_mq_end_request_batch bio_endio iomap_writepage_end_bio iomap_finish_ioend 0.55% folio_mkclean - 8.08% filemap_get_folios_tag 1.84% xas_find_marked - 1.89% __pagevec_release 1.87% release_pages - 1.65% __might_sleep 1.33% __might_resched 1.22% folio_unlock - 3.68% ret_from_fork kthread worker_thread process_one_work wb_workfn wb_writeback __writeback_inodes_wb writeback_sb_inodes __writeback_single_inode do_writepages xfs_vm_writepages iomap_writepages write_cache_pagesThis indicates that 35% of writeback submission CPU is in
__folio_start_writeback(), 13% is infolio_clear_dirty_for_io(), 8% is infilemap_get_folios_tag()and only ~8% of CPU time is in the rest of the iomap/XFS code building and submitting bios from the folios passed to it. i.e. it looks a lot like writeback is is contending with the incomingwrite(), IO completion and memory reclaim contexts for access to the page cache mapping and mm accounting structures.Unfortunately, I don't have access to hardware that I can use to confirm this is the cause, but it doesn't look like it's directly an XFS/iomap issue at this point. The larger batch sizes reduce both memory reclaim and IO completion competition with submission, so it kinda points in this direction.
Quoted 4 lines
I suspect we need to start using high order folios in the write path where we have large user IOs for streaming writes, but I also wonder if there isn't some sort of batched accounting/mapping tree updates we could do for all the adjacent folios in a single bio....
Is there some comment from Matthew Wilcox?
since it seems a folios problem?
Best Regards
Wang Yugui ([email protected])
2023/05/17
Quoted 15 lines · Wang Yugui
Quoted 15 lines
This indicates that 35% of writeback submission CPU is in
__folio_start_writeback(), 13% is infolio_clear_dirty_for_io(), 8% is infilemap_get_folios_tag()and only ~8% of CPU time is in the rest of the iomap/XFS code building and submitting bios from the folios passed to it. i.e. it looks a lot like writeback is is contending with the incomingwrite(), IO completion and memory reclaim contexts for access to the page cache mapping and mm accounting structures.Unfortunately, I don't have access to hardware that I can use to confirm this is the cause, but it doesn't look like it's directly an XFS/iomap issue at this point. The larger batch sizes reduce both memory reclaim and IO completion competition with submission, so it kinda points in this direction.
Quoted 8 lines
Quoted 4 lines
I suspect we need to start using high order folios in the write path where we have large user IOs for streaming writes, but I also wonder if there isn't some sort of batched accounting/mapping tree updates we could do for all the adjacent folios in a single bio....
Is there some comment from Matthew Wilcox?
since it seems a folios problem?
None of these are new "folio problems" - we've known about these scalability limitations of page-based writeback caching for over 15 years. e.g. from 2006:
https://www.kernel.org/doc/ols/2006/ols2006v1-pages-177-192.pdf
The fundamental problem is the huge number of page cache objects that buffered IO must handle when moving multiple GB/s to/from storage devices. Folios offer a way to mitigate that by reducing the number of page cache objects via using large folios in the write() path, but we have not enabled that functionality yet.
If you want to look at making the iomap path and filemap_get_folio() paths allocate high order folios, then that will largely mitigate the worst of the performance degredation.
Another possible avenue is to batch all the folio updates in the IO completion path. We currently do that one folio at a time, so a typical IO might be doing a several dozen (or more) page cache updates that largely could be done as a single update per IO. Worse is that these individual updates are typically done under exclusive locking, so this means the lock holds are no only more frequent than they need to be, they are also longer than they need to be.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Matthew Wilcox
maintainerCreating large folios in iomap buffered write path
In reply to Wang YuguiQuoted 8 lines · Wang Yugui
Quoted 4 lines · Dave Chinner
I suspect we need to start using high order folios in the write path where we have large user IOs for streaming writes, but I also wonder if there isn't some sort of batched accounting/mapping tree updates we could do for all the adjacent folios in a single bio....
Is there some comment from Matthew Wilcox?
since it seems a folios problem?
Not so much a "folio problem" as "an enhancement nobody got around to doing yet". Here's a first attempt. It's still churning through an xfstests run for me. I have seen this warning trigger:
WARN_ON_ONCE(!folio_test_uptodate(folio) &&
folio_test_dirty(folio));in iomap_invalidate_folio() as it's now possible to create a folio for write that is larger than the write, and therefore we won't mark it uptodate. Maybe we should create slightly smaller folios.
Anyway, how does this help your performance problem?
fs/gfs2/bmap.c +1 -1
@@ -971,7 +971,7 @@ gfs2_iomap_get_folio(struct iomap_iter *iter, loff_t pos, unsigned len)if (status)return ERR_PTR (status); - folio =iomap_get_folio (iter, pos); + folio =iomap_get_folio (iter, pos, len);if (IS_ERR (folio))gfs2_trans_end (sdp);return folio;
fs/iomap/buffered-io.c +4 -2
@@ -461,16 +461,18 @@ EXPORT_SYMBOL_GPL(iomap_is_partially_uptodate);* iomap_get_folio - get a folio reference for writing * @iter: iteration structure * @pos: start offset of write +* @len: length of write * * Returns a locked reference to the folio at @pos, or an error pointer if the * folio could not be obtained. */ -struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos) +struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos,size_t len) {unsigned fgp =FGP_WRITEBEGIN |FGP_NOFS ;if (iter->flags &IOMAP_NOWAIT ) fgp |=FGP_NOWAIT ; + fgp |=fgp_order (len);return __filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , fgp,mapping_gfp_mask (iter->inode->i_mapping));
@@ -603,7 +605,7 @@ static struct folio *__iomap_get_folio(struct iomap_iter *iter, loff_t pos,if (folio_ops && folio_ops->get_folio)return folio_ops->get_folio (iter, pos, len);else -return iomap_get_folio (iter, pos); +return iomap_get_folio (iter, pos, len); } static void __iomap_put_folio(struct iomap_iter *iter, loff_t pos, size_t ret,
include/linux/iomap.h +1 -1
@@ -261,7 +261,7 @@ int iomap_file_buffered_write_punch_delalloc(struct inode *inode,int iomap_read_folio (struct folio *folio,const struct iomap_ops *ops);void iomap_readahead (struct readahead_control *,const struct iomap_ops *ops);bool iomap_is_partially_uptodate (struct folio *,size_t from,size_t count); -struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos); +struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos,size_t len);bool iomap_release_folio (struct folio *folio,gfp_t gfp_flags);void iomap_invalidate_folio (struct folio *folio,size_t offset,size_t len); int iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
include/linux/pagemap.h +22 -3
@@ -466,6 +466,19 @@ static inline void *detach_page_private(struct page *page)return folio_detach_private (page_folio (page)); } +/* +* There are some parts of the kernel which assume that PMD entries +* are exactly HPAGE_PMD_ORDER. Those should be fixed, but until then, +* limit the maximum allocation order to PMD size. I'm not aware of any +* assumptions about maximum order if THP are disabled, but 8 seems like +* a good order (that's 1MB if you're using 4kB pages) +*/ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define MAX_PAGECACHE_ORDER HPAGE_PMD_ORDER +#else +#define MAX_PAGECACHE_ORDER 8 +#endif +#ifdef CONFIG_NUMA struct folio *filemap_alloc_folio (gfp_t gfp,unsigned int order);#else
@@ -505,14 +518,20 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,#define FGP_NOWAIT 0x00000020#define FGP_FOR_MMAP 0x00000040#define FGP_STABLE 0x00000080 +#define FGP_ORDER (fgp) ((fgp) >> 26)/* top 6 bits */ #define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE) +static inline unsigned fgp_order (size_t size) +{ +return get_order (size) <<26 ; +} +void *filemap_get_entry (struct address_space *mapping,pgoff_t index);struct folio *__filemap_get_folio (struct address_space *mapping,pgoff_t index, -int fgp_flags,gfp_t gfp); +unsigned fgp_flags,gfp_t gfp);struct page *pagecache_get_page (struct address_space *mapping,pgoff_t index, -int fgp_flags,gfp_t gfp); +unsigned fgp_flags,gfp_t gfp);/** * filemap_get_folio - Find and get a folio.
@@ -586,7 +605,7 @@ static inline struct page *find_get_page(struct address_space *mapping, }static inline struct page *find_get_page_flags (struct address_space *mapping, -pgoff_t offset,int fgp_flags) +pgoff_t offset,unsigned fgp_flags) {return pagecache_get_page (mapping, offset, fgp_flags,0 ); }
mm/filemap.c +28 -14
68 lines not on this page · raw message
mm/folio-compat.c +1 -1
@@ -92,7 +92,7 @@ EXPORT_SYMBOL(add_to_page_cache_lru);noinline struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index, -int fgp_flags,gfp_t gfp) +unsigned fgp_flags,gfp_t gfp) {struct folio *folio;
mm/readahead.c +0 -13
@@ -462,19 +462,6 @@ static int try_context_readahead(struct address_space *mapping,return 1 ; } -/* -* There are some parts of the kernel which assume that PMD entries -* are exactly HPAGE_PMD_ORDER. Those should be fixed, but until then, -* limit the maximum allocation order to PMD size. I'm not aware of any -* assumptions about maximum order if THP are disabled, but 8 seems like -* a good order (that's 1MB if you're using 4kB pages) -*/ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define MAX_PAGECACHE_ORDER HPAGE_PMD_ORDER -#else -#define MAX_PAGECACHE_ORDER 8 -#endif -static inline int ra_alloc_folio (struct readahead_control *ractl,pgoff_t index,pgoff_t mark,unsigned int order,gfp_t gfp) {
Matthew Wilcox
maintainerRe: Creating large folios in iomap buffered write path
Quoted 10 lines · Matthew Wilcox
Not so much a "folio problem" as "an enhancement nobody got around to doing yet". Here's a first attempt. It's still churning through an xfstests run for me. I have seen this warning trigger:
WARN_ON_ONCE(!folio_test_uptodate(folio) && folio_test_dirty(folio));in
iomap_invalidate_folio()as it's now possible to create a folio for write that is larger than the write, and therefore we won't mark it uptodate. Maybe we should create slightly smaller folios.
Here's one that does. A couple of other small problems also fixed.
fs/gfs2/bmap.c +1 -1
@@ -971,7 +971,7 @@ gfs2_iomap_get_folio(struct iomap_iter *iter, loff_t pos, unsigned len)if (status)return ERR_PTR (status); - folio =iomap_get_folio (iter, pos); + folio =iomap_get_folio (iter, pos, len);if (IS_ERR (folio))gfs2_trans_end (sdp);return folio;
fs/iomap/buffered-io.c +11 -5
@@ -461,19 +461,25 @@ EXPORT_SYMBOL_GPL(iomap_is_partially_uptodate);* iomap_get_folio - get a folio reference for writing * @iter: iteration structure * @pos: start offset of write +* @len: length of write * * Returns a locked reference to the folio at @pos, or an error pointer if the * folio could not be obtained. */ -struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos) +struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos,size_t len) {unsigned fgp =FGP_WRITEBEGIN |FGP_NOFS ; +struct folio *folio;if (iter->flags &IOMAP_NOWAIT ) fgp |=FGP_NOWAIT ; + fgp |=fgp_order (len); -return __filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , + folio =__filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , fgp,mapping_gfp_mask (iter->inode->i_mapping)); +if (!IS_ERR (folio) &&folio_test_large (folio)) +printk ("index:%lu len:%zu order:%u\n" , (unsigned long )(pos /PAGE_SIZE ), len,folio_order (folio)); +return folio; }EXPORT_SYMBOL_GPL (iomap_get_folio);
@@ -510,8 +516,8 @@ void iomap_invalidate_folio(struct folio *folio, size_t offset, size_t len)iomap_page_release (folio); }else if (folio_test_large (folio)) {/* Must release the iop so the page can be split */ -WARN_ON_ONCE (!folio_test_uptodate (folio) && -folio_test_dirty (folio)); +VM_WARN_ON_ONCE_FOLIO (!folio_test_uptodate (folio) && +folio_test_dirty (folio), folio);iomap_page_release (folio); } }
@@ -603,7 +609,7 @@ static struct folio *__iomap_get_folio(struct iomap_iter *iter, loff_t pos,if (folio_ops && folio_ops->get_folio)return folio_ops->get_folio (iter, pos, len);else -return iomap_get_folio (iter, pos); +return iomap_get_folio (iter, pos, len); } static void __iomap_put_folio(struct iomap_iter *iter, loff_t pos, size_t ret,
include/linux/iomap.h +1 -1
@@ -261,7 +261,7 @@ int iomap_file_buffered_write_punch_delalloc(struct inode *inode,int iomap_read_folio (struct folio *folio,const struct iomap_ops *ops);void iomap_readahead (struct readahead_control *,const struct iomap_ops *ops);bool iomap_is_partially_uptodate (struct folio *,size_t from,size_t count); -struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos); +struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos,size_t len);bool iomap_release_folio (struct folio *folio,gfp_t gfp_flags);void iomap_invalidate_folio (struct folio *folio,size_t offset,size_t len); int iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
include/linux/pagemap.h +26 -3
@@ -466,6 +466,19 @@ static inline void *detach_page_private(struct page *page)return folio_detach_private (page_folio (page)); } +/* +* There are some parts of the kernel which assume that PMD entries +* are exactly HPAGE_PMD_ORDER. Those should be fixed, but until then, +* limit the maximum allocation order to PMD size. I'm not aware of any +* assumptions about maximum order if THP are disabled, but 8 seems like +* a good order (that's 1MB if you're using 4kB pages) +*/ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define MAX_PAGECACHE_ORDER HPAGE_PMD_ORDER +#else +#define MAX_PAGECACHE_ORDER 8 +#endif +#ifdef CONFIG_NUMA struct folio *filemap_alloc_folio (gfp_t gfp,unsigned int order);#else
@@ -505,14 +518,24 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,#define FGP_NOWAIT 0x00000020#define FGP_FOR_MMAP 0x00000040#define FGP_STABLE 0x00000080 +#define FGP_ORDER (fgp) ((fgp) >> 26)/* top 6 bits */ #define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE) +static inline unsigned fgp_order (size_t size) +{ +unsigned int shift =ilog2 (size); + +if (shift <=PAGE_SHIFT ) +return 0 ; +return (shift -PAGE_SHIFT ) <<26 ; +} +void *filemap_get_entry (struct address_space *mapping,pgoff_t index);struct folio *__filemap_get_folio (struct address_space *mapping,pgoff_t index, -int fgp_flags,gfp_t gfp); +unsigned fgp_flags,gfp_t gfp);struct page *pagecache_get_page (struct address_space *mapping,pgoff_t index, -int fgp_flags,gfp_t gfp); +unsigned fgp_flags,gfp_t gfp);/** * filemap_get_folio - Find and get a folio.
@@ -586,7 +609,7 @@ static inline struct page *find_get_page(struct address_space *mapping, }static inline struct page *find_get_page_flags (struct address_space *mapping, -pgoff_t offset,int fgp_flags) +pgoff_t offset,unsigned fgp_flags) {return pagecache_get_page (mapping, offset, fgp_flags,0 ); }
mm/filemap.c +28 -14
68 lines not on this page · raw message
mm/folio-compat.c +1 -1
@@ -92,7 +92,7 @@ EXPORT_SYMBOL(add_to_page_cache_lru);noinline struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index, -int fgp_flags,gfp_t gfp) +unsigned fgp_flags,gfp_t gfp) {struct folio *folio;
mm/readahead.c +0 -13
@@ -462,19 +462,6 @@ static int try_context_readahead(struct address_space *mapping,return 1 ; } -/* -* There are some parts of the kernel which assume that PMD entries -* are exactly HPAGE_PMD_ORDER. Those should be fixed, but until then, -* limit the maximum allocation order to PMD size. I'm not aware of any -* assumptions about maximum order if THP are disabled, but 8 seems like -* a good order (that's 1MB if you're using 4kB pages) -*/ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define MAX_PAGECACHE_ORDER HPAGE_PMD_ORDER -#else -#define MAX_PAGECACHE_ORDER 8 -#endif -static inline int ra_alloc_folio (struct readahead_control *ractl,pgoff_t index,pgoff_t mark,unsigned int order,gfp_t gfp) {
Matthew Wilcox
maintainerRe: Creating large folios in iomap buffered write path
Quoted 7 lines · Matthew Wilcox
fs/iomap/buffered-io.c +2
-1-struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos) +struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos,size_t len) {unsigned fgp =FGP_WRITEBEGIN |FGP_NOFS ; +struct folio *folio;if (iter->flags &IOMAP_NOWAIT )
fs/iomap/buffered-io.c EXPORT_SYMBOL_GPL(iomap_is_partially_uptodate); · line 483
Quoted 10 lines
fs/iomap/buffered-io.c +5
-1fgp |=FGP_NOWAIT ; + fgp |=fgp_order (len); -return __filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , + folio =__filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , fgp,mapping_gfp_mask (iter->inode->i_mapping)); +if (!IS_ERR (folio) &&folio_test_large (folio)) +printk ("index:%lu len:%zu order:%u\n" , (unsigned long )(pos /PAGE_SIZE ), len,folio_order (folio)); +return folio; }
Forgot to take the debugging out. This should read:
-struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos) +struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
{
unsigned fgp = FGP_WRITEBEGIN | FGP_NOFS;
if (iter->flags & IOMAP_NOWAIT)
fgp |= FGP_NOWAIT;
+ fgp |= fgp_order(len);
return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,
fgp, mapping_gfp_mask(iter->inode->i_mapping));
}
Re: Creating large folios in iomap buffered write path
Hi,
Quoted 22 lines · Matthew Wilcox
Quoted 7 lines
(quoted diff) +2
-1-struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos) +struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len) { unsigned fgp = FGP_WRITEBEGIN | FGP_NOFS; + struct folio *folio; if (iter->flags & IOMAP_NOWAIT)Review
Quoted 10 lines
(quoted diff) +5
-1fgp |= FGP_NOWAIT; + fgp |= fgp_order(len); - return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT, + folio = __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT, fgp, mapping_gfp_mask(iter->inode->i_mapping)); + if (!IS_ERR(folio) && folio_test_large(folio)) + printk("index:%lu len:%zu order:%u\n", (unsigned long)(pos / PAGE_SIZE), len, folio_order(folio)); + return folio; }Forgot to take the debugging out. This should read:
(quoted diff) +0
-1-struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos)
Review
Quoted 10 lines
(quoted diff) +2
-0+struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len) { unsigned fgp = FGP_WRITEBEGIN | FGP_NOFS; if (iter->flags & IOMAP_NOWAIT) fgp |= FGP_NOWAIT; + fgp |= fgp_order(len); return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT, fgp, mapping_gfp_mask(iter->inode->i_mapping)); }
I test it (attachment file) on 6.4.0-rc2.
fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1 -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=4 -directory=/mnt/test
fio WRITE: bw=2430MiB/s.
expected value: > 6000MiB/s
so yet no fio write bandwidth improvement.
Best Regards
Wang Yugui ([email protected])
2023/05/19
From 46a20dc267cd2624841e0e12af12ea6c05e5f6b9 Mon Sep 17 00:00:00 2001
From: Matthew Wilcox <[email protected]>
Date: Thu, 18 May 2023 22:46:43 +0100
Subject: [PATCH] Creating large folios in iomap buffered write path
Quoted 10 lines · Matthew Wilcox
Not so much a "folio problem" as "an enhancement nobody got around to doing yet". Here's a first attempt. It's still churning through an xfstests run for me. I have seen this warning trigger:
WARN_ON_ONCE(!folio_test_uptodate(folio) && folio_test_dirty(folio));in
iomap_invalidate_folio()as it's now possible to create a folio for write that is larger than the write, and therefore we won't mark it uptodate. Maybe we should create slightly smaller folios.
Here's one that does. A couple of other small problems also fixed.
--- fs/gfs2/bmap.c | 2 +- fs/iomap/buffered-io.c | 10 ++++++---- include/linux/iomap.h | 2 +- include/linux/pagemap.h | 29 +++++++++++++++++++++++++--- mm/filemap.c | 42 +++++++++++++++++++++++++++-------------- mm/folio-compat.c | 2 +- mm/readahead.c | 13 ------------- 7 files changed, 63 insertions(+), 37 deletions(-)
fs/gfs2/bmap.c +1 -1
@@ -971,7 +971,7 @@ gfs2_iomap_get_folio(struct iomap_iter *iter, loff_t pos, unsigned len)if (status)return ERR_PTR (status); - folio =iomap_get_folio (iter, pos); + folio =iomap_get_folio (iter, pos, len);if (IS_ERR (folio))gfs2_trans_end (sdp);return folio;
fs/iomap/buffered-io.c +6 -4
@@ -461,16 +461,18 @@ EXPORT_SYMBOL_GPL(iomap_is_partially_uptodate);* iomap_get_folio - get a folio reference for writing * @iter: iteration structure * @pos: start offset of write +* @len: length of write * * Returns a locked reference to the folio at @pos, or an error pointer if the * folio could not be obtained. */ -struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos) +struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos,size_t len) {unsigned fgp =FGP_WRITEBEGIN |FGP_NOFS ;if (iter->flags &IOMAP_NOWAIT ) fgp |=FGP_NOWAIT ; + fgp |=fgp_order (len);return __filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , fgp,mapping_gfp_mask (iter->inode->i_mapping));
@@ -510,8 +512,8 @@ void iomap_invalidate_folio(struct folio *folio, size_t offset, size_t len)iomap_page_release (folio); }else if (folio_test_large (folio)) {/* Must release the iop so the page can be split */ -WARN_ON_ONCE (!folio_test_uptodate (folio) && -folio_test_dirty (folio)); +VM_WARN_ON_ONCE_FOLIO (!folio_test_uptodate (folio) && +folio_test_dirty (folio), folio);iomap_page_release (folio); } }
@@ -603,7 +605,7 @@ static struct folio *__iomap_get_folio(struct iomap_iter *iter, loff_t pos,if (folio_ops && folio_ops->get_folio)return folio_ops->get_folio (iter, pos, len);else -return iomap_get_folio (iter, pos); +return iomap_get_folio (iter, pos, len); } static void __iomap_put_folio(struct iomap_iter *iter, loff_t pos, size_t ret,
include/linux/iomap.h +1 -1
@@ -261,7 +261,7 @@ int iomap_file_buffered_write_punch_delalloc(struct inode *inode,int iomap_read_folio (struct folio *folio,const struct iomap_ops *ops);void iomap_readahead (struct readahead_control *,const struct iomap_ops *ops);bool iomap_is_partially_uptodate (struct folio *,size_t from,size_t count); -struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos); +struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos,size_t len);bool iomap_release_folio (struct folio *folio,gfp_t gfp_flags);void iomap_invalidate_folio (struct folio *folio,size_t offset,size_t len); int iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
include/linux/pagemap.h +26 -3
@@ -466,6 +466,19 @@ static inline void *detach_page_private(struct page *page)return folio_detach_private (page_folio (page)); } +/* +* There are some parts of the kernel which assume that PMD entries +* are exactly HPAGE_PMD_ORDER. Those should be fixed, but until then, +* limit the maximum allocation order to PMD size. I'm not aware of any +* assumptions about maximum order if THP are disabled, but 8 seems like +* a good order (that's 1MB if you're using 4kB pages) +*/ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define MAX_PAGECACHE_ORDER HPAGE_PMD_ORDER +#else +#define MAX_PAGECACHE_ORDER 8 +#endif +#ifdef CONFIG_NUMA struct folio *filemap_alloc_folio (gfp_t gfp,unsigned int order);#else
@@ -505,14 +518,24 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,#define FGP_NOWAIT 0x00000020#define FGP_FOR_MMAP 0x00000040#define FGP_STABLE 0x00000080 +#define FGP_ORDER (fgp) ((fgp) >> 26)/* top 6 bits */ #define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE) +static inline unsigned fgp_order (size_t size) +{ +unsigned int shift =ilog2 (size); + +if (shift <=PAGE_SHIFT ) +return 0 ; +return (shift -PAGE_SHIFT ) <<26 ; +} +void *filemap_get_entry (struct address_space *mapping,pgoff_t index);struct folio *__filemap_get_folio (struct address_space *mapping,pgoff_t index, -int fgp_flags,gfp_t gfp); +unsigned fgp_flags,gfp_t gfp);struct page *pagecache_get_page (struct address_space *mapping,pgoff_t index, -int fgp_flags,gfp_t gfp); +unsigned fgp_flags,gfp_t gfp);/** * filemap_get_folio - Find and get a folio.
@@ -586,7 +609,7 @@ static inline struct page *find_get_page(struct address_space *mapping, }static inline struct page *find_get_page_flags (struct address_space *mapping, -pgoff_t offset,int fgp_flags) +pgoff_t offset,unsigned fgp_flags) {return pagecache_get_page (mapping, offset, fgp_flags,0 ); }
mm/filemap.c +28 -14
68 lines not on this page · raw message
mm/folio-compat.c +1 -1
@@ -92,7 +92,7 @@ EXPORT_SYMBOL(add_to_page_cache_lru);noinline struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index, -int fgp_flags,gfp_t gfp) +unsigned fgp_flags,gfp_t gfp) {struct folio *folio;
mm/readahead.c +0 -13
@@ -462,19 +462,6 @@ static int try_context_readahead(struct address_space *mapping,return 1 ; } -/* -* There are some parts of the kernel which assume that PMD entries -* are exactly HPAGE_PMD_ORDER. Those should be fixed, but until then, -* limit the maximum allocation order to PMD size. I'm not aware of any -* assumptions about maximum order if THP are disabled, but 8 seems like -* a good order (that's 1MB if you're using 4kB pages) -*/ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define MAX_PAGECACHE_ORDER HPAGE_PMD_ORDER -#else -#define MAX_PAGECACHE_ORDER 8 -#endif -static inline int ra_alloc_folio (struct readahead_control *ractl,pgoff_t index,pgoff_t mark,unsigned int order,gfp_t gfp) {
Signature · 1 line
2.36.2
Matthew Wilcox
maintainerRe: Creating large folios in iomap buffered write path
Quoted 38 lines · Wang Yugui
Hi,
Quoted 22 lines · Matthew Wilcox
Quoted 7 lines
fs/iomap/buffered-io.c +2
-1-struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos) +struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos,size_t len) {unsigned fgp =FGP_WRITEBEGIN |FGP_NOFS ; +struct folio *folio;if (iter->flags &IOMAP_NOWAIT )fs/iomap/buffered-io.c EXPORT_SYMBOL_GPL(iomap_is_partially_uptodate);
Quoted 10 lines
fs/iomap/buffered-io.c +5
-1fgp |=FGP_NOWAIT ; + fgp |=fgp_order (len); -return __filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , + folio =__filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , fgp,mapping_gfp_mask (iter->inode->i_mapping)); +if (!IS_ERR (folio) &&folio_test_large (folio)) +printk ("index:%lu len:%zu order:%u\n" , (unsigned long )(pos /PAGE_SIZE ), len,folio_order (folio)); +return folio; }Forgot to take the debugging out. This should read:
fs/iomap/buffered-io.c +0
-1-struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos)fs/iomap/buffered-io.c EXPORT_SYMBOL_GPL(iomap_is_partially_uptodate);
Quoted 10 lines
fs/iomap/buffered-io.c +2
-0+struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos,size_t len) {unsigned fgp =FGP_WRITEBEGIN |FGP_NOFS ;if (iter->flags &IOMAP_NOWAIT ) fgp |=FGP_NOWAIT ; + fgp |=fgp_order (len);return __filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , fgp,mapping_gfp_mask (iter->inode->i_mapping)); }I test it (attachment file) on 6.4.0-rc2.
fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1 -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=4 -directory=/mnt/test
Quoted 3 lines
fio WRITE: bw=2430MiB/s.
expected value: > 6000MiB/s
so yet no fio write bandwidth improvement.
That's basically unchanged. There's no per-page or per-block work being done in start/end writeback, so if Dave's investigation is applicable to your situation, I'd expect to see an improvement.
Maybe try the second version of the patch I sent with the debug in, to confirm you really are seeing large folios being created (you might want to use printk_ratelimit() instead of printk to ensure it doesn't overwhelm your system)? That fio command you were using ought to create them, but there's always a chance it doesn't.
Perhaps you could use perf (the command Dave used) to see where the time is being spent.
Re: Creating large folios in iomap buffered write path
Hi,
Quoted 47 lines · Wang Yugui
Quoted 38 lines
Hi,
Quoted 22 lines · Matthew Wilcox
Quoted 7 lines
fs/iomap/buffered-io.c +2
-1-struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos) +struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos,size_t len) {unsigned fgp =FGP_WRITEBEGIN |FGP_NOFS ; +struct folio *folio;if (iter->flags &IOMAP_NOWAIT )fs/iomap/buffered-io.c EXPORT_SYMBOL_GPL(iomap_is_partially_uptodate);
Quoted 10 lines
fs/iomap/buffered-io.c +5
-1fgp |=FGP_NOWAIT ; + fgp |=fgp_order (len); -return __filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , + folio =__filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , fgp,mapping_gfp_mask (iter->inode->i_mapping)); +if (!IS_ERR (folio) &&folio_test_large (folio)) +printk ("index:%lu len:%zu order:%u\n" , (unsigned long )(pos /PAGE_SIZE ), len,folio_order (folio)); +return folio; }Forgot to take the debugging out. This should read:
fs/iomap/buffered-io.c +0
-1-struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos)fs/iomap/buffered-io.c EXPORT_SYMBOL_GPL(iomap_is_partially_uptodate);
Quoted 10 lines
fs/iomap/buffered-io.c +2
-0+struct folio *iomap_get_folio (struct iomap_iter *iter,loff_t pos,size_t len) {unsigned fgp =FGP_WRITEBEGIN |FGP_NOFS ;if (iter->flags &IOMAP_NOWAIT ) fgp |=FGP_NOWAIT ; + fgp |=fgp_order (len);return __filemap_get_folio (iter->inode->i_mapping, pos >>PAGE_SHIFT , fgp,mapping_gfp_mask (iter->inode->i_mapping)); }I test it (attachment file) on 6.4.0-rc2.
fio -name write-bandwidth -rw=write -bs=1024Ki -size=32Gi -runtime=30 -iodepth 1 -ioengine sync -zero_buffers=1 -direct=0 -end_fsync=1 -numjobs=4 -directory=/mnt/testQuoted 3 lines
fio WRITE: bw=2430MiB/s.
expected value: > 6000MiB/s
so yet no fio write bandwidth improvement.That's basically unchanged. There's no per-page or per-block work being done in start/end writeback, so if Dave's investigation is applicable to your situation, I'd expect to see an improvement.
Quoted 8 lines
Maybe try the second version of the patch I sent with the debug in, to confirm you really are seeing large folios being created (you might want to use
printk_ratelimit()instead of printk to ensure it doesn't overwhelm your system)? That fio command you were using ought to create them, but there's always a chance it doesn't.Perhaps you could use perf (the command Dave used) to see where the time is being spent.
test result of the second version of the patch.
# dmesg |grep 'index\|suppressed' [ 89.376149] index:0 len:292 order:2 [ 97.862938] index:0 len:4096 order:2 [ 98.340665] index:0 len:4096 order:2 [ 98.346633] index:0 len:4096 order:2 [ 98.352323] index:0 len:4096 order:2 [ 98.359952] index:0 len:4096 order:2 [ 98.364015] index:3 len:4096 order:2 [ 98.368943] index:0 len:4096 order:2 [ 98.374285] index:0 len:4096 order:2 [ 98.379163] index:3 len:4096 order:2 [ 98.384760] index:0 len:4096 order:2 [ 181.103751] iomap_get_folio: 342 callbacks suppressed [ 181.103761] index:0 len:292 order:2
'perf report -g' result:
Samples: 344K of event 'cycles', Event count (approx.): 103747884662
Children Self Command Shared Object Symbol + 58.73% 0.01% fio [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe + 58.72% 0.01% fio [kernel.kallsyms] [k] do_syscall_64 + 58.53% 0.00% fio libpthread-2.17.so [.] 0x00007f83e400e6fd + 58.47% 0.01% fio [kernel.kallsyms] [k] ksys_write + 58.45% 0.02% fio [kernel.kallsyms] [k] vfs_write + 58.41% 0.03% fio [kernel.kallsyms] [k] xfs_file_buffered_write + 57.96% 0.57% fio [kernel.kallsyms] [k] iomap_file_buffered_write + 27.57% 1.29% fio [kernel.kallsyms] [k] iomap_write_begin + 25.32% 0.43% fio [kernel.kallsyms] [k] iomap_get_folio + 24.84% 0.70% fio [kernel.kallsyms] [k] __filemap_get_folio.part.69 + 20.11% 0.00% kworker/u98:15- [kernel.kallsyms] [k] ret_from_fork + 20.11% 0.00% kworker/u98:15- [kernel.kallsyms] [k] kthread + 20.11% 0.00% kworker/u98:15- [kernel.kallsyms] [k] worker_thread + 20.11% 0.00% kworker/u98:15- [kernel.kallsyms] [k] process_one_work + 20.11% 0.00% kworker/u98:15- [kernel.kallsyms] [k] wb_workfn + 20.11% 0.00% kworker/u98:15- [kernel.kallsyms] [k] wb_writeback + 20.11% 0.00% kworker/u98:15- [kernel.kallsyms] [k] __writeback_inodes_wb + 20.11% 0.00% kworker/u98:15- [kernel.kallsyms] [k] writeback_sb_inodes + 20.11% 0.00% kworker/u98:15- [kernel.kallsyms] [k] __writeback_single_inode + 20.11% 0.00% kworker/u98:15- [kernel.kallsyms] [k] do_writepages + 20.11% 0.00% kworker/u98:15- [kernel.kallsyms] [k] xfs_vm_writepages + 20.10% 0.00% kworker/u98:15- [kernel.kallsyms] [k] iomap_writepages + 20.10% 0.69% kworker/u98:15- [kernel.kallsyms] [k] write_cache_pages + 16.95% 0.39% fio [kernel.kallsyms] [k] copy_page_from_iter_atomic + 16.53% 0.10% fio [kernel.kallsyms] [k] copyin
'perf report ' result:
Samples: 335K of event 'cycles', Event count (approx.): 108508755052
Overhead Command Shared Object Symbol 17.70% fio [kernel.kallsyms] [k] rep_movs_alternative 2.89% kworker/u98:2-e [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 2.88% fio [kernel.kallsyms] [k] get_page_from_freelist 2.67% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 2.59% fio [kernel.kallsyms] [k] asm_exc_nmi 2.25% swapper [kernel.kallsyms] [k] intel_idle 1.59% kworker/u98:2-e [kernel.kallsyms] [k] __folio_start_writeback 1.52% fio [kernel.kallsyms] [k] xas_load 1.45% fio [kernel.kallsyms] [k] lru_add_fn 1.44% fio [kernel.kallsyms] [k] xas_descend 1.32% fio [kernel.kallsyms] [k] iomap_write_begin 1.29% fio [kernel.kallsyms] [k] __filemap_add_folio 1.08% kworker/u98:2-e [kernel.kallsyms] [k] folio_clear_dirty_for_io 1.07% fio [kernel.kallsyms] [k] __folio_mark_dirty 0.94% fio [kernel.kallsyms] [k] iomap_set_range_uptodate.part.24 0.93% fio [kernel.kallsyms] [k] node_dirty_ok 0.92% kworker/u98:2-e [kernel.kallsyms] [k] _raw_spin_lock_irqsave 0.83% fio [kernel.kallsyms] [k] xas_start 0.83% fio [kernel.kallsyms] [k] __alloc_pages 0.83% fio [kernel.kallsyms] [k] _raw_spin_lock_irqsave 0.81% kworker/u98:2-e [kernel.kallsyms] [k] asm_exc_nmi 0.79% fio [kernel.kallsyms] [k] percpu_counter_add_batch 0.75% kworker/u98:2-e [kernel.kallsyms] [k] iomap_writepage_map 0.74% kworker/u98:2-e [kernel.kallsyms] [k] __mod_lruvec_page_state 0.70% fio fio [.] 0x000000000001b1ac 0.70% fio [kernel.kallsyms] [k] filemap_dirty_folio 0.69% kworker/u98:2-e [kernel.kallsyms] [k] write_cache_pages 0.69% fio [kernel.kallsyms] [k] __filemap_get_folio.part.69 0.67% kworker/1:0-eve [kernel.kallsyms] [k] native_queued_spin_lock_slowpath 0.66% fio [kernel.kallsyms] [k] __mod_lruvec_page_state 0.64% fio [kernel.kallsyms] [k] __mod_node_page_state 0.64% fio [kernel.kallsyms] [k] folio_add_lru 0.64% fio [kernel.kallsyms] [k] balance_dirty_pages_ratelimited_flags 0.62% fio [kernel.kallsyms] [k] __mod_memcg_lruvec_state 0.61% fio [kernel.kallsyms] [k] iomap_write_end 0.60% fio [kernel.kallsyms] [k] xas_find_conflict 0.59% fio [kernel.kallsyms] [k] bad_range 0.58% kworker/u98:2-e [kernel.kallsyms] [k] xas_load 0.57% fio [kernel.kallsyms] [k] iomap_file_buffered_write 0.56% kworker/u98:2-e [kernel.kallsyms] [k] percpu_counter_add_batch 0.49% fio [kernel.kallsyms] [k] __might_resched
'perf top -g -U' result:
Samples: 78K of event 'cycles', 4000 Hz, Event count (approx.): 29400815085 lost: 0/0 drop: 0/0
Children Self Shared Object Symbol + 62.59% 0.03% [kernel] [k] entry_SYSCALL_64_after_hwframe + 60.15% 0.02% [kernel] [k] do_syscall_64 + 59.45% 0.02% [kernel] [k] vfs_write + 59.09% 0.54% [kernel] [k] iomap_file_buffered_write + 57.41% 0.00% [kernel] [k] ksys_write + 57.36% 0.01% [kernel] [k] xfs_file_buffered_write + 37.82% 0.00% libpthread-2.17.so [.] 0x00007fce6f20e6fd + 26.83% 1.20% [kernel] [k] iomap_write_begin + 24.65% 0.45% [kernel] [k] iomap_get_folio + 24.15% 0.74% [kernel] [k] __filemap_get_folio.part.69 + 20.17% 0.00% [kernel] [k] __writeback_single_inode + 20.17% 0.65% [kernel] [k] write_cache_pages + 17.66% 0.43% [kernel] [k] copy_page_from_iter_atomic + 17.18% 0.12% [kernel] [k] copyin + 17.08% 16.71% [kernel] [k] rep_movs_alternative + 16.97% 0.00% [kernel] [k] ret_from_fork + 16.97% 0.00% [kernel] [k] kthread + 16.87% 0.00% [kernel] [k] worker_thread + 16.84% 0.00% [kernel] [k] process_one_work + 14.86% 0.17% [kernel] [k] filemap_add_folio + 13.83% 0.77% [kernel] [k] iomap_writepage_map + 11.90% 0.33% [kernel] [k] iomap_finish_ioend + 11.57% 0.23% [kernel] [k] folio_end_writeback + 11.51% 0.73% [kernel] [k] iomap_write_end + 11.30% 2.14% [kernel] [k] __folio_end_writeback + 10.70% 0.00% [kernel] [k] wb_workfn + 10.70% 0.00% [kernel] [k] wb_writeback + 10.70% 0.00% [kernel] [k] __writeback_inodes_wb + 10.70% 0.00% [kernel] [k] writeback_sb_inodes + 10.70% 0.00% [kernel] [k] do_writepages + 10.70% 0.00% [kernel] [k] xfs_vm_writepages + 10.70% 0.00% [kernel] [k] iomap_writepages + 10.19% 2.68% [kernel] [k] _raw_spin_lock_irqsave + 10.17% 1.35% [kernel] [k] __filemap_add_folio + 9.94% 0.00% [unknown] [k] 0x0000000001942a70 + 9.94% 0.00% [unknown] [k] 0x0000000001942ac0 + 9.94% 0.00% [unknown] [k] 0x0000000001942b30
Best Regards
Wang Yugui ([email protected])
2023/05/20
Matthew Wilcox
maintainerRe: Creating large folios in iomap buffered write path
Quoted 16 lines · Wang Yugui
test result of the second version of the patch.
# dmesg |grep 'index\|suppressed' [ 89.376149] index:0 len:292 order:2 [ 97.862938] index:0 len:4096 order:2 [ 98.340665] index:0 len:4096 order:2 [ 98.346633] index:0 len:4096 order:2 [ 98.352323] index:0 len:4096 order:2 [ 98.359952] index:0 len:4096 order:2 [ 98.364015] index:3 len:4096 order:2 [ 98.368943] index:0 len:4096 order:2 [ 98.374285] index:0 len:4096 order:2 [ 98.379163] index:3 len:4096 order:2 [ 98.384760] index:0 len:4096 order:2 [ 181.103751] iomap_get_folio: 342 callbacks suppressed [ 181.103761] index:0 len:292 order:2
Thanks. Clearly we're not creating large folios in the write path. I tracked it down, and used your fio command. My system creates 1MB folios, so I think yours will too. Patch series incoming (I fixed a couple of other oversights too).