“place pages to tail” regress memory read bandwidth about 10% under our test cases
Hi David,
We met a performance regression(lmbench bw_mmap_rd/bw_mem) on our server with patch 7fef431be9c9(mm/page_alloc: place pages to tail in __free_pages_core()),
Case A. ./bw_mmap_rd -P 1 512m mmap_only out.file Case B. ./bw_mem -P 1 512m frd
mode0: IMC Interleave = auto, Channel Interleave = auto, Rank Interleave = auto
without with regression
case A 10535.14 9645.02 8.4% case B 10526.88 9797.63 6.9%
also we found different memory interleaving have different results
mode1: IMC Interleave = 1-way, Channel Interleave = 2-way, Rank Interleave =2-way
without with regression
case A 10543.01 10643.12 -0.9% case B 10971.33 10853.99 1%
The 7fef431be9c9 changes the different memory layout, it seems that the issue is related with memory interleave. But the patch does lead to performance regression.
Any suggestion about this regression?
thanks.
Peng Liu
Quoted 3 lines · liupeng (DM)
Hi David,
Hi!
Quoted 34 lines
We met a performance
regression(lmbench bw_mmap_rd/bw_mem)on our server withpatch 7fef431be9c9(mm/page_alloc: place pages to tail in
__free_pages_core()),Case A. ./bw_mmap_rd -P 1 512m
mmap_onlyout.fileCase B. ./bw_mem -P 1 512m frd
mode0: IMC Interleave = auto, Channel Interleave = auto, Rank Interleave = auto
without with regression
case A 10535.14 9645.02 8.4%
case B 10526.88 9797.63 6.9%
also we found different memory interleaving have different results
mode1: IMC Interleave = 1-way, Channel Interleave = 2-way, Rank Interleave =2-way
without with regression
Quoted 8 lines
case A 10543.01 10643.12 -0.9%
case B 10971.33 10853.99 1%
The
7fef431be9c9changes the different memory layout, it seems that the issue
It doesn't change the physical memory layout. It changes the order in which physical memory will get allocated.
Quoted 7 lines
is related with memory interleave. But the patch does lead to performance regression.
Any suggestion about this regression?
Does it happen only on specific HW?
Commit 7fef431be9c9 discovered a bunch of BUGs already, including physical memory that has different performance characteristics due to firmware bugs, which indicated some physical memory e.g., as uncached to the OS.
That commit triggers allocation of physical memory in a different order, which sometimes makes "bad physical memory" getting allocated earlier than later in some benchmarks. Such effects are usually only visible on a freshly started system; once the system was busy for some time and the free page lists are nondeterministic, you might observe similar results.
I wrote a kernel module that was able to identify bad physical memory regions in some affected setups:
https://github.com/davidhildenbrand/kstream/blob/main/README.md
You can try running it after a freshly booted system to eventually identify some "bad" physical memory areas. But it might not be the root cause. Maybe it's just some caching effects?
Signature · 3 lines
Thanks, David / dhildenb