Skip to contentSkip to conversation

“place pages to tail” regress memory read bandwidth about 10% under our test cases

QUESTION liupeng (DM) 2 participants 2 messages Nov 30, 2021 Memory management

lore
Share & tools
lore
#0 lore raw

Hi David,

We met a performance regression(lmbench bw_mmap_rd/bw_mem) on our server with patch 7fef431be9c9(mm/page_alloc: place pages to tail in __free_pages_core()),

Case A. ./bw_mmap_rd -P 1 512m mmap_only out.file Case B. ./bw_mem -P 1 512m frd

mode0: IMC Interleave = auto, Channel Interleave = auto, Rank Interleave = auto
without with regression

case A           10535.14                       9645.02      8.4%
case B           10526.88                       9797.63      6.9%

also we found different memory interleaving have different results

mode1: IMC Interleave = 1-way, Channel Interleave = 2-way, Rank Interleave =2-way
without with regression

case A           10543.01                       10643.12       -0.9%
case B           10971.33                       10853.99       1%

The 7fef431be9c9 changes the different memory layout, it seems that the issue is related with memory interleave. But the patch does lead to performance regression.

Any suggestion about this regression?
thanks.

Peng Liu

#1 lore raw
Quoted 3 lines · liupeng (DM)

Hi David,

Hi!

Quoted 34 lines

We met a performance regression(lmbench bw_mmap_rd/bw_mem) on our server with

patch 7fef431be9c9(mm/page_alloc: place pages to tail in __free_pages_core()),

Case A.  ./bw_mmap_rd -P 1 512m mmap_only out.file

Case B.  ./bw_mem -P 1 512m frd

mode0: IMC Interleave = auto, Channel Interleave = auto, Rank Interleave = auto

                     without                        with        regression

case A           10535.14                       9645.02      8.4%

case B           10526.88                       9797.63      6.9%

also we found different memory interleaving have different results

mode1: IMC Interleave = 1-way, Channel Interleave = 2-way, Rank Interleave =2-way

                      without                        with regression

Quoted 8 lines

case A           10543.01                       10643.12       -0.9%

case B           10971.33                       10853.99       1%

The 7fef431be9c9 changes the different memory layout,  it seems that the issue

It doesn't change the physical memory layout. It changes the order in which physical memory will get allocated.

Quoted 7 lines

is related with memory interleave. But the patch does lead to performance regression.

Any suggestion about this regression?

Does it happen only on specific HW?

Commit 7fef431be9c9 discovered a bunch of BUGs already, including physical memory that has different performance characteristics due to firmware bugs, which indicated some physical memory e.g., as uncached to the OS.

That commit triggers allocation of physical memory in a different order, which sometimes makes "bad physical memory" getting allocated earlier than later in some benchmarks. Such effects are usually only visible on a freshly started system; once the system was busy for some time and the free page lists are nondeterministic, you might observe similar results.

I wrote a kernel module that was able to identify bad physical memory regions in some affected setups:
https://github.com/davidhildenbrand/kstream/blob/main/README.md

You can try running it after a freshly booted system to eventually identify some "bad" physical memory areas. But it might not be the root cause. Maybe it's just some caching effects?

Signature · 3 lines
Thanks,

David / dhildenb