mm: add per-migratetype counts to buddy allocator and optimize pagetypeinfo access
Subsystems (5)
cover mm: add per-migratetype counts to buddy allocator and optimize pagetypeinfo access
On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.
Under high load scenarios, reading /proc/pagetypeinfo causes memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention
Changes:
1. Add per-migratetype counts to the buddy allocator to track free page block counts for each migratetype and order
2. Optimize /proc/pagetypeinfo access by utilizing these per-migratetype counts instead of iterating through free lists under zone lock
Performance testing shows following improvements:
1. /proc/pagetypeinfo access latency reduced
+-----------------------+----------+------------+
| | no-patch | with-patch |
+-----------------------+----------+------------+
| Just after boot | 700.9 us | 268.6 us |
+-----------------------+----------+------------+
| After building kernel | 28.7 ms | 269.8 us |
+-----------------------+----------+------------+2. When /proc/pagetypeinfo is accessed concurrently, memory allocation and free performance degradation is reduced compared to the previous implementation
Test setup:
- Using config-pagealloc-micro
- Monitor set to proc-pagetypeinfo, update frequency set to 10ms
- PAGEALLOC_ORDER_MIN=4, PAGEALLOC_ORDER_MAX=4
Without patch test results:
vanilla vanilla
no-monitor monitor
Min alloc-odr4-1 8539.00 ( 0.00%) 8762.00 ( -2.61%)
Min alloc-odr4-2 6501.00 ( 0.00%) 6683.00 ( -2.80%)
Min alloc-odr4-4 5537.00 ( 0.00%) 5873.00 ( -6.07%)
Min alloc-odr4-8 5030.00 ( 0.00%) 5361.00 ( -6.58%)
Min alloc-odr4-16 4782.00 ( 0.00%) 5162.00 ( -7.95%)
Min alloc-odr4-32 5838.00 ( 0.00%) 6499.00 ( -11.32%)
Min alloc-odr4-64 6565.00 ( 0.00%) 7413.00 ( -12.92%)
Min alloc-odr4-128 6896.00 ( 0.00%) 7898.00 ( -14.53%)
Min alloc-odr4-256 7303.00 ( 0.00%) 8163.00 ( -11.78%)
Min alloc-odr4-512 10179.00 ( 0.00%) 11985.00 ( -17.74%)
Min alloc-odr4-1024 11000.00 ( 0.00%) 12165.00 ( -10.59%)
Min free-odr4-1 820.00 ( 0.00%) 1230.00 ( -50.00%)
Min free-odr4-2 511.00 ( 0.00%) 952.00 ( -86.30%)
Min free-odr4-4 347.00 ( 0.00%) 434.00 ( -25.07%)
Min free-odr4-8 286.00 ( 0.00%) 399.00 ( -39.51%)
Min free-odr4-16 250.00 ( 0.00%) 405.00 ( -62.00%)
Min free-odr4-32 294.00 ( 0.00%) 405.00 ( -37.76%)
Min free-odr4-64 333.00 ( 0.00%) 363.00 ( -9.01%)
Min free-odr4-128 340.00 ( 0.00%) 412.00 ( -21.18%)
Min free-odr4-256 339.00 ( 0.00%) 329.00 ( 2.95%)
Min free-odr4-512 361.00 ( 0.00%) 409.00 ( -13.30%)
Min free-odr4-1024 300.00 ( 0.00%) 361.00 ( -20.33%)
Stddev alloc-odr4-1 7.29 ( 0.00%) 90.78 (-1146.00%)
Stddev alloc-odr4-2 3.87 ( 0.00%) 51.30 (-1225.75%)
Stddev alloc-odr4-4 3.20 ( 0.00%) 50.90 (-1491.24%)
Stddev alloc-odr4-8 4.67 ( 0.00%) 52.23 (-1019.35%)
Stddev alloc-odr4-16 5.72 ( 0.00%) 27.53 (-381.04%)
Stddev alloc-odr4-32 6.25 ( 0.00%) 641.23 (-10154.46%)
Stddev alloc-odr4-64 2.06 ( 0.00%) 386.99 (-18714.22%)
Stddev alloc-odr4-128 14.36 ( 0.00%) 52.39 (-264.77%)
Stddev alloc-odr4-256 32.42 ( 0.00%) 326.19 (-906.05%)
Stddev alloc-odr4-512 65.58 ( 0.00%) 184.49 (-181.31%)
Stddev alloc-odr4-1024 8.88 ( 0.00%) 153.01 (-1622.67%)
Stddev free-odr4-1 2.29 ( 0.00%) 152.27 (-6549.85%)
Stddev free-odr4-2 10.99 ( 0.00%) 73.10 (-564.89%)
Stddev free-odr4-4 1.99 ( 0.00%) 28.40 (-1324.45%)
Stddev free-odr4-8 2.51 ( 0.00%) 52.93 (-2007.64%)
Stddev free-odr4-16 2.85 ( 0.00%) 26.04 (-814.88%)
Stddev free-odr4-32 4.04 ( 0.00%) 27.05 (-569.79%)
Stddev free-odr4-64 2.10 ( 0.00%) 48.07 (-2185.66%)
Stddev free-odr4-128 2.63 ( 0.00%) 26.23 (-897.86%)
Stddev free-odr4-256 6.29 ( 0.00%) 37.04 (-488.71%)
Stddev free-odr4-512 2.56 ( 0.00%) 10.65 (-315.28%)
Stddev free-odr4-1024 0.95 ( 0.00%) 6.46 (-582.22%)
Max alloc-odr4-1 8564.00 ( 0.00%) 9099.00 ( -6.25%)
Max alloc-odr4-2 6511.00 ( 0.00%) 6844.00 ( -5.11%)
Max alloc-odr4-4 5549.00 ( 0.00%) 6038.00 ( -8.81%)
Max alloc-odr4-8 5045.00 ( 0.00%) 5551.00 ( -10.03%)
Max alloc-odr4-16 4800.00 ( 0.00%) 5257.00 ( -9.52%)
Max alloc-odr4-32 5861.00 ( 0.00%) 8115.00 ( -38.46%)
Max alloc-odr4-64 6571.00 ( 0.00%) 8292.00 ( -26.19%)
Max alloc-odr4-128 6930.00 ( 0.00%) 8081.00 ( -16.61%)
Max alloc-odr4-256 7372.00 ( 0.00%) 9150.00 ( -24.12%)
Max alloc-odr4-512 10333.00 ( 0.00%) 12636.00 ( -22.29%)
Max alloc-odr4-1024 11035.00 ( 0.00%) 12590.00 ( -14.09%)
Max free-odr4-1 828.00 ( 0.00%) 1724.00 (-108.21%)
Max free-odr4-2 543.00 ( 0.00%) 1192.00 (-119.52%)
Max free-odr4-4 354.00 ( 0.00%) 519.00 ( -46.61%)
Max free-odr4-8 293.00 ( 0.00%) 617.00 (-110.58%)
Max free-odr4-16 260.00 ( 0.00%) 483.00 ( -85.77%)
Max free-odr4-32 308.00 ( 0.00%) 488.00 ( -58.44%)
Max free-odr4-64 341.00 ( 0.00%) 505.00 ( -48.09%)
Max free-odr4-128 346.00 ( 0.00%) 497.00 ( -43.64%)
Max free-odr4-256 353.00 ( 0.00%) 463.00 ( -31.16%)
Max free-odr4-512 367.00 ( 0.00%) 442.00 ( -20.44%)
Max free-odr4-1024 303.00 ( 0.00%) 381.00 ( -25.74%)With patch test results:
patched patched
no-monitor monitor
Min alloc-odr4-1 8488.00 ( 0.00%) 8514.00 ( -0.31%)
Min alloc-odr4-2 6551.00 ( 0.00%) 6527.00 ( 0.37%)
Min alloc-odr4-4 5536.00 ( 0.00%) 5591.00 ( -0.99%)
Min alloc-odr4-8 5008.00 ( 0.00%) 5098.00 ( -1.80%)
Min alloc-odr4-16 4760.00 ( 0.00%) 4857.00 ( -2.04%)
Min alloc-odr4-32 5827.00 ( 0.00%) 5919.00 ( -1.58%)
Min alloc-odr4-64 6561.00 ( 0.00%) 6680.00 ( -1.81%)
Min alloc-odr4-128 6898.00 ( 0.00%) 7014.00 ( -1.68%)
Min alloc-odr4-256 7311.00 ( 0.00%) 7464.00 ( -2.09%)
Min alloc-odr4-512 10181.00 ( 0.00%) 10286.00 ( -1.03%)
Min alloc-odr4-1024 11205.00 ( 0.00%) 11725.00 ( -4.64%)
Min free-odr4-1 789.00 ( 0.00%) 867.00 ( -9.89%)
Min free-odr4-2 490.00 ( 0.00%) 526.00 ( -7.35%)
Min free-odr4-4 350.00 ( 0.00%) 360.00 ( -2.86%)
Min free-odr4-8 272.00 ( 0.00%) 287.00 ( -5.51%)
Min free-odr4-16 247.00 ( 0.00%) 254.00 ( -2.83%)
Min free-odr4-32 298.00 ( 0.00%) 304.00 ( -2.01%)
Min free-odr4-64 334.00 ( 0.00%) 325.00 ( 2.69%)
Min free-odr4-128 334.00 ( 0.00%) 329.00 ( 1.50%)
Min free-odr4-256 336.00 ( 0.00%) 336.00 ( 0.00%)
Min free-odr4-512 360.00 ( 0.00%) 342.00 ( 5.00%)
Min free-odr4-1024 327.00 ( 0.00%) 355.00 ( -8.56%)
Stddev alloc-odr4-1 5.19 ( 0.00%) 45.38 (-775.09%)
Stddev alloc-odr4-2 6.99 ( 0.00%) 37.63 (-437.98%)
Stddev alloc-odr4-4 3.91 ( 0.00%) 17.85 (-356.28%)
Stddev alloc-odr4-8 5.15 ( 0.00%) 9.34 ( -81.47%)
Stddev alloc-odr4-16 3.83 ( 0.00%) 5.34 ( -39.34%)
Stddev alloc-odr4-32 1.96 ( 0.00%) 10.28 (-425.09%)
Stddev alloc-odr4-64 1.32 ( 0.00%) 333.30 (-25141.39%)
Stddev alloc-odr4-128 2.06 ( 0.00%) 7.37 (-258.28%)
Stddev alloc-odr4-256 15.56 ( 0.00%) 113.48 (-629.25%)
Stddev alloc-odr4-512 61.25 ( 0.00%) 165.09 (-169.53%)
Stddev alloc-odr4-1024 18.89 ( 0.00%) 2.93 ( 84.51%)
Stddev free-odr4-1 4.45 ( 0.00%) 40.12 (-800.98%)
Stddev free-odr4-2 1.50 ( 0.00%) 29.30 (-1850.31%)
Stddev free-odr4-4 1.27 ( 0.00%) 19.49 (-1439.40%)
Stddev free-odr4-8 0.97 ( 0.00%) 8.93 (-823.07%)
Stddev free-odr4-16 8.38 ( 0.00%) 4.51 ( 46.21%)
Stddev free-odr4-32 3.18 ( 0.00%) 6.59 (-107.42%)
Stddev free-odr4-64 2.40 ( 0.00%) 3.09 ( -28.50%)
Stddev free-odr4-128 1.55 ( 0.00%) 2.53 ( -62.92%)
Stddev free-odr4-256 0.41 ( 0.00%) 2.80 (-585.57%)
Stddev free-odr4-512 1.60 ( 0.00%) 4.84 (-202.08%)
Stddev free-odr4-1024 0.66 ( 0.00%) 1.19 ( -80.68%)
Max alloc-odr4-1 8505.00 ( 0.00%) 8676.00 ( -2.01%)
Max alloc-odr4-2 6572.00 ( 0.00%) 6651.00 ( -1.20%)
Max alloc-odr4-4 5552.00 ( 0.00%) 5646.00 ( -1.69%)
Max alloc-odr4-8 5024.00 ( 0.00%) 5131.00 ( -2.13%)
Max alloc-odr4-16 4774.00 ( 0.00%) 4875.00 ( -2.12%)
Max alloc-odr4-32 5834.00 ( 0.00%) 5950.00 ( -1.99%)
Max alloc-odr4-64 6565.00 ( 0.00%) 7434.00 ( -13.24%)
Max alloc-odr4-128 6907.00 ( 0.00%) 7034.00 ( -1.84%)
Max alloc-odr4-256 7347.00 ( 0.00%) 7843.00 ( -6.75%)
Max alloc-odr4-512 10315.00 ( 0.00%) 10866.00 ( -5.34%)
Max alloc-odr4-1024 11278.00 ( 0.00%) 11733.00 ( -4.03%)
Max free-odr4-1 803.00 ( 0.00%) 1009.00 ( -25.65%)
Max free-odr4-2 495.00 ( 0.00%) 607.00 ( -22.63%)
Max free-odr4-4 354.00 ( 0.00%) 417.00 ( -17.80%)
Max free-odr4-8 275.00 ( 0.00%) 313.00 ( -13.82%)
Max free-odr4-16 273.00 ( 0.00%) 272.00 ( 0.37%)
Max free-odr4-32 309.00 ( 0.00%) 324.00 ( -4.85%)
Max free-odr4-64 340.00 ( 0.00%) 335.00 ( 1.47%)
Max free-odr4-128 340.00 ( 0.00%) 338.00 ( 0.59%)
Max free-odr4-256 338.00 ( 0.00%) 346.00 ( -2.37%)
Max free-odr4-512 364.00 ( 0.00%) 359.00 ( 1.37%)
Max free-odr4-1024 329.00 ( 0.00%) 359.00 ( -9.12%)The main overhead is a slight increase in latency on the memory allocation and free paths due to additional per-migratetype counting, with theoretically minimal impact on overall performance.
This patch series is based on v6.18-rc7
Hongru Zhang (3):
mm/page_alloc: add per-migratetype counts to buddy allocator mm/vmstat: get fragmentation statistics from per-migragetype count
mm: optimize free_area_empty() check using per-migratetype counts
include/linux/mmzone.h | 1 + mm/internal.h | 2 +- mm/mm_init.c | 1 + mm/page_alloc.c | 9 ++++++++- mm/vmstat.c | 30 +++++++----------------------- 5 files changed, 18 insertions(+), 25 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/3 mm/page_alloc: add per-migratetype counts to buddy allocator
From: Hongru Zhang <[email protected]>
On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.
Under high load scenarios, reading /proc/pagetypeinfo causes memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention
This patch adds per-migratetype counts to the buddy allocator in preparation for optimizing /proc/pagetypeinfo access.
The optimized implementation:
- Make per-migratetype count updates protected by zone lock on the write side while /proc/pagetypeinfo reads are lock-free, which reduces interrupt-disabled spinlock duration and improves system real-time performance (addressing issue #1)
- Reduce blocking time for memory management components when reading /proc/pagetypeinfo, enabling more rapid acquisition of memory fragmentation information (addressing issue #2)
- Minimize the critical section held during /proc/pagetypeinfo reads to reduce zone lock contention on memory allocation and free paths (addressing issue #3)
The main overhead is a slight increase in latency on the memory allocation and free paths due to additional per-migratetype counting, with theoretically minimal impact on overall performance.
--- include/linux/mmzone.h | 1 + mm/mm_init.c | 1 + mm/page_alloc.c | 7 ++++++- 3 files changed, 8 insertions(+), 1 deletion(-)
include/linux/mmzone.h +1 -0
@@ -138,6 +138,7 @@ extern int page_group_by_mobility_disabled;struct free_area {struct list_head free_list[MIGRATE_TYPES ];unsigned long nr_free; +unsigned long mt_nr_free[MIGRATE_TYPES ]; };struct pglist_data ;
mm/mm_init.c +1 -0
@@ -1439,6 +1439,7 @@ static void __meminit zone_init_free_lists(struct zone *zone)for_each_migratetype_order (order, t) {INIT_LIST_HEAD (&zone->free_area[order].free_list[t]); zone->free_area[order].nr_free =0 ; + zone->free_area[order].mt_nr_free[t] =0 ; }#ifdef CONFIG_UNACCEPTED_MEMORY
mm/page_alloc.c +6 -1
@@ -818,6 +818,7 @@ static inline void __add_to_free_list(struct page *page, struct zone *zone,else list_add (&page->buddy_list, &area->free_list[migratetype]); area->nr_free++; + area->mt_nr_free[migratetype]++;if (order >= pageblock_order && !is_migrate_isolate (migratetype))__mod_zone_page_state (zone,NR_FREE_PAGES_BLOCKS , nr_pages);
@@ -840,6 +841,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone, get_pageblock_migratetype(page), old_mt, nr_pages);list_move_tail (&page->buddy_list, &area->free_list[new_mt]); + area->mt_nr_free[old_mt]--; + area->mt_nr_free[new_mt]++;account_freepages (zone, -nr_pages, old_mt);account_freepages (zone, nr_pages, new_mt);
@@ -855,6 +858,7 @@ static inline void move_to_free_list(struct page *page, struct zone *zone,static inline void __del_page_from_free_list (struct page *page,struct zone *zone,unsigned int order,int migratetype) { +struct free_area *area = &zone->free_area[order];int nr_pages =1 << order; VM_WARN_ONCE(get_pageblock_migratetype(page) != migratetype,
@@ -868,7 +872,8 @@ static inline void __del_page_from_free_list(struct page *page, struct zone *zonlist_del (&page->buddy_list);__ClearPageBuddy (page);set_page_private (page,0 ); - zone->free_area[order].nr_free--; + area->nr_free--; + area->mt_nr_free[migratetype]--;if (order >= pageblock_order && !is_migrate_isolate (migratetype))__mod_zone_page_state (zone,NR_FREE_PAGES_BLOCKS , -nr_pages);
Signature · 1 line
2.43.0
Re: [PATCH 1/3] mm/page_alloc: add per-migratetype counts to buddy allocator
Quoted 1 line · Hongru Zhang
[...]
Quoted 13 lines
mm/page_alloc.c +1
-0@@ -818,6 +818,7 @@ static inline void __add_to_free_list(struct page *page, struct zone *zone,else list_add (&page->buddy_list, &area->free_list[migratetype]); area->nr_free++; + area->mt_nr_free[migratetype]++;if (order >= pageblock_order && !is_migrate_isolate (migratetype))__mod_zone_page_state (zone,NR_FREE_PAGES_BLOCKS , nr_pages);
mm/page_alloc.c static inline void move_to_free_list(struct page *page, struct zone *zone, · line 845
Quoted 5 lines
mm/page_alloc.c +2
-0@@ -840,6 +841,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone, get_pageblock_migratetype(page), old_mt, nr_pages);list_move_tail (&page->buddy_list, &area->free_list[new_mt]); + area->mt_nr_free[old_mt]--; + area->mt_nr_free[new_mt]++;
The overhead comes from effectively counting twice. Have we checked whether the readers of area->nr_free are on a hot path? If not, we might just drop nr_free and compute the sum each time.
Buddyinfo and compaction do not seem to be on a hot path ?
Thanks
Barry
PATCH 2/3 mm/vmstat: get fragmentation statistics from per-migragetype count
In reply to Hongru ZhangFrom: Hongru Zhang <[email protected]>
This patch optimizes /proc/pagetypeinfo access by utilizing the per-migratetype free page block counts already maintained by the buddy allocator, instead of iterating through free lists under zone lock.
Accuracy. Both implementations have accuracy limitations. The previous implementation required acquiring and releasing the zone lock for counting each order and migratetype, making it potentially inaccurate. Under high memory pressure, accuracy would further degrade due to zone lock contention or fragmentation. The new implementation collects data within a short time window, which helps maintain relatively small errors, and is unaffected by memory pressure. Furthermore, user-space memory management components inherently experience decision latency - by the time they process the collected data and execute actions, the memory state has already changed. This means that even perfectly accurate data at collection time becomes stale by decision time. Considering these factors, the accuracy trade-off introduced by the new implementation should be acceptable for practical use cases, offering a balance between performance and accuracy requirements.
Performance benefits:
System setup:
- 12th Gen Intel(R) Core(TM) i7-12700
- 1 NUMA node, 16G memory in total
- Turbo disabled
- cpufreq governor set to performance
1. Average latency over 10,000 /proc/pagetypeinfo accesses
+-----------------------+----------+------------+ | | no-patch | with-patch | +-----------------------+----------+------------+ | Just after boot | 700.9 us | 268.6 us | +-----------------------+----------+------------+ | After building kernel | 28.7 ms | 269.8 us | +-----------------------+----------+------------+
2. Page alloc/free latency with concurrent /proc/pagetypeinfo access Test setup:
- Using config-pagealloc-micro
- Monitor set to proc-pagetypeinfo, update frequency set to 10ms
- PAGEALLOC_ORDER_MIN=4, PAGEALLOC_ORDER_MAX=4
Without patch test results:
vanilla vanilla
no-monitor monitor
Min alloc-odr4-1 8539.00 ( 0.00%) 8762.00 ( -2.61%)
Min alloc-odr4-2 6501.00 ( 0.00%) 6683.00 ( -2.80%)
Min alloc-odr4-4 5537.00 ( 0.00%) 5873.00 ( -6.07%)
Min alloc-odr4-8 5030.00 ( 0.00%) 5361.00 ( -6.58%)
Min alloc-odr4-16 4782.00 ( 0.00%) 5162.00 ( -7.95%)
Min alloc-odr4-32 5838.00 ( 0.00%) 6499.00 ( -11.32%)
Min alloc-odr4-64 6565.00 ( 0.00%) 7413.00 ( -12.92%)
Min alloc-odr4-128 6896.00 ( 0.00%) 7898.00 ( -14.53%)
Min alloc-odr4-256 7303.00 ( 0.00%) 8163.00 ( -11.78%)
Min alloc-odr4-512 10179.00 ( 0.00%) 11985.00 ( -17.74%)
Min alloc-odr4-1024 11000.00 ( 0.00%) 12165.00 ( -10.59%)
Min free-odr4-1 820.00 ( 0.00%) 1230.00 ( -50.00%)
Min free-odr4-2 511.00 ( 0.00%) 952.00 ( -86.30%)
Min free-odr4-4 347.00 ( 0.00%) 434.00 ( -25.07%)
Min free-odr4-8 286.00 ( 0.00%) 399.00 ( -39.51%)
Min free-odr4-16 250.00 ( 0.00%) 405.00 ( -62.00%)
Min free-odr4-32 294.00 ( 0.00%) 405.00 ( -37.76%)
Min free-odr4-64 333.00 ( 0.00%) 363.00 ( -9.01%)
Min free-odr4-128 340.00 ( 0.00%) 412.00 ( -21.18%)
Min free-odr4-256 339.00 ( 0.00%) 329.00 ( 2.95%)
Min free-odr4-512 361.00 ( 0.00%) 409.00 ( -13.30%)
Min free-odr4-1024 300.00 ( 0.00%) 361.00 ( -20.33%)
Stddev alloc-odr4-1 7.29 ( 0.00%) 90.78 (-1146.00%)
Stddev alloc-odr4-2 3.87 ( 0.00%) 51.30 (-1225.75%)
Stddev alloc-odr4-4 3.20 ( 0.00%) 50.90 (-1491.24%)
Stddev alloc-odr4-8 4.67 ( 0.00%) 52.23 (-1019.35%)
Stddev alloc-odr4-16 5.72 ( 0.00%) 27.53 (-381.04%)
Stddev alloc-odr4-32 6.25 ( 0.00%) 641.23 (-10154.46%)
Stddev alloc-odr4-64 2.06 ( 0.00%) 386.99 (-18714.22%)
Stddev alloc-odr4-128 14.36 ( 0.00%) 52.39 (-264.77%)
Stddev alloc-odr4-256 32.42 ( 0.00%) 326.19 (-906.05%)
Stddev alloc-odr4-512 65.58 ( 0.00%) 184.49 (-181.31%)
Stddev alloc-odr4-1024 8.88 ( 0.00%) 153.01 (-1622.67%)
Stddev free-odr4-1 2.29 ( 0.00%) 152.27 (-6549.85%)
Stddev free-odr4-2 10.99 ( 0.00%) 73.10 (-564.89%)
Stddev free-odr4-4 1.99 ( 0.00%) 28.40 (-1324.45%)
Stddev free-odr4-8 2.51 ( 0.00%) 52.93 (-2007.64%)
Stddev free-odr4-16 2.85 ( 0.00%) 26.04 (-814.88%)
Stddev free-odr4-32 4.04 ( 0.00%) 27.05 (-569.79%)
Stddev free-odr4-64 2.10 ( 0.00%) 48.07 (-2185.66%)
Stddev free-odr4-128 2.63 ( 0.00%) 26.23 (-897.86%)
Stddev free-odr4-256 6.29 ( 0.00%) 37.04 (-488.71%)
Stddev free-odr4-512 2.56 ( 0.00%) 10.65 (-315.28%)
Stddev free-odr4-1024 0.95 ( 0.00%) 6.46 (-582.22%)
Max alloc-odr4-1 8564.00 ( 0.00%) 9099.00 ( -6.25%)
Max alloc-odr4-2 6511.00 ( 0.00%) 6844.00 ( -5.11%)
Max alloc-odr4-4 5549.00 ( 0.00%) 6038.00 ( -8.81%)
Max alloc-odr4-8 5045.00 ( 0.00%) 5551.00 ( -10.03%)
Max alloc-odr4-16 4800.00 ( 0.00%) 5257.00 ( -9.52%)
Max alloc-odr4-32 5861.00 ( 0.00%) 8115.00 ( -38.46%)
Max alloc-odr4-64 6571.00 ( 0.00%) 8292.00 ( -26.19%)
Max alloc-odr4-128 6930.00 ( 0.00%) 8081.00 ( -16.61%)
Max alloc-odr4-256 7372.00 ( 0.00%) 9150.00 ( -24.12%)
Max alloc-odr4-512 10333.00 ( 0.00%) 12636.00 ( -22.29%)
Max alloc-odr4-1024 11035.00 ( 0.00%) 12590.00 ( -14.09%)
Max free-odr4-1 828.00 ( 0.00%) 1724.00 (-108.21%)
Max free-odr4-2 543.00 ( 0.00%) 1192.00 (-119.52%)
Max free-odr4-4 354.00 ( 0.00%) 519.00 ( -46.61%)
Max free-odr4-8 293.00 ( 0.00%) 617.00 (-110.58%)
Max free-odr4-16 260.00 ( 0.00%) 483.00 ( -85.77%)
Max free-odr4-32 308.00 ( 0.00%) 488.00 ( -58.44%)
Max free-odr4-64 341.00 ( 0.00%) 505.00 ( -48.09%)
Max free-odr4-128 346.00 ( 0.00%) 497.00 ( -43.64%)
Max free-odr4-256 353.00 ( 0.00%) 463.00 ( -31.16%)
Max free-odr4-512 367.00 ( 0.00%) 442.00 ( -20.44%)
Max free-odr4-1024 303.00 ( 0.00%) 381.00 ( -25.74%)With patch test results:
patched patched
no-monitor monitor
Min alloc-odr4-1 8488.00 ( 0.00%) 8514.00 ( -0.31%)
Min alloc-odr4-2 6551.00 ( 0.00%) 6527.00 ( 0.37%)
Min alloc-odr4-4 5536.00 ( 0.00%) 5591.00 ( -0.99%)
Min alloc-odr4-8 5008.00 ( 0.00%) 5098.00 ( -1.80%)
Min alloc-odr4-16 4760.00 ( 0.00%) 4857.00 ( -2.04%)
Min alloc-odr4-32 5827.00 ( 0.00%) 5919.00 ( -1.58%)
Min alloc-odr4-64 6561.00 ( 0.00%) 6680.00 ( -1.81%)
Min alloc-odr4-128 6898.00 ( 0.00%) 7014.00 ( -1.68%)
Min alloc-odr4-256 7311.00 ( 0.00%) 7464.00 ( -2.09%)
Min alloc-odr4-512 10181.00 ( 0.00%) 10286.00 ( -1.03%)
Min alloc-odr4-1024 11205.00 ( 0.00%) 11725.00 ( -4.64%)
Min free-odr4-1 789.00 ( 0.00%) 867.00 ( -9.89%)
Min free-odr4-2 490.00 ( 0.00%) 526.00 ( -7.35%)
Min free-odr4-4 350.00 ( 0.00%) 360.00 ( -2.86%)
Min free-odr4-8 272.00 ( 0.00%) 287.00 ( -5.51%)
Min free-odr4-16 247.00 ( 0.00%) 254.00 ( -2.83%)
Min free-odr4-32 298.00 ( 0.00%) 304.00 ( -2.01%)
Min free-odr4-64 334.00 ( 0.00%) 325.00 ( 2.69%)
Min free-odr4-128 334.00 ( 0.00%) 329.00 ( 1.50%)
Min free-odr4-256 336.00 ( 0.00%) 336.00 ( 0.00%)
Min free-odr4-512 360.00 ( 0.00%) 342.00 ( 5.00%)
Min free-odr4-1024 327.00 ( 0.00%) 355.00 ( -8.56%)
Stddev alloc-odr4-1 5.19 ( 0.00%) 45.38 (-775.09%)
Stddev alloc-odr4-2 6.99 ( 0.00%) 37.63 (-437.98%)
Stddev alloc-odr4-4 3.91 ( 0.00%) 17.85 (-356.28%)
Stddev alloc-odr4-8 5.15 ( 0.00%) 9.34 ( -81.47%)
Stddev alloc-odr4-16 3.83 ( 0.00%) 5.34 ( -39.34%)
Stddev alloc-odr4-32 1.96 ( 0.00%) 10.28 (-425.09%)
Stddev alloc-odr4-64 1.32 ( 0.00%) 333.30 (-25141.39%)
Stddev alloc-odr4-128 2.06 ( 0.00%) 7.37 (-258.28%)
Stddev alloc-odr4-256 15.56 ( 0.00%) 113.48 (-629.25%)
Stddev alloc-odr4-512 61.25 ( 0.00%) 165.09 (-169.53%)
Stddev alloc-odr4-1024 18.89 ( 0.00%) 2.93 ( 84.51%)
Stddev free-odr4-1 4.45 ( 0.00%) 40.12 (-800.98%)
Stddev free-odr4-2 1.50 ( 0.00%) 29.30 (-1850.31%)
Stddev free-odr4-4 1.27 ( 0.00%) 19.49 (-1439.40%)
Stddev free-odr4-8 0.97 ( 0.00%) 8.93 (-823.07%)
Stddev free-odr4-16 8.38 ( 0.00%) 4.51 ( 46.21%)
Stddev free-odr4-32 3.18 ( 0.00%) 6.59 (-107.42%)
Stddev free-odr4-64 2.40 ( 0.00%) 3.09 ( -28.50%)
Stddev free-odr4-128 1.55 ( 0.00%) 2.53 ( -62.92%)
Stddev free-odr4-256 0.41 ( 0.00%) 2.80 (-585.57%)
Stddev free-odr4-512 1.60 ( 0.00%) 4.84 (-202.08%)
Stddev free-odr4-1024 0.66 ( 0.00%) 1.19 ( -80.68%)
Max alloc-odr4-1 8505.00 ( 0.00%) 8676.00 ( -2.01%)
Max alloc-odr4-2 6572.00 ( 0.00%) 6651.00 ( -1.20%)
Max alloc-odr4-4 5552.00 ( 0.00%) 5646.00 ( -1.69%)
Max alloc-odr4-8 5024.00 ( 0.00%) 5131.00 ( -2.13%)
Max alloc-odr4-16 4774.00 ( 0.00%) 4875.00 ( -2.12%)
Max alloc-odr4-32 5834.00 ( 0.00%) 5950.00 ( -1.99%)
Max alloc-odr4-64 6565.00 ( 0.00%) 7434.00 ( -13.24%)
Max alloc-odr4-128 6907.00 ( 0.00%) 7034.00 ( -1.84%)
Max alloc-odr4-256 7347.00 ( 0.00%) 7843.00 ( -6.75%)
Max alloc-odr4-512 10315.00 ( 0.00%) 10866.00 ( -5.34%)
Max alloc-odr4-1024 11278.00 ( 0.00%) 11733.00 ( -4.03%)
Max free-odr4-1 803.00 ( 0.00%) 1009.00 ( -25.65%)
Max free-odr4-2 495.00 ( 0.00%) 607.00 ( -22.63%)
Max free-odr4-4 354.00 ( 0.00%) 417.00 ( -17.80%)
Max free-odr4-8 275.00 ( 0.00%) 313.00 ( -13.82%)
Max free-odr4-16 273.00 ( 0.00%) 272.00 ( 0.37%)
Max free-odr4-32 309.00 ( 0.00%) 324.00 ( -4.85%)
Max free-odr4-64 340.00 ( 0.00%) 335.00 ( 1.47%)
Max free-odr4-128 340.00 ( 0.00%) 338.00 ( 0.59%)
Max free-odr4-256 338.00 ( 0.00%) 346.00 ( -2.37%)
Max free-odr4-512 364.00 ( 0.00%) 359.00 ( 1.37%)
Max free-odr4-1024 329.00 ( 0.00%) 359.00 ( -9.12%)--- mm/page_alloc.c | 10 ++++++---- mm/vmstat.c | 30 +++++++----------------------- 2 files changed, 13 insertions(+), 27 deletions(-)
mm/page_alloc.c +6 -4
@@ -818,7 +818,8 @@ static inline void __add_to_free_list(struct page *page, struct zone *zone,else list_add (&page->buddy_list, &area->free_list[migratetype]); area->nr_free++; - area->mt_nr_free[migratetype]++; +WRITE_ONCE (area->mt_nr_free[migratetype], + area->mt_nr_free[migratetype] +1 );if (order >= pageblock_order && !is_migrate_isolate (migratetype))__mod_zone_page_state (zone,NR_FREE_PAGES_BLOCKS , nr_pages);
@@ -841,8 +842,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone, get_pageblock_migratetype(page), old_mt, nr_pages);list_move_tail (&page->buddy_list, &area->free_list[new_mt]); - area->mt_nr_free[old_mt]--; - area->mt_nr_free[new_mt]++; +WRITE_ONCE (area->mt_nr_free[old_mt], area->mt_nr_free[old_mt] -1 ); +WRITE_ONCE (area->mt_nr_free[new_mt], area->mt_nr_free[new_mt] +1 );account_freepages (zone, -nr_pages, old_mt);account_freepages (zone, nr_pages, new_mt);
@@ -873,7 +874,8 @@ static inline void __del_page_from_free_list(struct page *page, struct zone *zon__ClearPageBuddy (page);set_page_private (page,0 ); area->nr_free--; - area->mt_nr_free[migratetype]--; +WRITE_ONCE (area->mt_nr_free[migratetype], + area->mt_nr_free[migratetype] -1 );if (order >= pageblock_order && !is_migrate_isolate (migratetype))__mod_zone_page_state (zone,NR_FREE_PAGES_BLOCKS , -nr_pages);
mm/vmstat.c +7 -23
@@ -1590,32 +1590,16 @@ static void pagetypeinfo_showfree_print(struct seq_file *m, zone->name, migratetype_names[mtype]);for (order =0 ; order <NR_PAGE_ORDERS ; ++order) { -unsigned long freecount =0 ; -struct free_area *area; -struct list_head *curr; +unsigned long freecount;bool overflow = false; - area = &(zone->free_area[order]); - -list_for_each (curr, &area->free_list[mtype]) { -/* -* Cap the free_list iteration because it might -* be really large and we are under a spinlock -* so a long time spent here could trigger a -* hard lockup detector. Anyway this is a -* debugging tool so knowing there is a handful -* of pages of this order should be more than -* sufficient. -*/ -if (++freecount >=100000 ) { - overflow = true; -break ; - } +/* Keep the same output format for user-space tools compatibility */ + freecount =READ_ONCE (zone->free_area[order].mt_nr_free[mtype]); +if (freecount >=100000 ) { + overflow = true; + freecount =100000 ; }seq_printf (m,"%s%6lu " , overflow ?">" :"" , freecount); -spin_unlock_irq (&zone->lock); -cond_resched (); -spin_lock_irq (&zone->lock); }seq_putc (m,'\n' ); }
@@ -1633,7 +1617,7 @@ static void pagetypeinfo_showfree(struct seq_file *m, void *arg)seq_printf (m,"%6d " , order);seq_putc (m,'\n' ); -walk_zones_in_node (m, pgdat, true, false, pagetypeinfo_showfree_print); +walk_zones_in_node (m, pgdat, true, true, pagetypeinfo_showfree_print); } static void pagetypeinfo_showblockcount_print(struct seq_file *m,
Signature · 1 line
2.43.0
Re: [PATCH 2/3] mm/vmstat: get fragmentation statistics from per-migragetype count
Hi, Hongru
Quoted 26 lines
mm/page_alloc.c +4
-3@@ -818,7 +818,8 @@ static inline void __add_to_free_list(struct page *page, struct zone *zone,else list_add (&page->buddy_list, &area->free_list[migratetype]); area->nr_free++; - area->mt_nr_free[migratetype]++; +WRITE_ONCE (area->mt_nr_free[migratetype], + area->mt_nr_free[migratetype] +1 );if (order >= pageblock_order && !is_migrate_isolate (migratetype))__mod_zone_page_state (zone,NR_FREE_PAGES_BLOCKS , nr_pages);@@ -841,8 +842,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone, get_pageblock_migratetype(page), old_mt, nr_pages);list_move_tail (&page->buddy_list, &area->free_list[new_mt]); - area->mt_nr_free[old_mt]--; - area->mt_nr_free[new_mt]++; +WRITE_ONCE (area->mt_nr_free[old_mt], area->mt_nr_free[old_mt] -1 ); +WRITE_ONCE (area->mt_nr_free[new_mt], area->mt_nr_free[new_mt] +1 );account_freepages (zone, -nr_pages, old_mt);account_freepages (zone, nr_pages, new_mt);
mm/page_alloc.c static inline void __del_page_from_free_list(struct page *page, struct zone *zon · line 878
Quoted 6 lines
mm/page_alloc.c +2
-1@@ -873,7 +874,8 @@ static inline void __del_page_from_free_list(struct page *page, struct zone *zon__ClearPageBuddy (page);set_page_private (page,0 ); area->nr_free--; - area->mt_nr_free[migratetype]--; +WRITE_ONCE (area->mt_nr_free[migratetype], + area->mt_nr_free[migratetype] -1 );
It doesn't seem like a good idea to use WRITE_ONCE on the hot path.
Quoted 25 lines
mm/page_alloc.c +0
-0if (order >= pageblock_order && !is_migrate_isolate (migratetype))__mod_zone_page_state (zone,NR_FREE_PAGES_BLOCKS , -nr_pages);mm/vmstat.c +1
-11@@ -1590,32 +1590,16 @@ static void pagetypeinfo_showfree_print(struct seq_file *m, zone->name, migratetype_names[mtype]);for (order =0 ; order <NR_PAGE_ORDERS ; ++order) { -unsigned long freecount =0 ; -struct free_area *area; -struct list_head *curr; +unsigned long freecount;bool overflow = false; - area = &(zone->free_area[order]); - -list_for_each (curr, &area->free_list[mtype]) { -/* -* Cap the free_list iteration because it might -* be really large and we are under a spinlock -* so a long time spent here could trigger a -* hard lockup detector. Anyway this is a
mm/vmstat.c static void pagetypeinfo_showfree_print(struct seq_file *m, · line 1597
Quoted 10 lines
mm/vmstat.c +2
-8-* debugging tool so knowing there is a handful -* of pages of this order should be more than -* sufficient. -*/ -if (++freecount >=100000 ) { - overflow = true; -break ; - } +/* Keep the same output format for user-space tools compatibility */ + freecount =READ_ONCE (zone->free_area[order].mt_nr_free[mtype]);
I think it might be better for using an array of size NR_PAGE_ORDERS to store the free count for each order. Like the code below.
unsigned long freecount[NR_PAGE_ORDERS]
spin_lock_irq(&zone->lock)
for_each_order freecount[order] = zone->free_area[order].mt_nr_free[mtype]
spin_unlock_irq(&zone->lock)
for_each_order
print freecount[order]
Quoted 18 lines
mm/vmstat.c +4
-4+if (freecount >=100000 ) { + overflow = true; + freecount =100000 ; }seq_printf (m,"%s%6lu " , overflow ?">" :"" , freecount); -spin_unlock_irq (&zone->lock); -cond_resched (); -spin_lock_irq (&zone->lock); }seq_putc (m,'\n' ); }@@ -1633,7 +1617,7 @@ static void pagetypeinfo_showfree(struct seq_file *m, void *arg)seq_printf (m,"%6d " , order);seq_putc (m,'\n' ); -walk_zones_in_node (m, pgdat, true, false, pagetypeinfo_showfree_print); +walk_zones_in_node (m, pgdat, true, true, pagetypeinfo_showfree_print); }
Re: [PATCH 2/3] mm/vmstat: get fragmentation statistics from per-migragetype count
Quoted 29 lines
Quoted 29 lines
if (order >= pageblock_order && !is_migrate_isolate(migratetype)) __mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, -nr_pages);mm/vmstat.c +1
-16@@ -1590,32 +1590,16 @@ static void pagetypeinfo_showfree_print(struct seq_file *m, zone->name, migratetype_names[mtype]);for (order =0 ; order <NR_PAGE_ORDERS ; ++order) { -unsigned long freecount =0 ; -struct free_area *area; -struct list_head *curr; +unsigned long freecount;bool overflow = false; - area = &(zone->free_area[order]); - -list_for_each (curr, &area->free_list[mtype]) { -/* -* Cap the free_list iteration because it might -* be really large and we are under a spinlock -* so a long time spent here could trigger a -* hard lockup detector. Anyway this is a -* debugging tool so knowing there is a handful -* of pages of this order should be more than -* sufficient. -*/ -if (++freecount >=100000 ) {
Quoted 8 lines
mm/vmstat.c static void pagetypeinfo_showfree_print(struct seq_file *m, · line 1597
Quoted 5 lines
mm/vmstat.c +2
-3- overflow = true; -break ; - } +/* Keep the same output format for user-space tools compatibility */ + freecount =READ_ONCE (zone->free_area[order].mt_nr_free[mtype]);I think it might be better for using an array of size
NR_PAGE_ORDERSto store the free count for each order. Like the code below.
Right. If we want the freecount to accurately reflect the current system state, we still need to take the zone lock.
Multiple independent WRITE_ONCE and READ_ONCE operations do not guarantee correctness. They may ensure single-copy atomicity per access, but not for the overall result.
Thanks
Barry
Re: [PATCH 2/3] mm/vmstat: get fragmentation statistics from per-migragetype count
Quoted 39 lines · Barry Song
Quoted 30 lines
Quoted 30 lines
if (order >= pageblock_order && !is_migrate_isolate(migratetype)) __mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, -nr_pages);mm/vmstat.c +1
-16@@ -1590,32 +1590,16 @@ static void pagetypeinfo_showfree_print(struct seq_file *m, zone->name, migratetype_names[mtype]);for (order =0 ; order <NR_PAGE_ORDERS ; ++order) { -unsigned long freecount =0 ; -struct free_area *area; -struct list_head *curr; +unsigned long freecount;bool overflow = false; - area = &(zone->free_area[order]); - -list_for_each (curr, &area->free_list[mtype]) { -/* -* Cap the free_list iteration because it might -* be really large and we are under a spinlock -* so a long time spent here could trigger a -* hard lockup detector. Anyway this is a -* debugging tool so knowing there is a handful -* of pages of this order should be more than -* sufficient. -*/ -if (++freecount >=100000 ) {Quoted 8 lines
mm/vmstat.c static void pagetypeinfo_showfree_print(struct seq_file *m, · line 1597
Quoted 5 lines
mm/vmstat.c +2
-3- overflow = true; -break ; - } +/* Keep the same output format for user-space tools compatibility */ + freecount =READ_ONCE (zone->free_area[order].mt_nr_free[mtype]);I think it might be better for using an array of size
NR_PAGE_ORDERSto store the free count for each order. Like the code below.
Quoted 6 lines
Right. If we want the freecount to accurately reflect the current system state, we still need to take the zone lock.
Multiple independent
WRITE_ONCEandREAD_ONCEoperations do not guarantee correctness. They may ensure single-copy atomicity per access, but not for the overall result.
On second thought, the original code releases and re-acquires the spinlock for each order, so cross-variable consistency may not be a real issue. Adding data_race() to silence KCSAN warnings should be sufficient? I mean something like the following.
@@ -843,8 +842,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone,
get_pageblock_migratetype(page), old_mt, nr_pages);
list_move_tail(&page->buddy_list, &area->free_list[new_mt]);
- WRITE_ONCE(area->mt_nr_free[old_mt], area->mt_nr_free[old_mt] - 1);
- WRITE_ONCE(area->mt_nr_free[new_mt], area->mt_nr_free[new_mt] + 1);
+ area->mt_nr_free[old_mt]--;
+ area->mt_nr_free[new_mt]++; account_freepages(zone, -nr_pages, old_mt);
account_freepages(zone, nr_pages, new_mt);
@@ -875,8 +874,7 @@ static inline void
__del_page_from_free_list(struct page *page, struct zone *zon
__ClearPageBuddy(page);
set_page_private(page, 0);
area->nr_free--;- WRITE_ONCE(area->mt_nr_free[migratetype],
- area->mt_nr_free[migratetype] - 1);
+ area->mt_nr_free[migratetype]--;
if (order >= pageblock_order && !is_migrate_isolate(migratetype))
__mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, -nr_pages);mm/vmstat.c +0 -0
@@ -1599,7 +1599,7 @@ static void pagetypeinfo_showfree_print(struct
seq_file *m,
bool overflow = false;
/* Keep the same output format for user-spacetools compatibility */
- freecount =READ_ONCE(zone->free_area[order].mt_nr_free[mtype]);
+ freecount =
data_race(zone->free_area[order].mt_nr_free[mtype]);
if (freecount >= 100000) {
overflow = true;
freecount = 100000;Thanks
Barry
Re: [PATCH 2/3] mm/vmstat: get fragmentation statistics from per-migragetype count
In reply to Barry SongQuoted 2 lines
Right. If we want the freecount to accurately reflect the current system state, we still need to take the zone lock.
Yeah, as I mentioned in patch (2/3), this implementation has accuracy limitation:
"Accuracy. Both implementations have accuracy limitations. The previous implementation required acquiring and releasing the zone lock for counting each order and migratetype, making it potentially inaccurate. Under high memory pressure, accuracy would further degrade due to zone lock contention or fragmentation. The new implementation collects data within a short time window, which helps maintain relatively small errors, and is unaffected by memory pressure. Furthermore, user-space memory management components inherently experience decision latency - by the time they process the collected data and execute actions, the memory state has already changed. This means that even perfectly accurate data at collection time becomes stale by decision time. Considering these factors, the accuracy trade-off introduced by the new implementation should be acceptable for practical use cases, offering a balance between performance and accuracy requirements."
Additional data:
1. average latency of pagetypeinfo_showfree_print() over 1,000,000 times is 4.67 us
2. average latency is 125 ns, if seq_printf() is taken out of the loop
Example code:
+unsigned long total_lat = 0;
+unsigned long total_count = 0;
+
static void pagetypeinfo_showfree_print(struct seq_file *m,
pg_data_t *pgdat, struct zone *zone)
{
int order, mtype;
+ ktime_t start;
+ u64 lat;
+ unsigned long freecounts[NR_PAGE_ORDERS][MIGRATE_TYPES]; /* ignore potential stack overflow */+
+ start = ktime_get();
+ for (order = 0; order < NR_PAGE_ORDERS; ++order)
+ for (mtype = 0; mtype < MIGRATE_TYPES; mtype++)
+ freecounts[order][mtype] = READ_ONCE(zone->free_area[order].mt_nr_free[mtype]);
+
+ lat = ktime_to_ns(ktime_sub(ktime_get(), start)); + total_count++; + total_lat += lat;
for (mtype = 0; mtype < MIGRATE_TYPES; mtype++) {
seq_printf(m, "Node %4d, zone %8s, type %12s ",
@@ -1594,7 +1609,7 @@ static void pagetypeinfo_showfree_print(struct seq_file *m,
bool overflow = false;
/* Keep the same output format for user-space tools compatibility */
- freecount = READ_ONCE(zone->free_area[order].mt_nr_free[mtype]);
+ freecount = freecounts[order][mtype];
if (freecount >= 100000) {
overflow = true;
freecount = 100000;@@ -1692,6 +1707,13 @@ static void pagetypeinfo_showmixedcount(struct seq_file *m, pg_data_t *pgdat)
#endif /* CONFIG_PAGE_OWNER */
}
I think both are small time window (if IRQ is disabled, latency is more deterministic).
Quoted 3 lines
Multiple independent
WRITE_ONCEandREAD_ONCEoperations do not guarantee correctness. They may ensure single-copy atomicity per access, but not for the overall result.
I know this does not guarantee correctness of the overall result. READ_ONCE() and WRITE_ONCE() in this patch are used to avoid potential store tearing and read tearing caused by compiler optimizations.
In fact, I have already noticed /proc/buddyinfo, which collects data under zone lock and uses data_race to avoid KCSAN reports. But I'm wondering if we could remove its zone lock as well, for the same reasons as /proc/pagetypeinfo.
Re: [PATCH 2/3] mm/vmstat: get fragmentation statistics from per-migragetype count
Quoted 68 lines · Hongru Zhang
Quoted 3 lines
Right. If we want the freecount to accurately reflect the current system state, we still need to take the zone lock.
Yeah, as I mentioned in patch (2/3), this implementation has accuracy limitation:
"Accuracy. Both implementations have accuracy limitations. The previous implementation required acquiring and releasing the zone lock for counting each order and migratetype, making it potentially inaccurate. Under high memory pressure, accuracy would further degrade due to zone lock contention or fragmentation. The new implementation collects data within a short time window, which helps maintain relatively small errors, and is unaffected by memory pressure. Furthermore, user-space memory management components inherently experience decision latency - by the time they process the collected data and execute actions, the memory state has already changed. This means that even perfectly accurate data at collection time becomes stale by decision time. Considering these factors, the accuracy trade-off introduced by the new implementation should be acceptable for practical use cases, offering a balance between performance and accuracy requirements."
Additional data:
1. average latency ofpagetypeinfo_showfree_print()over 1,000,000 times is 4.67 us2. average latency is 125 ns, if
seq_printf()is taken out of the loopExample code:
(quoted diff) +16
-1+unsigned long total_lat = 0; +unsigned long total_count = 0; + static void pagetypeinfo_showfree_print(struct seq_file *m, pg_data_t *pgdat, struct zone *zone) { int order, mtype; + ktime_t start; + u64 lat; + unsigned long freecounts[NR_PAGE_ORDERS][MIGRATE_TYPES]; /* ignore potential stack overflow */ + + start = ktime_get(); + for (order = 0; order < NR_PAGE_ORDERS; ++order) + for (mtype = 0; mtype < MIGRATE_TYPES; mtype++) + freecounts[order][mtype] = READ_ONCE(zone->free_area[order].mt_nr_free[mtype]); + + lat = ktime_to_ns(ktime_sub(ktime_get(), start)); + total_count++; + total_lat += lat; for (mtype = 0; mtype < MIGRATE_TYPES; mtype++) { seq_printf(m, "Node %4d, zone %8s, type %12s ",@@ -1594,7 +1609,7 @@ static void pagetypeinfo_showfree_print(struct seq_file *m, bool overflow = false; /* Keep the same output format for user-space tools compatibility */ - freecount = READ_ONCE(zone->free_area[order].mt_nr_free[mtype]); + freecount = freecounts[order][mtype]; if (freecount >= 100000) { overflow = true; freecount = 100000;@@ -1692,6 +1707,13 @@ static void pagetypeinfo_showmixedcount(struct seq_file *m, pg_data_t *pgdat) #endif /* CONFIG_PAGE_OWNER */ }I think both are small time window (if IRQ is disabled, latency is more deterministic).
Quoted 7 lines
Quoted 3 lines
Multiple independent
WRITE_ONCEandREAD_ONCEoperations do not guarantee correctness. They may ensure single-copy atomicity per access, but not for the overall result.I know this does not guarantee correctness of the overall result.
READ_ONCE()andWRITE_ONCE()in this patch are used to avoid potential store tearing and read tearing caused by compiler optimizations.
Yes, I realized that correctness might not be a major concern, so I sent a follow-up email [1] after replying to you.
Quoted 5 lines
In fact, I have already noticed
/proc/buddyinfo, which collects data under zone lock and usesdata_raceto avoid KCSAN reports. But I'm wondering if we could remove its zone lock as well, for the same reasons as/proc/pagetypeinfo.
That might be correct. However, if it doesn’t significantly affect performance and buddyinfo is accessed much less frequently than the buddy list, we may just leave it as is.
[1] https://lore.kernel.org/linux-mm/CAGsJ_4wUQdQyB_3y0Buf3uG34hvgpMAP3qHHwJM3=R01RJOuvw@mail.gmail.com/
Thanks
Barry
PATCH 3/3 mm: optimize free_area_empty() check using per-migratetype counts
In reply to Hongru ZhangFrom: Hongru Zhang <[email protected]>
Use per-migratetype counts instead of list_empty() helps reduce a few cpu instructions.
--- mm/internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/internal.h +1 -1
@@ -954,7 +954,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,static inline bool free_area_empty (struct free_area *area,int migratetype) { -return list_empty (&area->free_list[migratetype]); +return !READ_ONCE (area->mt_nr_free[migratetype]); }/* mm/util.c */
Signature · 1 line
2.43.0
Re: [PATCH 3/3] mm: optimize free_area_empty() check using per-migratetype counts
Quoted 16 lines · Hongru Zhang
From: Hongru Zhang <[email protected]>
Use per-migratetype counts instead of
list_empty()helps reduce a few cpu instructions.Signed-off-by Hongru Zhang <[email protected]>--- mm/internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/internal.h +0
-0
mm/internal.h int find_suitable_fallback(struct free_area *area, unsigned int order, · line 957
Quoted 5 lines
mm/internal.h +1
-1@@ -954,7 +954,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,static inline bool free_area_empty (struct free_area *area,int migratetype) { -return list_empty (&area->free_list[migratetype]); +return !READ_ONCE (area->mt_nr_free[migratetype]);
I'm not quite sure about this. Since the counter is written and read more frequently, cache coherence traffic may actually be higher than for the list head.
I'd prefer to drop this unless there is real data showing it performs better.
Thanks
Barry
Re: [PATCH 3/3] mm: optimize free_area_empty() check using per-migratetype counts
Quoted 28 lines · Barry Song
Quoted 16 lines · Hongru Zhang
From: Hongru Zhang <[email protected]>
Use per-migratetype counts instead of
list_empty()helps reduce a few cpu instructions.Signed-off-by Hongru Zhang <[email protected]>--- mm/internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/internal.h +0
-0mm/internal.h int find_suitable_fallback(struct free_area *area, unsigned int order, · line 957
Quoted 5 lines
mm/internal.h +1
-1@@ -954,7 +954,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,static inline bool free_area_empty (struct free_area *area,int migratetype) { -return list_empty (&area->free_list[migratetype]); +return !READ_ONCE (area->mt_nr_free[migratetype]);I'm not quite sure about this. Since the counter is written and read more frequently, cache coherence traffic may actually be higher than for the list head.
Quoted 1 line
I'd prefer to drop this unless there is real data showing it performs better.
If the goal is to optimize free_area list checks and list_add, a reasonable approach is to organize the data structure to reduce false sharing between different mt and order entries.
struct mt_free_area {
struct list_head free_list;
unsigned long nr_free;
} ____cacheline_aligned;struct free_area {
struct mt_free_area mt_free_area[MIGRATE_TYPES];
};However, without supporting data, it’s unclear if the space increase is justified :-)
Thanks
Barry
Re: [PATCH 3/3] mm: optimize free_area_empty() check using per-migratetype counts
Quoted 40 lines · Barry Song
Quoted 28 lines · Hongru Zhang
Quoted 16 lines
From: Hongru Zhang <[email protected]>
Use per-migratetype counts instead of
list_empty()helps reduce a few cpu instructions.Signed-off-by Hongru Zhang <[email protected]>--- mm/internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/internal.h +0
-0mm/internal.h int find_suitable_fallback(struct free_area *area, unsigned int order, · line 957
Quoted 5 lines
mm/internal.h +1
-1@@ -954,7 +954,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,static inline bool free_area_empty (struct free_area *area,int migratetype) { -return list_empty (&area->free_list[migratetype]); +return !READ_ONCE (area->mt_nr_free[migratetype]);I'm not quite sure about this. Since the counter is written and read more frequently, cache coherence traffic may actually be higher than for the list head.
Quoted 1 line
I'd prefer to drop this unless there is real data showing it performs better.
If the goal is to optimize
free_arealist checks andlist_add, a reasonable approach is to organize the data structure to reduce false sharing between different mt and order entries.struct mt_free_area { struct list_head free_list; unsigned long nr_free; } ____cacheline_aligned;
Quoted 6 lines
struct free_area { struct mt_free_area mt_free_area[MIGRATE_TYPES]; };However, without supporting data, it’s unclear if the space increase is justified :-)
I designed a test model to trigger more false sharing and collected data under it to see which layout performs better.
Test model
- Based on the microbench that was removed from mmtests
Commit: beeaeb89 ("pagealloc: Remove bit-rotted benchmark")
- Goal: Generate concurrent kernel page alloc/free activity across multiple orders and migratetypes to observe cacheline sharing and contention in the buddy free_area
- Mechanism: A systemtap module exposes a write-only /proc/mmtests-pagealloc-micro. Writing a 64-bit encoded value triggers repeated page alloc/free in kernel space
- bits 7:0 -> mt (0=UNMOVABLE, 1=MOVABLE, 2=RECLAIMABLE)
- bits 15:8 -> order
- bits 63:16 -> batch
- Workload distribution:
- order = cpu % 4 (orders 0/1/2/3)
- mt = cpu % 3 (UNMOVABLE/MOVABLE/RECLAIMABLE)
- cpu0 and cpu1 are not used for the test
- Sampling:
- load stap
- determine encoded value according to cpu id and bind it to the cpu
- after a short delay, runs 'perf mem record' for 100s
- unload stap
- Test tool:
- https://gist.github.com/zhr250/72e56f87ac703e833b11b5341d616cb0
- Data analysis tool:
- https://gist.github.com/zhr250/f4a385ffa9fae2993d22748f31e18588
CPU topo info of my machine:
Package L#0
NUMANode L#0 (P#0 15GB)
L3 L#0 (25MB)
L2 L#0 (1280KB) + L1d L#0 (48KB) + L1i L#0 (32KB) + Core L#0
PU L#0 (P#0)
PU L#1 (P#1)
L2 L#1 (1280KB) + L1d L#1 (48KB) + L1i L#1 (32KB) + Core L#1
PU L#2 (P#2)
PU L#3 (P#3)
L2 L#2 (1280KB) + L1d L#2 (48KB) + L1i L#2 (32KB) + Core L#2
PU L#4 (P#4)
PU L#5 (P#5)
L2 L#3 (1280KB) + L1d L#3 (48KB) + L1i L#3 (32KB) + Core L#3
PU L#6 (P#6)
PU L#7 (P#7)
L2 L#4 (1280KB) + L1d L#4 (48KB) + L1i L#4 (32KB) + Core L#4
PU L#8 (P#8)
PU L#9 (P#9)
L2 L#5 (1280KB) + L1d L#5 (48KB) + L1i L#5 (32KB) + Core L#5
PU L#10 (P#10)
PU L#11 (P#11)
L2 L#6 (1280KB) + L1d L#6 (48KB) + L1i L#6 (32KB) + Core L#6
PU L#12 (P#12)
PU L#13 (P#13)
L2 L#7 (1280KB) + L1d L#7 (48KB) + L1i L#7 (32KB) + Core L#7
PU L#14 (P#14)
PU L#15 (P#15)
L2 L#8 (2048KB)
L1d L#8 (32KB) + L1i L#8 (64KB) + Core L#8 + PU L#16 (P#16)
L1d L#9 (32KB) + L1i L#9 (64KB) + Core L#9 + PU L#17 (P#17)
L1d L#10 (32KB) + L1i L#10 (64KB) + Core L#10 + PU L#18 (P#18)
L1d L#11 (32KB) + L1i L#11 (64KB) + Core L#11 + PU L#19 (P#19)Actual (order, mt) distribution on my machine:
order=0, mt=0: cpu12
order=0, mt=1: cpu4, cpu16
order=0, mt=2: cpu8
order=1, mt=0: cpu9
order=1, mt=1: cpu13
order=1, mt=2: cpu5, cpu17
order=2, mt=0: cpu6, cpu18
order=2, mt=1: cpu10
order=2, mt=2: cpu2, cpu14
order=3, mt=0: cpu3, cpu15
order=3, mt=1: cpu7, cpu19
order=3, mt=2: cpu11
Different migratetype/order combinations are placed on CPUs that do not share L1/L2 caches to maximize cacheline contention. For our test goal, I think this distribution is relatively reasonable. I ran 10 rounds for each kernel and found the data to be stable.
Layouts tested (capturing load/store samples in free_area[0..MAX_PAGE_ORDER]):
- vanilla kernel:
struct free_area
{
struct list_head free_list[MIGRATE_TYPES];
unsigned long nr_free;
};
- patched kernel:
struct free_area {
struct list_head free_list[MIGRATE_TYPES];
unsigned long nr_free;
+ unsigned long mt_nr_free[MIGRATE_TYPES];
};- mtlist kernel:
+struct mt_free_list {
+ struct list_head list;
+ unsigned long nr_free;
+};
+
struct free_area {
- struct list_head free_list[MIGRATE_TYPES];
+ struct mt_free_list mt_free_list[MIGRATE_TYPES];
unsigned long nr_free;
};summary:
+---------+-----------------+-----------------+------------------------+---------------+---------------------+---------------+ | Kernel | inrange samples | HitM (%) | L1 hit inc LFB/MAB (%) | L2 hit (%) | L3 hit inc HitM (%) | RAM hit (%) | +---------+-----------------+-----------------+------------------------+---------------+---------------------+---------------+ | vanilla | 192,468 | 45,421 (23.60%) | 94,486 (49.09%) | 1,952 (1.01%) | 91,240 (47.41%) | 4,790 (2.49%) | +---------+-----------------+-----------------+------------------------+---------------+---------------------+---------------+ | patched | 227,196 | 27,293 (12.01%) | 165,238 (72.73%) | 1,194 (0.53%) | 54,609 (24.04%) | 6,155 (2.71%) | +---------+-----------------+-----------------+------------------------+---------------+---------------------+---------------+ | mtlist | 240,694 | 50,911 (21.15%) | 132,827 (55.19%) | 3,165 (1.31%) | 98,556 (40.95%) | 6,146 (2.55%) | +---------+-----------------+-----------------+------------------------+---------------+---------------------+---------------+
Detailed data:
- https://gist.github.com/zhr250/2ccf8902080ecaf85477d9c051e72a96
For both L1 hit and HitM, the patched kernel is the best among the three.
In this test model, I also collected memory allocation counts. The patched kernel delivers the best performance — about 7.00% higher than vanilla and 4.93% higher than mtlist.
Detailed data:
https://gist.github.com/zhr250/4439523b7ca3c18f4a2d2c97b24c4965
Re: [PATCH 3/3] mm: optimize free_area_empty() check using per-migratetype counts
In reply to Barry SongQuoted 26 lines
Quoted 16 lines
From: Hongru Zhang <[email protected]>
Use per-migratetype counts instead of
list_empty()helps reduce a few cpu instructions.Signed-off-by Hongru Zhang <[email protected]>--- mm/internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/internal.h +0
-0mm/internal.h int find_suitable_fallback(struct free_area *area, unsigned int order, · line 957
Quoted 5 lines
mm/internal.h +1
-1@@ -954,7 +954,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,static inline bool free_area_empty (struct free_area *area,int migratetype) { -return list_empty (&area->free_list[migratetype]); +return !READ_ONCE (area->mt_nr_free[migratetype]);I'm not quite sure about this. Since the counter is written and read more frequently, cache coherence traffic may actually be higher than for the list head.
Quoted 1 line
I'd prefer to drop this unless there is real data showing it performs better.
Under the model described in email [1], with this patch (3/3), memory allocation performance is 0.36% better than without it.
Detailed data:
- https://gist.github.com/zhr250/4439523b7ca3c18f4a2d2c97b24c4965
Reference:
[1] https://lore.kernel.org/linux-mm/[email protected]/
Barry Song
reviewerRe: [PATCH 3/3] mm: optimize free_area_empty() check using per-migratetype counts
Quoted 27 lines · Hongru Zhang
Quoted 27 lines
Quoted 17 lines
From: Hongru Zhang <[email protected]>
Use per-migratetype counts instead of
list_empty()helps reduce a few cpu instructions.Signed-off-by Hongru Zhang <[email protected]>--- mm/internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/internal.h +0
-0mm/internal.h int find_suitable_fallback(struct free_area *area, unsigned int order, · line 957
Quoted 5 lines
mm/internal.h +1
-1@@ -954,7 +954,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,static inline bool free_area_empty (struct free_area *area,int migratetype) { -return list_empty (&area->free_list[migratetype]); +return !READ_ONCE (area->mt_nr_free[migratetype]);I'm not quite sure about this. Since the counter is written and read more frequently, cache coherence traffic may actually be higher than for the list head.
Quoted 4 lines
Quoted 1 line
I'd prefer to drop this unless there is real data showing it performs better.
Under the model described in email [1], with this patch (3/3), memory allocation performance is 0.36% better than without it.
I'm fine with the data, though my actual suggestion was to place each mt_free_area in a separate cacheline to avoid false sharing. That one could be separated if you see an obvious performance improvement.
For this patchset, I feel the main issue is convincing people of the value of increasing counters in the alloc/free hotpath. As Johannes commented [1], the new counters should provide more use cases than pagetypeinfo, or as Zi Yan commented [2], clearly demonstrate the value that userspace derives from pagetypeinfo.
Right now, pagetypeinfo is considered a debug interface, so its implementation is quite poor.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Quoted 6 lines
Detailed data:
- https://gist.github.com/zhr250/4439523b7ca3c18f4a2d2c97b24c4965Reference:
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks
Barry
Just a general plea :) could we please try not to send larger series like this so late.
We're at the last day before the merge window, this is better sent during 6.19-rc1 or if now as an RFC.
Thanks, Lorenzo
Quoted 204 lines · Hongru Zhang
On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.
Under high load scenarios, reading
/proc/pagetypeinfocauses memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contentionChanges:
1. Add per-migratetype counts to the buddy allocator to track free page block counts for each migratetype and order
2. Optimize/proc/pagetypeinfoaccess by utilizing these per-migratetype counts instead of iterating through free lists under zone lockPerformance testing shows following improvements:
1./proc/pagetypeinfoaccess latency reduced+-----------------------+----------+------------+ | | no-patch | with-patch | +-----------------------+----------+------------+ | Just after boot | 700.9 us | 268.6 us | +-----------------------+----------+------------+ | After building kernel | 28.7 ms | 269.8 us | +-----------------------+----------+------------+2. When
/proc/pagetypeinfois accessed concurrently, memory allocation and free performance degradation is reduced compared to the previous implementationTest setup:
- Using config-pagealloc-micro
- Monitor set to proc-pagetypeinfo, update frequency set to 10ms
- PAGEALLOC_ORDER_MIN=4, PAGEALLOC_ORDER_MAX=4Without patch test results:
vanilla vanilla no-monitor monitor Min alloc-odr4-1 8539.00 ( 0.00%) 8762.00 ( -2.61%) Min alloc-odr4-2 6501.00 ( 0.00%) 6683.00 ( -2.80%) Min alloc-odr4-4 5537.00 ( 0.00%) 5873.00 ( -6.07%) Min alloc-odr4-8 5030.00 ( 0.00%) 5361.00 ( -6.58%) Min alloc-odr4-16 4782.00 ( 0.00%) 5162.00 ( -7.95%) Min alloc-odr4-32 5838.00 ( 0.00%) 6499.00 ( -11.32%) Min alloc-odr4-64 6565.00 ( 0.00%) 7413.00 ( -12.92%) Min alloc-odr4-128 6896.00 ( 0.00%) 7898.00 ( -14.53%) Min alloc-odr4-256 7303.00 ( 0.00%) 8163.00 ( -11.78%) Min alloc-odr4-512 10179.00 ( 0.00%) 11985.00 ( -17.74%) Min alloc-odr4-1024 11000.00 ( 0.00%) 12165.00 ( -10.59%) Min free-odr4-1 820.00 ( 0.00%) 1230.00 ( -50.00%) Min free-odr4-2 511.00 ( 0.00%) 952.00 ( -86.30%) Min free-odr4-4 347.00 ( 0.00%) 434.00 ( -25.07%) Min free-odr4-8 286.00 ( 0.00%) 399.00 ( -39.51%) Min free-odr4-16 250.00 ( 0.00%) 405.00 ( -62.00%) Min free-odr4-32 294.00 ( 0.00%) 405.00 ( -37.76%) Min free-odr4-64 333.00 ( 0.00%) 363.00 ( -9.01%) Min free-odr4-128 340.00 ( 0.00%) 412.00 ( -21.18%) Min free-odr4-256 339.00 ( 0.00%) 329.00 ( 2.95%) Min free-odr4-512 361.00 ( 0.00%) 409.00 ( -13.30%) Min free-odr4-1024 300.00 ( 0.00%) 361.00 ( -20.33%) Stddev alloc-odr4-1 7.29 ( 0.00%) 90.78 (-1146.00%) Stddev alloc-odr4-2 3.87 ( 0.00%) 51.30 (-1225.75%) Stddev alloc-odr4-4 3.20 ( 0.00%) 50.90 (-1491.24%) Stddev alloc-odr4-8 4.67 ( 0.00%) 52.23 (-1019.35%) Stddev alloc-odr4-16 5.72 ( 0.00%) 27.53 (-381.04%) Stddev alloc-odr4-32 6.25 ( 0.00%) 641.23 (-10154.46%) Stddev alloc-odr4-64 2.06 ( 0.00%) 386.99 (-18714.22%) Stddev alloc-odr4-128 14.36 ( 0.00%) 52.39 (-264.77%) Stddev alloc-odr4-256 32.42 ( 0.00%) 326.19 (-906.05%) Stddev alloc-odr4-512 65.58 ( 0.00%) 184.49 (-181.31%) Stddev alloc-odr4-1024 8.88 ( 0.00%) 153.01 (-1622.67%) Stddev free-odr4-1 2.29 ( 0.00%) 152.27 (-6549.85%) Stddev free-odr4-2 10.99 ( 0.00%) 73.10 (-564.89%) Stddev free-odr4-4 1.99 ( 0.00%) 28.40 (-1324.45%) Stddev free-odr4-8 2.51 ( 0.00%) 52.93 (-2007.64%) Stddev free-odr4-16 2.85 ( 0.00%) 26.04 (-814.88%) Stddev free-odr4-32 4.04 ( 0.00%) 27.05 (-569.79%) Stddev free-odr4-64 2.10 ( 0.00%) 48.07 (-2185.66%) Stddev free-odr4-128 2.63 ( 0.00%) 26.23 (-897.86%) Stddev free-odr4-256 6.29 ( 0.00%) 37.04 (-488.71%) Stddev free-odr4-512 2.56 ( 0.00%) 10.65 (-315.28%) Stddev free-odr4-1024 0.95 ( 0.00%) 6.46 (-582.22%) Max alloc-odr4-1 8564.00 ( 0.00%) 9099.00 ( -6.25%) Max alloc-odr4-2 6511.00 ( 0.00%) 6844.00 ( -5.11%) Max alloc-odr4-4 5549.00 ( 0.00%) 6038.00 ( -8.81%) Max alloc-odr4-8 5045.00 ( 0.00%) 5551.00 ( -10.03%) Max alloc-odr4-16 4800.00 ( 0.00%) 5257.00 ( -9.52%) Max alloc-odr4-32 5861.00 ( 0.00%) 8115.00 ( -38.46%) Max alloc-odr4-64 6571.00 ( 0.00%) 8292.00 ( -26.19%) Max alloc-odr4-128 6930.00 ( 0.00%) 8081.00 ( -16.61%) Max alloc-odr4-256 7372.00 ( 0.00%) 9150.00 ( -24.12%) Max alloc-odr4-512 10333.00 ( 0.00%) 12636.00 ( -22.29%) Max alloc-odr4-1024 11035.00 ( 0.00%) 12590.00 ( -14.09%) Max free-odr4-1 828.00 ( 0.00%) 1724.00 (-108.21%) Max free-odr4-2 543.00 ( 0.00%) 1192.00 (-119.52%) Max free-odr4-4 354.00 ( 0.00%) 519.00 ( -46.61%) Max free-odr4-8 293.00 ( 0.00%) 617.00 (-110.58%) Max free-odr4-16 260.00 ( 0.00%) 483.00 ( -85.77%) Max free-odr4-32 308.00 ( 0.00%) 488.00 ( -58.44%) Max free-odr4-64 341.00 ( 0.00%) 505.00 ( -48.09%) Max free-odr4-128 346.00 ( 0.00%) 497.00 ( -43.64%) Max free-odr4-256 353.00 ( 0.00%) 463.00 ( -31.16%) Max free-odr4-512 367.00 ( 0.00%) 442.00 ( -20.44%) Max free-odr4-1024 303.00 ( 0.00%) 381.00 ( -25.74%)With patch test results:
patched patched no-monitor monitor Min alloc-odr4-1 8488.00 ( 0.00%) 8514.00 ( -0.31%) Min alloc-odr4-2 6551.00 ( 0.00%) 6527.00 ( 0.37%) Min alloc-odr4-4 5536.00 ( 0.00%) 5591.00 ( -0.99%) Min alloc-odr4-8 5008.00 ( 0.00%) 5098.00 ( -1.80%) Min alloc-odr4-16 4760.00 ( 0.00%) 4857.00 ( -2.04%) Min alloc-odr4-32 5827.00 ( 0.00%) 5919.00 ( -1.58%) Min alloc-odr4-64 6561.00 ( 0.00%) 6680.00 ( -1.81%) Min alloc-odr4-128 6898.00 ( 0.00%) 7014.00 ( -1.68%) Min alloc-odr4-256 7311.00 ( 0.00%) 7464.00 ( -2.09%) Min alloc-odr4-512 10181.00 ( 0.00%) 10286.00 ( -1.03%) Min alloc-odr4-1024 11205.00 ( 0.00%) 11725.00 ( -4.64%) Min free-odr4-1 789.00 ( 0.00%) 867.00 ( -9.89%) Min free-odr4-2 490.00 ( 0.00%) 526.00 ( -7.35%) Min free-odr4-4 350.00 ( 0.00%) 360.00 ( -2.86%) Min free-odr4-8 272.00 ( 0.00%) 287.00 ( -5.51%) Min free-odr4-16 247.00 ( 0.00%) 254.00 ( -2.83%) Min free-odr4-32 298.00 ( 0.00%) 304.00 ( -2.01%) Min free-odr4-64 334.00 ( 0.00%) 325.00 ( 2.69%) Min free-odr4-128 334.00 ( 0.00%) 329.00 ( 1.50%) Min free-odr4-256 336.00 ( 0.00%) 336.00 ( 0.00%) Min free-odr4-512 360.00 ( 0.00%) 342.00 ( 5.00%) Min free-odr4-1024 327.00 ( 0.00%) 355.00 ( -8.56%) Stddev alloc-odr4-1 5.19 ( 0.00%) 45.38 (-775.09%) Stddev alloc-odr4-2 6.99 ( 0.00%) 37.63 (-437.98%) Stddev alloc-odr4-4 3.91 ( 0.00%) 17.85 (-356.28%) Stddev alloc-odr4-8 5.15 ( 0.00%) 9.34 ( -81.47%) Stddev alloc-odr4-16 3.83 ( 0.00%) 5.34 ( -39.34%) Stddev alloc-odr4-32 1.96 ( 0.00%) 10.28 (-425.09%) Stddev alloc-odr4-64 1.32 ( 0.00%) 333.30 (-25141.39%) Stddev alloc-odr4-128 2.06 ( 0.00%) 7.37 (-258.28%) Stddev alloc-odr4-256 15.56 ( 0.00%) 113.48 (-629.25%) Stddev alloc-odr4-512 61.25 ( 0.00%) 165.09 (-169.53%) Stddev alloc-odr4-1024 18.89 ( 0.00%) 2.93 ( 84.51%) Stddev free-odr4-1 4.45 ( 0.00%) 40.12 (-800.98%) Stddev free-odr4-2 1.50 ( 0.00%) 29.30 (-1850.31%) Stddev free-odr4-4 1.27 ( 0.00%) 19.49 (-1439.40%) Stddev free-odr4-8 0.97 ( 0.00%) 8.93 (-823.07%) Stddev free-odr4-16 8.38 ( 0.00%) 4.51 ( 46.21%) Stddev free-odr4-32 3.18 ( 0.00%) 6.59 (-107.42%) Stddev free-odr4-64 2.40 ( 0.00%) 3.09 ( -28.50%) Stddev free-odr4-128 1.55 ( 0.00%) 2.53 ( -62.92%) Stddev free-odr4-256 0.41 ( 0.00%) 2.80 (-585.57%) Stddev free-odr4-512 1.60 ( 0.00%) 4.84 (-202.08%) Stddev free-odr4-1024 0.66 ( 0.00%) 1.19 ( -80.68%) Max alloc-odr4-1 8505.00 ( 0.00%) 8676.00 ( -2.01%) Max alloc-odr4-2 6572.00 ( 0.00%) 6651.00 ( -1.20%) Max alloc-odr4-4 5552.00 ( 0.00%) 5646.00 ( -1.69%) Max alloc-odr4-8 5024.00 ( 0.00%) 5131.00 ( -2.13%) Max alloc-odr4-16 4774.00 ( 0.00%) 4875.00 ( -2.12%) Max alloc-odr4-32 5834.00 ( 0.00%) 5950.00 ( -1.99%) Max alloc-odr4-64 6565.00 ( 0.00%) 7434.00 ( -13.24%) Max alloc-odr4-128 6907.00 ( 0.00%) 7034.00 ( -1.84%) Max alloc-odr4-256 7347.00 ( 0.00%) 7843.00 ( -6.75%) Max alloc-odr4-512 10315.00 ( 0.00%) 10866.00 ( -5.34%) Max alloc-odr4-1024 11278.00 ( 0.00%) 11733.00 ( -4.03%) Max free-odr4-1 803.00 ( 0.00%) 1009.00 ( -25.65%) Max free-odr4-2 495.00 ( 0.00%) 607.00 ( -22.63%) Max free-odr4-4 354.00 ( 0.00%) 417.00 ( -17.80%) Max free-odr4-8 275.00 ( 0.00%) 313.00 ( -13.82%) Max free-odr4-16 273.00 ( 0.00%) 272.00 ( 0.37%) Max free-odr4-32 309.00 ( 0.00%) 324.00 ( -4.85%) Max free-odr4-64 340.00 ( 0.00%) 335.00 ( 1.47%) Max free-odr4-128 340.00 ( 0.00%) 338.00 ( 0.59%) Max free-odr4-256 338.00 ( 0.00%) 346.00 ( -2.37%) Max free-odr4-512 364.00 ( 0.00%) 359.00 ( 1.37%) Max free-odr4-1024 329.00 ( 0.00%) 359.00 ( -9.12%)The main overhead is a slight increase in latency on the memory allocation and free paths due to additional per-migratetype counting, with theoretically minimal impact on overall performance.
This patch series is based on v6.18-rc7
Hongru Zhang (3):
mm/page_alloc: add per-migratetype counts to buddy allocator mm/vmstat: get fragmentation statistics from per-migragetype count
mm: optimizefree_area_empty()check using per-migratetype countsinclude/linux/mmzone.h | 1 + mm/internal.h | 2 +- mm/mm_init.c | 1 + mm/page_alloc.c | 9 ++++++++- mm/vmstat.c | 30 +++++++----------------------- 5 files changed, 18 insertions(+), 25 deletions(-)--
2.43.0
Quoted 7 lines
Just a general plea :) could we please try not to send larger series like this so late.
We're at the last day before the merge window, this is better sent during 6.19-rc1 or if now as an RFC.
Thanks, Lorenzo
Hi Lorenzo,
Thank you for your feedback and sorry for the late submission. You're right - this series should have been sent earlier. Apologize for not following the proper submission timing guidelines. I'll make sure to follow the community norms and submit similar work well in advance in the future.
Thanks again for your patience and guidance.
Quoted 10 lines · Hongru Zhang
Quoted 7 lines
Just a general plea :) could we please try not to send larger series like this so late.
We're at the last day before the merge window, this is better sent during 6.19-rc1 or if now as an RFC.
Thanks, Lorenzo
Hi Lorenzo,
Quoted 7 lines
Thank you for your feedback and sorry for the late submission. You're right - this series should have been sent earlier. Apologize for not following the proper submission timing guidelines. I'll make sure to follow the community norms and submit similar work well in advance in the future.
Thanks again for your patience and guidance.
Hi Hongru,
Sorry I don't mean to be critical here and you weren't to know :) rather just in general - a plea for how we do things in mm.
Your series is very much appreciated, you didn't do anything wrong at all - this is just essentially an admin thing :P
We will absolutely review your series it's just about timing. And of course I'm just sort of making a point here, reviewers can choose to review as and when they want! :)
Cheers, Lorenzo
Quoted 4 lines · Hongru Zhang
On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.
Hm /proc/buddyinfo could be enough to determine fragmentation? Also we have in-kernel proactive compaction these days.
Quoted 11 lines
Under high load scenarios, reading
/proc/pagetypeinfocauses memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention
It could be argued that not capturing /proc/pagetypeinfo (often) would help. I wonder if we can find also other benefits from the counters in the kernel itself.
Adding these migratetype counters is something that wouldn't be even possible in the past, until the freelist migratetype hygiene was merged. So now it should be AFAIK possible, but it's still some overhead in relatively hot paths. I wonder if we even considered this before in the context of migratetype hygiene? Couldn't find anything quickly.
Quoted 184 lines
Changes:
1. Add per-migratetype counts to the buddy allocator to track free page block counts for each migratetype and order
2. Optimize/proc/pagetypeinfoaccess by utilizing these per-migratetype counts instead of iterating through free lists under zone lockPerformance testing shows following improvements:
1./proc/pagetypeinfoaccess latency reduced+-----------------------+----------+------------+ | | no-patch | with-patch | +-----------------------+----------+------------+ | Just after boot | 700.9 us | 268.6 us | +-----------------------+----------+------------+ | After building kernel | 28.7 ms | 269.8 us | +-----------------------+----------+------------+2. When
/proc/pagetypeinfois accessed concurrently, memory allocation and free performance degradation is reduced compared to the previous implementationTest setup:
- Using config-pagealloc-micro
- Monitor set to proc-pagetypeinfo, update frequency set to 10ms
- PAGEALLOC_ORDER_MIN=4, PAGEALLOC_ORDER_MAX=4Without patch test results:
vanilla vanilla no-monitor monitor Min alloc-odr4-1 8539.00 ( 0.00%) 8762.00 ( -2.61%) Min alloc-odr4-2 6501.00 ( 0.00%) 6683.00 ( -2.80%) Min alloc-odr4-4 5537.00 ( 0.00%) 5873.00 ( -6.07%) Min alloc-odr4-8 5030.00 ( 0.00%) 5361.00 ( -6.58%) Min alloc-odr4-16 4782.00 ( 0.00%) 5162.00 ( -7.95%) Min alloc-odr4-32 5838.00 ( 0.00%) 6499.00 ( -11.32%) Min alloc-odr4-64 6565.00 ( 0.00%) 7413.00 ( -12.92%) Min alloc-odr4-128 6896.00 ( 0.00%) 7898.00 ( -14.53%) Min alloc-odr4-256 7303.00 ( 0.00%) 8163.00 ( -11.78%) Min alloc-odr4-512 10179.00 ( 0.00%) 11985.00 ( -17.74%) Min alloc-odr4-1024 11000.00 ( 0.00%) 12165.00 ( -10.59%) Min free-odr4-1 820.00 ( 0.00%) 1230.00 ( -50.00%) Min free-odr4-2 511.00 ( 0.00%) 952.00 ( -86.30%) Min free-odr4-4 347.00 ( 0.00%) 434.00 ( -25.07%) Min free-odr4-8 286.00 ( 0.00%) 399.00 ( -39.51%) Min free-odr4-16 250.00 ( 0.00%) 405.00 ( -62.00%) Min free-odr4-32 294.00 ( 0.00%) 405.00 ( -37.76%) Min free-odr4-64 333.00 ( 0.00%) 363.00 ( -9.01%) Min free-odr4-128 340.00 ( 0.00%) 412.00 ( -21.18%) Min free-odr4-256 339.00 ( 0.00%) 329.00 ( 2.95%) Min free-odr4-512 361.00 ( 0.00%) 409.00 ( -13.30%) Min free-odr4-1024 300.00 ( 0.00%) 361.00 ( -20.33%) Stddev alloc-odr4-1 7.29 ( 0.00%) 90.78 (-1146.00%) Stddev alloc-odr4-2 3.87 ( 0.00%) 51.30 (-1225.75%) Stddev alloc-odr4-4 3.20 ( 0.00%) 50.90 (-1491.24%) Stddev alloc-odr4-8 4.67 ( 0.00%) 52.23 (-1019.35%) Stddev alloc-odr4-16 5.72 ( 0.00%) 27.53 (-381.04%) Stddev alloc-odr4-32 6.25 ( 0.00%) 641.23 (-10154.46%) Stddev alloc-odr4-64 2.06 ( 0.00%) 386.99 (-18714.22%) Stddev alloc-odr4-128 14.36 ( 0.00%) 52.39 (-264.77%) Stddev alloc-odr4-256 32.42 ( 0.00%) 326.19 (-906.05%) Stddev alloc-odr4-512 65.58 ( 0.00%) 184.49 (-181.31%) Stddev alloc-odr4-1024 8.88 ( 0.00%) 153.01 (-1622.67%) Stddev free-odr4-1 2.29 ( 0.00%) 152.27 (-6549.85%) Stddev free-odr4-2 10.99 ( 0.00%) 73.10 (-564.89%) Stddev free-odr4-4 1.99 ( 0.00%) 28.40 (-1324.45%) Stddev free-odr4-8 2.51 ( 0.00%) 52.93 (-2007.64%) Stddev free-odr4-16 2.85 ( 0.00%) 26.04 (-814.88%) Stddev free-odr4-32 4.04 ( 0.00%) 27.05 (-569.79%) Stddev free-odr4-64 2.10 ( 0.00%) 48.07 (-2185.66%) Stddev free-odr4-128 2.63 ( 0.00%) 26.23 (-897.86%) Stddev free-odr4-256 6.29 ( 0.00%) 37.04 (-488.71%) Stddev free-odr4-512 2.56 ( 0.00%) 10.65 (-315.28%) Stddev free-odr4-1024 0.95 ( 0.00%) 6.46 (-582.22%) Max alloc-odr4-1 8564.00 ( 0.00%) 9099.00 ( -6.25%) Max alloc-odr4-2 6511.00 ( 0.00%) 6844.00 ( -5.11%) Max alloc-odr4-4 5549.00 ( 0.00%) 6038.00 ( -8.81%) Max alloc-odr4-8 5045.00 ( 0.00%) 5551.00 ( -10.03%) Max alloc-odr4-16 4800.00 ( 0.00%) 5257.00 ( -9.52%) Max alloc-odr4-32 5861.00 ( 0.00%) 8115.00 ( -38.46%) Max alloc-odr4-64 6571.00 ( 0.00%) 8292.00 ( -26.19%) Max alloc-odr4-128 6930.00 ( 0.00%) 8081.00 ( -16.61%) Max alloc-odr4-256 7372.00 ( 0.00%) 9150.00 ( -24.12%) Max alloc-odr4-512 10333.00 ( 0.00%) 12636.00 ( -22.29%) Max alloc-odr4-1024 11035.00 ( 0.00%) 12590.00 ( -14.09%) Max free-odr4-1 828.00 ( 0.00%) 1724.00 (-108.21%) Max free-odr4-2 543.00 ( 0.00%) 1192.00 (-119.52%) Max free-odr4-4 354.00 ( 0.00%) 519.00 ( -46.61%) Max free-odr4-8 293.00 ( 0.00%) 617.00 (-110.58%) Max free-odr4-16 260.00 ( 0.00%) 483.00 ( -85.77%) Max free-odr4-32 308.00 ( 0.00%) 488.00 ( -58.44%) Max free-odr4-64 341.00 ( 0.00%) 505.00 ( -48.09%) Max free-odr4-128 346.00 ( 0.00%) 497.00 ( -43.64%) Max free-odr4-256 353.00 ( 0.00%) 463.00 ( -31.16%) Max free-odr4-512 367.00 ( 0.00%) 442.00 ( -20.44%) Max free-odr4-1024 303.00 ( 0.00%) 381.00 ( -25.74%)With patch test results:
patched patched no-monitor monitor Min alloc-odr4-1 8488.00 ( 0.00%) 8514.00 ( -0.31%) Min alloc-odr4-2 6551.00 ( 0.00%) 6527.00 ( 0.37%) Min alloc-odr4-4 5536.00 ( 0.00%) 5591.00 ( -0.99%) Min alloc-odr4-8 5008.00 ( 0.00%) 5098.00 ( -1.80%) Min alloc-odr4-16 4760.00 ( 0.00%) 4857.00 ( -2.04%) Min alloc-odr4-32 5827.00 ( 0.00%) 5919.00 ( -1.58%) Min alloc-odr4-64 6561.00 ( 0.00%) 6680.00 ( -1.81%) Min alloc-odr4-128 6898.00 ( 0.00%) 7014.00 ( -1.68%) Min alloc-odr4-256 7311.00 ( 0.00%) 7464.00 ( -2.09%) Min alloc-odr4-512 10181.00 ( 0.00%) 10286.00 ( -1.03%) Min alloc-odr4-1024 11205.00 ( 0.00%) 11725.00 ( -4.64%) Min free-odr4-1 789.00 ( 0.00%) 867.00 ( -9.89%) Min free-odr4-2 490.00 ( 0.00%) 526.00 ( -7.35%) Min free-odr4-4 350.00 ( 0.00%) 360.00 ( -2.86%) Min free-odr4-8 272.00 ( 0.00%) 287.00 ( -5.51%) Min free-odr4-16 247.00 ( 0.00%) 254.00 ( -2.83%) Min free-odr4-32 298.00 ( 0.00%) 304.00 ( -2.01%) Min free-odr4-64 334.00 ( 0.00%) 325.00 ( 2.69%) Min free-odr4-128 334.00 ( 0.00%) 329.00 ( 1.50%) Min free-odr4-256 336.00 ( 0.00%) 336.00 ( 0.00%) Min free-odr4-512 360.00 ( 0.00%) 342.00 ( 5.00%) Min free-odr4-1024 327.00 ( 0.00%) 355.00 ( -8.56%) Stddev alloc-odr4-1 5.19 ( 0.00%) 45.38 (-775.09%) Stddev alloc-odr4-2 6.99 ( 0.00%) 37.63 (-437.98%) Stddev alloc-odr4-4 3.91 ( 0.00%) 17.85 (-356.28%) Stddev alloc-odr4-8 5.15 ( 0.00%) 9.34 ( -81.47%) Stddev alloc-odr4-16 3.83 ( 0.00%) 5.34 ( -39.34%) Stddev alloc-odr4-32 1.96 ( 0.00%) 10.28 (-425.09%) Stddev alloc-odr4-64 1.32 ( 0.00%) 333.30 (-25141.39%) Stddev alloc-odr4-128 2.06 ( 0.00%) 7.37 (-258.28%) Stddev alloc-odr4-256 15.56 ( 0.00%) 113.48 (-629.25%) Stddev alloc-odr4-512 61.25 ( 0.00%) 165.09 (-169.53%) Stddev alloc-odr4-1024 18.89 ( 0.00%) 2.93 ( 84.51%) Stddev free-odr4-1 4.45 ( 0.00%) 40.12 (-800.98%) Stddev free-odr4-2 1.50 ( 0.00%) 29.30 (-1850.31%) Stddev free-odr4-4 1.27 ( 0.00%) 19.49 (-1439.40%) Stddev free-odr4-8 0.97 ( 0.00%) 8.93 (-823.07%) Stddev free-odr4-16 8.38 ( 0.00%) 4.51 ( 46.21%) Stddev free-odr4-32 3.18 ( 0.00%) 6.59 (-107.42%) Stddev free-odr4-64 2.40 ( 0.00%) 3.09 ( -28.50%) Stddev free-odr4-128 1.55 ( 0.00%) 2.53 ( -62.92%) Stddev free-odr4-256 0.41 ( 0.00%) 2.80 (-585.57%) Stddev free-odr4-512 1.60 ( 0.00%) 4.84 (-202.08%) Stddev free-odr4-1024 0.66 ( 0.00%) 1.19 ( -80.68%) Max alloc-odr4-1 8505.00 ( 0.00%) 8676.00 ( -2.01%) Max alloc-odr4-2 6572.00 ( 0.00%) 6651.00 ( -1.20%) Max alloc-odr4-4 5552.00 ( 0.00%) 5646.00 ( -1.69%) Max alloc-odr4-8 5024.00 ( 0.00%) 5131.00 ( -2.13%) Max alloc-odr4-16 4774.00 ( 0.00%) 4875.00 ( -2.12%) Max alloc-odr4-32 5834.00 ( 0.00%) 5950.00 ( -1.99%) Max alloc-odr4-64 6565.00 ( 0.00%) 7434.00 ( -13.24%) Max alloc-odr4-128 6907.00 ( 0.00%) 7034.00 ( -1.84%) Max alloc-odr4-256 7347.00 ( 0.00%) 7843.00 ( -6.75%) Max alloc-odr4-512 10315.00 ( 0.00%) 10866.00 ( -5.34%) Max alloc-odr4-1024 11278.00 ( 0.00%) 11733.00 ( -4.03%) Max free-odr4-1 803.00 ( 0.00%) 1009.00 ( -25.65%) Max free-odr4-2 495.00 ( 0.00%) 607.00 ( -22.63%) Max free-odr4-4 354.00 ( 0.00%) 417.00 ( -17.80%) Max free-odr4-8 275.00 ( 0.00%) 313.00 ( -13.82%) Max free-odr4-16 273.00 ( 0.00%) 272.00 ( 0.37%) Max free-odr4-32 309.00 ( 0.00%) 324.00 ( -4.85%) Max free-odr4-64 340.00 ( 0.00%) 335.00 ( 1.47%) Max free-odr4-128 340.00 ( 0.00%) 338.00 ( 0.59%) Max free-odr4-256 338.00 ( 0.00%) 346.00 ( -2.37%) Max free-odr4-512 364.00 ( 0.00%) 359.00 ( 1.37%) Max free-odr4-1024 329.00 ( 0.00%) 359.00 ( -9.12%)The main overhead is a slight increase in latency on the memory allocation and free paths due to additional per-migratetype counting, with theoretically minimal impact on overall performance.
This patch series is based on v6.18-rc7
Hongru Zhang (3):
mm/page_alloc: add per-migratetype counts to buddy allocator mm/vmstat: get fragmentation statistics from per-migragetype count
mm: optimizefree_area_empty()check using per-migratetype countsinclude/linux/mmzone.h | 1 + mm/internal.h | 2 +- mm/mm_init.c | 1 + mm/page_alloc.c | 9 ++++++++- mm/vmstat.c | 30 +++++++----------------------- 5 files changed, 18 insertions(+), 25 deletions(-)
Johannes Weiner
reviewerQuoted 21 lines · Vlastimil Babka
Quoted 4 lines · Hongru Zhang
On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.
Hm
/proc/buddyinfocould be enough to determine fragmentation? Also we have in-kernel proactive compaction these days.Quoted 11 lines
Under high load scenarios, reading
/proc/pagetypeinfocauses memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention
Quoted 3 lines
It could be argued that not capturing
/proc/pagetypeinfo(often) would help. I wonder if we can find also other benefits from the counters in the kernel itself.
In earlier iterations of the huge allocator patches, I played around with using these for compaction_suitable():
https://lore.kernel.org/linux-mm/[email protected]/
ISTR it cut down compaction numbers, because it would avoid runs where free pages are mostly in unsuitable targets (free_unmovable). But this was also in a series that used compaction_suitable() to stop kswapd, which in hindsight was a mistake; it would need re-evaluating by itself.
I also found these counters useful to have in OOM/allocfail dumps to see if allocator packing or compaction could have done better.
Quoted 7 lines
Quoted 4 lines
On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.
Hm
/proc/buddyinfocould be enough to determine fragmentation? Also we have in-kernel proactive compaction these days.
In fact, besides /proc/pagetypeinfo, other system resource information is also collected at appropriate times, and resource usage throughout the process lifecycle is appropriately tracked as well. User-space management components integrate this information together to make decisions and perform proper actions.
Quoted 12 lines
Quoted 11 lines
Under high load scenarios, reading
/proc/pagetypeinfocauses memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention
Quoted 3 lines
It could be argued that not capturing
/proc/pagetypeinfo(often) would help. I wonder if we can find also other benefits from the counters in the kernel itself.
Collecting system and app resource statistics and making decisions based on this information is a common practice among Android device manufacturers.
Currently, there should be over a billion Android phones being used daily worldwide. The diversity of hardware configurations across Android devices makes it difficult for kernel mechanisms alone to maintain good performance across all usage scenarios.
First, hardware capabilities vary greatly - flagship phones may have up to 24GB of memory, while low-end devices may have as little as 4GB. CPU, storage, battery, and passive cooling capabilities vary significantly due to market positioning and cost factors. Hardware resources seem always inadequate.
Second, usage scenarios also differ - some people use devices in hot environments while others in cold environments; some enjoy high-definition gaming while others simply browse the web.
Third, user habits vary as well. Some people rarely restart their phones except when the battery dies or the system crashes; others restart daily, like me. Some users never actively close apps, only switching them to the background, resulting in dozens of apps running in the background and keeping system resources consumed (especially memory). Yet others just use a few apps, closing unused apps rather than leaving them in the background.
Despite the above challenges, Android device manufacturers hope to ensure a good user experience (no UI jank) across all situations.
Even at 60 Hz frame refresh rate (90 Hz, 120 Hz also supported now), all work from user input to render and display should be done within 16.7 ms. To achieve this goal, the management components perform tasks such as:
- Track system resource status: what system has
(system resource awareness)
- Learn and predict app resource demands: what app needs (resource demand awareness)
- Monitor app launch, exit, and foreground-background switches: least important app gives back resource to system to serve most important one, usually the foreground app
(user intent awareness)
Tracking system resources seems necessary for Android devices, not optional. So the related paths are not that cold on Android devices.
All the above are from workload perspective. From the kernel perspective, regardless of when or how frequently user-space tools read statistical information, they should not affect the kernel's own efficiency significantly. That's why I submit this patch series to make the read side of /proc/pagetypeinfo lock-free. But this does introduce overhead in hot path, I would greatly appreciate if we can discuss how to improve it here.
Quoted 5 lines
Adding these migratetype counters is something that wouldn't be even possible in the past, until the freelist migratetype hygiene was merged. So now it should be AFAIK possible, but it's still some overhead in relatively hot paths. I wonder if we even considered this before in the context of migratetype hygiene? Couldn't find anything quickly.
Yes, I wrote the code on old kernel initially, at that time, I reused set_pcppage_migratetype (also renamed) to cache the exact migratetype list that the page block is on. After the freelist migratetype hygiene patches were merged, I removed that logic.
Zi Yan
reviewerQuoted 71 lines · Hongru Zhang
Quoted 7 lines
Quoted 4 lines
On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.
Hm
/proc/buddyinfocould be enough to determine fragmentation? Also we have in-kernel proactive compaction these days.In fact, besides
/proc/pagetypeinfo, other system resource information is also collected at appropriate times, and resource usage throughout the process lifecycle is appropriately tracked as well. User-space management components integrate this information together to make decisions and perform proper actions.Quoted 12 lines
Quoted 11 lines
Under high load scenarios, reading
/proc/pagetypeinfocauses memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contentionQuoted 3 lines
It could be argued that not capturing
/proc/pagetypeinfo(often) would help. I wonder if we can find also other benefits from the counters in the kernel itself.Collecting system and app resource statistics and making decisions based on this information is a common practice among Android device manufacturers.
Currently, there should be over a billion Android phones being used daily worldwide. The diversity of hardware configurations across Android devices makes it difficult for kernel mechanisms alone to maintain good performance across all usage scenarios.
First, hardware capabilities vary greatly - flagship phones may have up to 24GB of memory, while low-end devices may have as little as 4GB. CPU, storage, battery, and passive cooling capabilities vary significantly due to market positioning and cost factors. Hardware resources seem always inadequate.
Second, usage scenarios also differ - some people use devices in hot environments while others in cold environments; some enjoy high-definition gaming while others simply browse the web.
Third, user habits vary as well. Some people rarely restart their phones except when the battery dies or the system crashes; others restart daily, like me. Some users never actively close apps, only switching them to the background, resulting in dozens of apps running in the background and keeping system resources consumed (especially memory). Yet others just use a few apps, closing unused apps rather than leaving them in the background.
Despite the above challenges, Android device manufacturers hope to ensure a good user experience (no UI jank) across all situations.
Even at 60 Hz frame refresh rate (90 Hz, 120 Hz also supported now), all work from user input to render and display should be done within 16.7 ms. To achieve this goal, the management components perform tasks such as:
- Track system resource status: what system has
(system resource awareness)
- Learn and predict app resource demands: what app needs (resource demand awareness)
- Monitor app launch, exit, and foreground-background switches: least important app gives back resource to system to serve most important one, usually the foreground app
(user intent awareness)
Quoted 2 lines
Tracking system resources seems necessary for Android devices, not optional. So the related paths are not that cold on Android devices.
These are all good background information. But how does userspace monitor utilize pageblock migratetype information? Can you give a concrete example?
Something like when free_movable is low, background apps is killed to provide more free pages? Or is userspace monitor even trying to attribute different pageblock usage to each app by monitoring /proc/pagetypeinfo before and after an app launch?
Thanks.
Quoted 14 lines
All the above are from workload perspective. From the kernel perspective, regardless of when or how frequently user-space tools read statistical information, they should not affect the kernel's own efficiency significantly. That's why I submit this patch series to make the read side of
/proc/pagetypeinfolock-free. But this does introduce overhead in hot path, I would greatly appreciate if we can discuss how to improve it here.Quoted 5 lines
Adding these migratetype counters is something that wouldn't be even possible in the past, until the freelist migratetype hygiene was merged. So now it should be AFAIK possible, but it's still some overhead in relatively hot paths. I wonder if we even considered this before in the context of migratetype hygiene? Couldn't find anything quickly.
Quoted 4 lines
Yes, I wrote the code on old kernel initially, at that time, I reused
set_pcppage_migratetype(also renamed) to cache the exact migratetype list that the page block is on. After the freelist migratetype hygiene patches were merged, I removed that logic.
Best Regards,
Yan, Zi
Quoted 21 lines
Quoted 15 lines
Despite the above challenges, Android device manufacturers hope to ensure a good user experience (no UI jank) across all situations.
Even at 60 Hz frame refresh rate (90 Hz, 120 Hz also supported now), all work from user input to render and display should be done within 16.7 ms. To achieve this goal, the management components perform tasks such as:
- Track system resource status: what system has
(system resource awareness)
- Learn and predict app resource demands: what app needs (resource demand awareness)
- Monitor app launch, exit, and foreground-background switches: least important app gives back resource to system to serve most important one, usually the foreground app
(user intent awareness)Quoted 2 lines
Tracking system resources seems necessary for Android devices, not optional. So the related paths are not that cold on Android devices.
These are all good background information. But how does userspace monitor utilize pageblock migratetype information? Can you give a concrete example?
Quoted 6 lines
Something like when
free_movableis low, background apps is killed to provide more free pages? Or is userspace monitor even trying to attribute different pageblock usage to each app by monitoring/proc/pagetypeinfobefore and after an app launch?Thanks.
We have proprietary algorithms, but they are confidential. I cannot describe them publicly, sorry about that.
Barry Song
reviewerQuoted 6 lines · Hongru Zhang
On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.
Quoted 11 lines
Under high load scenarios, reading
/proc/pagetypeinfocauses memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention
Do you have an idea how long each seq_printf call takes?
Assuming seq_printf is costly, printing while holding zone->lock may be suboptimal. A further optimization might be:
mm/vmstat.c +58 -0
@@ -1570,6 +1570,7 @@ static int frag_show(struct seq_file *m, void *arg)return 0 ; } +#if 0 static void pagetypeinfo_showfree_print (struct seq_file *m,pg_data_t *pgdat,struct zone *zone) {
@@ -1611,6 +1612,63 @@ static void pagetypeinfo_showfree_print(struct seq_file *m, seq_putc(m, '\n'); } } +#endif + +static void pagetypeinfo_showfree_print (struct seq_file *m, +pg_data_t *pgdat, +struct zone *zone) +{ +unsigned long freecounts[MIGRATE_TYPES ][NR_PAGE_ORDERS ] = {0 }; +bool overflow[MIGRATE_TYPES ][NR_PAGE_ORDERS ] = {0 }; +int order, mtype; + +for (mtype =0 ; mtype <MIGRATE_TYPES ; mtype++) { +for (order =0 ; order <NR_PAGE_ORDERS ; ++order) { +struct free_area *area; +struct list_head *curr; +unsigned long freecount =0 ; + + area = &zone->free_area[order]; + +list_for_each (curr, &area->free_list[mtype]) { +/* +* Cap the free_list iteration because it might +* be really large and we are under a spinlock +* so a long time spent here could trigger a +* hard lockup detector. Anyway this is a +* debugging tool so knowing there is a handful +* of pages of this order should be more than +* sufficient. +*/ +if (++freecount >=100000 ) { + overflow[mtype][order] = true; +break ; + } + } + freecounts[mtype][order] = freecount; +spin_unlock_irq (&zone->lock); +cond_resched (); +spin_lock_irq (&zone->lock); + } + } + +/* printing completely outside the lock */ +spin_unlock_irq (&zone->lock); +for (mtype =0 ; mtype <MIGRATE_TYPES ; mtype++) { +seq_printf (m,"Node %4d, zone %8s, type %12s " , + pgdat->node_id, + zone->name, + migratetype_names[mtype]); + +for (order =0 ; order <NR_PAGE_ORDERS ; ++order) { +seq_printf (m,"%s%6lu " , + overflow[mtype][order] ?">" :"" , + freecounts[mtype][order]); + } +seq_putc (m,'\n' ); + } +spin_lock_irq (&zone->lock); +}/* Print out the free pages at each order for each migratetype */ static void pagetypeinfo_showfree (struct seq_file *m,void *arg)
Thanks
Barry
Quoted 90 lines · Hongru Zhang
Quoted 6 lines
On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.
Quoted 11 lines
Under high load scenarios, reading
/proc/pagetypeinfocauses memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contentionDo you have an idea how long each
seq_printfcall takes?Assuming
seq_printfis costly, printing while holdingzone->lockmay be suboptimal. A further optimization might be:mm/vmstat.c +51
-0@@ -1570,6 +1570,7 @@ static int frag_show(struct seq_file *m, void *arg)return 0 ; } +#if 0 static void pagetypeinfo_showfree_print (struct seq_file *m,pg_data_t *pgdat,struct zone *zone) {@@ -1611,6 +1612,63 @@ static void pagetypeinfo_showfree_print(struct seq_file *m, seq_putc(m, '\n'); } } +#endif + +static void pagetypeinfo_showfree_print (struct seq_file *m, +pg_data_t *pgdat, +struct zone *zone) +{ +unsigned long freecounts[MIGRATE_TYPES ][NR_PAGE_ORDERS ] = {0 }; +bool overflow[MIGRATE_TYPES ][NR_PAGE_ORDERS ] = {0 }; +int order, mtype; + +for (mtype =0 ; mtype <MIGRATE_TYPES ; mtype++) { +for (order =0 ; order <NR_PAGE_ORDERS ; ++order) { +struct free_area *area; +struct list_head *curr; +unsigned long freecount =0 ; + + area = &zone->free_area[order]; + +list_for_each (curr, &area->free_list[mtype]) { +/* +* Cap the free_list iteration because it might +* be really large and we are under a spinlock +* so a long time spent here could trigger a +* hard lockup detector. Anyway this is a +* debugging tool so knowing there is a handful +* of pages of this order should be more than +* sufficient. +*/ +if (++freecount >=100000 ) { + overflow[mtype][order] = true; +break ; + } + } + freecounts[mtype][order] = freecount; +spin_unlock_irq (&zone->lock); +cond_resched (); +spin_lock_irq (&zone->lock); + } + } + +/* printing completely outside the lock */ +spin_unlock_irq (&zone->lock); +for (mtype =0 ; mtype <MIGRATE_TYPES ; mtype++) { +seq_printf (m,"Node %4d, zone %8s, type %12s " , + pgdat->node_id, + zone->name, + migratetype_names[mtype]); + +for (order =0 ; order <NR_PAGE_ORDERS ; ++order) { + seq_printf(m, "%s%6lu ",
mm/vmstat.c static void pagetypeinfo_showfree_print(struct seq_file *m, · line 1674
Quoted 10 lines
mm/vmstat.c +7
-0+ overflow[mtype][order] ?">" :"" , + freecounts[mtype][order]); + } +seq_putc (m,'\n' ); + } +spin_lock_irq (&zone->lock); +}/* Print out the free pages at each order for each migratetype */ static void pagetypeinfo_showfree (struct seq_file *m,void *arg)
Yes. I included the data in [1]. This seq_printf loop costs about 5 us on my machine. I'll look into optimizing this in the next revision.