Skip to contentSkip to conversation

mm: add per-migratetype counts to buddy allocator and optimize pagetypeinfo access

PATCH In review Hongru Zhang 8 participants 25 messages Nov 28, 2025 – Apr 5

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover mm: add per-migratetype counts to buddy allocator and optimize pagetypeinfo access

On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.

Under high load scenarios, reading /proc/pagetypeinfo causes memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention

Changes:
1. Add per-migratetype counts to the buddy allocator to track free page block counts for each migratetype and order
2. Optimize /proc/pagetypeinfo access by utilizing these per-migratetype counts instead of iterating through free lists under zone lock

Performance testing shows following improvements:
1. /proc/pagetypeinfo access latency reduced

    +-----------------------+----------+------------+
    |                       | no-patch | with-patch |
    +-----------------------+----------+------------+
    |    Just after boot    | 700.9 us |  268.6 us  |
    +-----------------------+----------+------------+
    | After building kernel |  28.7 ms |  269.8 us  |
    +-----------------------+----------+------------+

2. When /proc/pagetypeinfo is accessed concurrently, memory allocation and free performance degradation is reduced compared to the previous implementation

Test setup:
- Using config-pagealloc-micro
- Monitor set to proc-pagetypeinfo, update frequency set to 10ms
- PAGEALLOC_ORDER_MIN=4, PAGEALLOC_ORDER_MAX=4

Without patch test results:

                                         vanilla                vanilla
                                       no-monitor               monitor
Min       alloc-odr4-1        8539.00 (   0.00%)     8762.00 (  -2.61%)
Min       alloc-odr4-2        6501.00 (   0.00%)     6683.00 (  -2.80%)
Min       alloc-odr4-4        5537.00 (   0.00%)     5873.00 (  -6.07%)
Min       alloc-odr4-8        5030.00 (   0.00%)     5361.00 (  -6.58%)
Min       alloc-odr4-16       4782.00 (   0.00%)     5162.00 (  -7.95%)
Min       alloc-odr4-32       5838.00 (   0.00%)     6499.00 ( -11.32%)
Min       alloc-odr4-64       6565.00 (   0.00%)     7413.00 ( -12.92%)
Min       alloc-odr4-128      6896.00 (   0.00%)     7898.00 ( -14.53%)
Min       alloc-odr4-256      7303.00 (   0.00%)     8163.00 ( -11.78%)
Min       alloc-odr4-512     10179.00 (   0.00%)    11985.00 ( -17.74%)
Min       alloc-odr4-1024    11000.00 (   0.00%)    12165.00 ( -10.59%)
Min       free-odr4-1          820.00 (   0.00%)     1230.00 ( -50.00%)
Min       free-odr4-2          511.00 (   0.00%)      952.00 ( -86.30%)
Min       free-odr4-4          347.00 (   0.00%)      434.00 ( -25.07%)
Min       free-odr4-8          286.00 (   0.00%)      399.00 ( -39.51%)
Min       free-odr4-16         250.00 (   0.00%)      405.00 ( -62.00%)
Min       free-odr4-32         294.00 (   0.00%)      405.00 ( -37.76%)
Min       free-odr4-64         333.00 (   0.00%)      363.00 (  -9.01%)
Min       free-odr4-128        340.00 (   0.00%)      412.00 ( -21.18%)
Min       free-odr4-256        339.00 (   0.00%)      329.00 (   2.95%)
Min       free-odr4-512        361.00 (   0.00%)      409.00 ( -13.30%)
Min       free-odr4-1024       300.00 (   0.00%)      361.00 ( -20.33%)
Stddev    alloc-odr4-1           7.29 (   0.00%)       90.78 (-1146.00%)
Stddev    alloc-odr4-2           3.87 (   0.00%)       51.30 (-1225.75%)
Stddev    alloc-odr4-4           3.20 (   0.00%)       50.90 (-1491.24%)
Stddev    alloc-odr4-8           4.67 (   0.00%)       52.23 (-1019.35%)
Stddev    alloc-odr4-16          5.72 (   0.00%)       27.53 (-381.04%)
Stddev    alloc-odr4-32          6.25 (   0.00%)      641.23 (-10154.46%)
Stddev    alloc-odr4-64          2.06 (   0.00%)      386.99 (-18714.22%)
Stddev    alloc-odr4-128        14.36 (   0.00%)       52.39 (-264.77%)
Stddev    alloc-odr4-256        32.42 (   0.00%)      326.19 (-906.05%)
Stddev    alloc-odr4-512        65.58 (   0.00%)      184.49 (-181.31%)
Stddev    alloc-odr4-1024        8.88 (   0.00%)      153.01 (-1622.67%)
Stddev    free-odr4-1            2.29 (   0.00%)      152.27 (-6549.85%)
Stddev    free-odr4-2           10.99 (   0.00%)       73.10 (-564.89%)
Stddev    free-odr4-4            1.99 (   0.00%)       28.40 (-1324.45%)
Stddev    free-odr4-8            2.51 (   0.00%)       52.93 (-2007.64%)
Stddev    free-odr4-16           2.85 (   0.00%)       26.04 (-814.88%)
Stddev    free-odr4-32           4.04 (   0.00%)       27.05 (-569.79%)
Stddev    free-odr4-64           2.10 (   0.00%)       48.07 (-2185.66%)
Stddev    free-odr4-128          2.63 (   0.00%)       26.23 (-897.86%)
Stddev    free-odr4-256          6.29 (   0.00%)       37.04 (-488.71%)
Stddev    free-odr4-512          2.56 (   0.00%)       10.65 (-315.28%)
Stddev    free-odr4-1024         0.95 (   0.00%)        6.46 (-582.22%)
Max       alloc-odr4-1        8564.00 (   0.00%)     9099.00 (  -6.25%)
Max       alloc-odr4-2        6511.00 (   0.00%)     6844.00 (  -5.11%)
Max       alloc-odr4-4        5549.00 (   0.00%)     6038.00 (  -8.81%)
Max       alloc-odr4-8        5045.00 (   0.00%)     5551.00 ( -10.03%)
Max       alloc-odr4-16       4800.00 (   0.00%)     5257.00 (  -9.52%)
Max       alloc-odr4-32       5861.00 (   0.00%)     8115.00 ( -38.46%)
Max       alloc-odr4-64       6571.00 (   0.00%)     8292.00 ( -26.19%)
Max       alloc-odr4-128      6930.00 (   0.00%)     8081.00 ( -16.61%)
Max       alloc-odr4-256      7372.00 (   0.00%)     9150.00 ( -24.12%)
Max       alloc-odr4-512     10333.00 (   0.00%)    12636.00 ( -22.29%)
Max       alloc-odr4-1024    11035.00 (   0.00%)    12590.00 ( -14.09%)
Max       free-odr4-1          828.00 (   0.00%)     1724.00 (-108.21%)
Max       free-odr4-2          543.00 (   0.00%)     1192.00 (-119.52%)
Max       free-odr4-4          354.00 (   0.00%)      519.00 ( -46.61%)
Max       free-odr4-8          293.00 (   0.00%)      617.00 (-110.58%)
Max       free-odr4-16         260.00 (   0.00%)      483.00 ( -85.77%)
Max       free-odr4-32         308.00 (   0.00%)      488.00 ( -58.44%)
Max       free-odr4-64         341.00 (   0.00%)      505.00 ( -48.09%)
Max       free-odr4-128        346.00 (   0.00%)      497.00 ( -43.64%)
Max       free-odr4-256        353.00 (   0.00%)      463.00 ( -31.16%)
Max       free-odr4-512        367.00 (   0.00%)      442.00 ( -20.44%)
Max       free-odr4-1024       303.00 (   0.00%)      381.00 ( -25.74%)

With patch test results:

                                         patched                patched
                                      no-monitor                monitor
Min       alloc-odr4-1        8488.00 (   0.00%)     8514.00 (  -0.31%)
Min       alloc-odr4-2        6551.00 (   0.00%)     6527.00 (   0.37%)
Min       alloc-odr4-4        5536.00 (   0.00%)     5591.00 (  -0.99%)
Min       alloc-odr4-8        5008.00 (   0.00%)     5098.00 (  -1.80%)
Min       alloc-odr4-16       4760.00 (   0.00%)     4857.00 (  -2.04%)
Min       alloc-odr4-32       5827.00 (   0.00%)     5919.00 (  -1.58%)
Min       alloc-odr4-64       6561.00 (   0.00%)     6680.00 (  -1.81%)
Min       alloc-odr4-128      6898.00 (   0.00%)     7014.00 (  -1.68%)
Min       alloc-odr4-256      7311.00 (   0.00%)     7464.00 (  -2.09%)
Min       alloc-odr4-512     10181.00 (   0.00%)    10286.00 (  -1.03%)
Min       alloc-odr4-1024    11205.00 (   0.00%)    11725.00 (  -4.64%)
Min       free-odr4-1          789.00 (   0.00%)      867.00 (  -9.89%)
Min       free-odr4-2          490.00 (   0.00%)      526.00 (  -7.35%)
Min       free-odr4-4          350.00 (   0.00%)      360.00 (  -2.86%)
Min       free-odr4-8          272.00 (   0.00%)      287.00 (  -5.51%)
Min       free-odr4-16         247.00 (   0.00%)      254.00 (  -2.83%)
Min       free-odr4-32         298.00 (   0.00%)      304.00 (  -2.01%)
Min       free-odr4-64         334.00 (   0.00%)      325.00 (   2.69%)
Min       free-odr4-128        334.00 (   0.00%)      329.00 (   1.50%)
Min       free-odr4-256        336.00 (   0.00%)      336.00 (   0.00%)
Min       free-odr4-512        360.00 (   0.00%)      342.00 (   5.00%)
Min       free-odr4-1024       327.00 (   0.00%)      355.00 (  -8.56%)
Stddev    alloc-odr4-1           5.19 (   0.00%)       45.38 (-775.09%)
Stddev    alloc-odr4-2           6.99 (   0.00%)       37.63 (-437.98%)
Stddev    alloc-odr4-4           3.91 (   0.00%)       17.85 (-356.28%)
Stddev    alloc-odr4-8           5.15 (   0.00%)        9.34 ( -81.47%)
Stddev    alloc-odr4-16          3.83 (   0.00%)        5.34 ( -39.34%)
Stddev    alloc-odr4-32          1.96 (   0.00%)       10.28 (-425.09%)
Stddev    alloc-odr4-64          1.32 (   0.00%)      333.30 (-25141.39%)
Stddev    alloc-odr4-128         2.06 (   0.00%)        7.37 (-258.28%)
Stddev    alloc-odr4-256        15.56 (   0.00%)      113.48 (-629.25%)
Stddev    alloc-odr4-512        61.25 (   0.00%)      165.09 (-169.53%)
Stddev    alloc-odr4-1024       18.89 (   0.00%)        2.93 (  84.51%)
Stddev    free-odr4-1            4.45 (   0.00%)       40.12 (-800.98%)
Stddev    free-odr4-2            1.50 (   0.00%)       29.30 (-1850.31%)
Stddev    free-odr4-4            1.27 (   0.00%)       19.49 (-1439.40%)
Stddev    free-odr4-8            0.97 (   0.00%)        8.93 (-823.07%)
Stddev    free-odr4-16           8.38 (   0.00%)        4.51 (  46.21%)
Stddev    free-odr4-32           3.18 (   0.00%)        6.59 (-107.42%)
Stddev    free-odr4-64           2.40 (   0.00%)        3.09 ( -28.50%)
Stddev    free-odr4-128          1.55 (   0.00%)        2.53 ( -62.92%)
Stddev    free-odr4-256          0.41 (   0.00%)        2.80 (-585.57%)
Stddev    free-odr4-512          1.60 (   0.00%)        4.84 (-202.08%)
Stddev    free-odr4-1024         0.66 (   0.00%)        1.19 ( -80.68%)
Max       alloc-odr4-1        8505.00 (   0.00%)     8676.00 (  -2.01%)
Max       alloc-odr4-2        6572.00 (   0.00%)     6651.00 (  -1.20%)
Max       alloc-odr4-4        5552.00 (   0.00%)     5646.00 (  -1.69%)
Max       alloc-odr4-8        5024.00 (   0.00%)     5131.00 (  -2.13%)
Max       alloc-odr4-16       4774.00 (   0.00%)     4875.00 (  -2.12%)
Max       alloc-odr4-32       5834.00 (   0.00%)     5950.00 (  -1.99%)
Max       alloc-odr4-64       6565.00 (   0.00%)     7434.00 ( -13.24%)
Max       alloc-odr4-128      6907.00 (   0.00%)     7034.00 (  -1.84%)
Max       alloc-odr4-256      7347.00 (   0.00%)     7843.00 (  -6.75%)
Max       alloc-odr4-512     10315.00 (   0.00%)    10866.00 (  -5.34%)
Max       alloc-odr4-1024    11278.00 (   0.00%)    11733.00 (  -4.03%)
Max       free-odr4-1          803.00 (   0.00%)     1009.00 ( -25.65%)
Max       free-odr4-2          495.00 (   0.00%)      607.00 ( -22.63%)
Max       free-odr4-4          354.00 (   0.00%)      417.00 ( -17.80%)
Max       free-odr4-8          275.00 (   0.00%)      313.00 ( -13.82%)
Max       free-odr4-16         273.00 (   0.00%)      272.00 (   0.37%)
Max       free-odr4-32         309.00 (   0.00%)      324.00 (  -4.85%)
Max       free-odr4-64         340.00 (   0.00%)      335.00 (   1.47%)
Max       free-odr4-128        340.00 (   0.00%)      338.00 (   0.59%)
Max       free-odr4-256        338.00 (   0.00%)      346.00 (  -2.37%)
Max       free-odr4-512        364.00 (   0.00%)      359.00 (   1.37%)
Max       free-odr4-1024       329.00 (   0.00%)      359.00 (  -9.12%)

The main overhead is a slight increase in latency on the memory allocation and free paths due to additional per-migratetype counting, with theoretically minimal impact on overall performance.

This patch series is based on v6.18-rc7

Hongru Zhang (3):
mm/page_alloc: add per-migratetype counts to buddy allocator mm/vmstat: get fragmentation statistics from per-migragetype count
mm: optimize free_area_empty() check using per-migratetype counts

 include/linux/mmzone.h |  1 +
 mm/internal.h          |  2 +-
 mm/mm_init.c           |  1 +
 mm/page_alloc.c        |  9 ++++++++-
 mm/vmstat.c            | 30 +++++++-----------------------
 5 files changed, 18 insertions(+), 25 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/3 mm/page_alloc: add per-migratetype counts to buddy allocator

From: Hongru Zhang <[email protected]>

On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.

Under high load scenarios, reading /proc/pagetypeinfo causes memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention

This patch adds per-migratetype counts to the buddy allocator in preparation for optimizing /proc/pagetypeinfo access.

The optimized implementation:
- Make per-migratetype count updates protected by zone lock on the write side while /proc/pagetypeinfo reads are lock-free, which reduces interrupt-disabled spinlock duration and improves system real-time performance (addressing issue #1)
- Reduce blocking time for memory management components when reading /proc/pagetypeinfo, enabling more rapid acquisition of memory fragmentation information (addressing issue #2)
- Minimize the critical section held during /proc/pagetypeinfo reads to reduce zone lock contention on memory allocation and free paths (addressing issue #3)

The main overhead is a slight increase in latency on the memory allocation and free paths due to additional per-migratetype counting, with theoretically minimal impact on overall performance.

---
 include/linux/mmzone.h | 1 +
 mm/mm_init.c           | 1 +
 mm/page_alloc.c        | 7 ++++++-
 3 files changed, 8 insertions(+), 1 deletion(-)
include/linux/mmzone.h +1 -0
index 7fb7331c5725..6eeefe6a3727 100644
@@ -138,6 +138,7 @@ extern int page_group_by_mobility_disabled;
 struct free_area {
 	struct list_head	free_list[MIGRATE_TYPES];
 	unsigned long		nr_free;
+	unsigned long		mt_nr_free[MIGRATE_TYPES];
 };
 
 struct pglist_data;
mm/mm_init.c +1 -0
index 7712d887b696..dca2be8cc3b1 100644
@@ -1439,6 +1439,7 @@ static void __meminit zone_init_free_lists(struct zone *zone)
 	for_each_migratetype_order(order, t) {
 		INIT_LIST_HEAD(&zone->free_area[order].free_list[t]);
 		zone->free_area[order].nr_free = 0;
+		zone->free_area[order].mt_nr_free[t] = 0;
 	}
 
 #ifdef CONFIG_UNACCEPTED_MEMORY
mm/page_alloc.c +6 -1
index ed82ee55e66a..9431073e7255 100644
@@ -818,6 +818,7 @@ static inline void __add_to_free_list(struct page *page, struct zone *zone,
 	else
 		list_add(&page->buddy_list, &area->free_list[migratetype]);
 	area->nr_free++;
+	area->mt_nr_free[migratetype]++;
 
 	if (order >= pageblock_order && !is_migrate_isolate(migratetype))
 		__mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, nr_pages);
@@ -840,6 +841,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone,
 		     get_pageblock_migratetype(page), old_mt, nr_pages);
 
 	list_move_tail(&page->buddy_list, &area->free_list[new_mt]);
+	area->mt_nr_free[old_mt]--;
+	area->mt_nr_free[new_mt]++;
 
 	account_freepages(zone, -nr_pages, old_mt);
 	account_freepages(zone, nr_pages, new_mt);
@@ -855,6 +858,7 @@ static inline void move_to_free_list(struct page *page, struct zone *zone,
 static inline void __del_page_from_free_list(struct page *page, struct zone *zone,
 					     unsigned int order, int migratetype)
 {
+	struct free_area *area = &zone->free_area[order];
 	int nr_pages = 1 << order;
 
         VM_WARN_ONCE(get_pageblock_migratetype(page) != migratetype,
@@ -868,7 +872,8 @@ static inline void __del_page_from_free_list(struct page *page, struct zone *zon
 	list_del(&page->buddy_list);
 	__ClearPageBuddy(page);
 	set_page_private(page, 0);
-	zone->free_area[order].nr_free--;
+	area->nr_free--;
+	area->mt_nr_free[migratetype]--;
 
 	if (order >= pageblock_order && !is_migrate_isolate(migratetype))
 		__mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, -nr_pages);
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH 1/3] mm/page_alloc: add per-migratetype counts to buddy allocator

Quoted 1 line · Hongru Zhang

[...]

Quoted 13 lines
mm/page_alloc.c +1 -0
index ed82ee55e66a..9431073e7255 100644
@@ -818,6 +818,7 @@ static inline void __add_to_free_list(struct page *page, struct zone *zone,
        else
                list_add(&page->buddy_list, &area->free_list[migratetype]);
        area->nr_free++;
+       area->mt_nr_free[migratetype]++;
 
        if (order >= pageblock_order && !is_migrate_isolate(migratetype))
                __mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, nr_pages);

mm/page_alloc.c static inline void move_to_free_list(struct page *page, struct zone *zone, · line 845

Quoted 5 lines
mm/page_alloc.c +2 -0
@@ -840,6 +841,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone,
                     get_pageblock_migratetype(page), old_mt, nr_pages);
 
        list_move_tail(&page->buddy_list, &area->free_list[new_mt]);
+       area->mt_nr_free[old_mt]--;
+       area->mt_nr_free[new_mt]++;

The overhead comes from effectively counting twice. Have we checked whether the readers of area->nr_free are on a hot path? If not, we might just drop nr_free and compute the sum each time.

Buddyinfo and compaction do not seem to be on a hot path ?

Thanks
Barry

#3 lore raw

PATCH 2/3 mm/vmstat: get fragmentation statistics from per-migragetype count

In reply to Hongru Zhang

From: Hongru Zhang <[email protected]>

This patch optimizes /proc/pagetypeinfo access by utilizing the per-migratetype free page block counts already maintained by the buddy allocator, instead of iterating through free lists under zone lock.

Accuracy. Both implementations have accuracy limitations. The previous implementation required acquiring and releasing the zone lock for counting each order and migratetype, making it potentially inaccurate. Under high memory pressure, accuracy would further degrade due to zone lock contention or fragmentation. The new implementation collects data within a short time window, which helps maintain relatively small errors, and is unaffected by memory pressure. Furthermore, user-space memory management components inherently experience decision latency - by the time they process the collected data and execute actions, the memory state has already changed. This means that even perfectly accurate data at collection time becomes stale by decision time. Considering these factors, the accuracy trade-off introduced by the new implementation should be acceptable for practical use cases, offering a balance between performance and accuracy requirements.

Performance benefits:

System setup:
- 12th Gen Intel(R) Core(TM) i7-12700
- 1 NUMA node, 16G memory in total
- Turbo disabled
- cpufreq governor set to performance

1. Average latency over 10,000 /proc/pagetypeinfo accesses

+-----------------------+----------+------------+
|                       | no-patch | with-patch |
+-----------------------+----------+------------+
|    Just after boot    | 700.9 us |  268.6 us  |
+-----------------------+----------+------------+
| After building kernel |  28.7 ms |  269.8 us  |
+-----------------------+----------+------------+

2. Page alloc/free latency with concurrent /proc/pagetypeinfo access Test setup:
- Using config-pagealloc-micro
- Monitor set to proc-pagetypeinfo, update frequency set to 10ms
- PAGEALLOC_ORDER_MIN=4, PAGEALLOC_ORDER_MAX=4

Without patch test results:

                                         vanilla                vanilla
                                       no-monitor               monitor
Min       alloc-odr4-1        8539.00 (   0.00%)     8762.00 (  -2.61%)
Min       alloc-odr4-2        6501.00 (   0.00%)     6683.00 (  -2.80%)
Min       alloc-odr4-4        5537.00 (   0.00%)     5873.00 (  -6.07%)
Min       alloc-odr4-8        5030.00 (   0.00%)     5361.00 (  -6.58%)
Min       alloc-odr4-16       4782.00 (   0.00%)     5162.00 (  -7.95%)
Min       alloc-odr4-32       5838.00 (   0.00%)     6499.00 ( -11.32%)
Min       alloc-odr4-64       6565.00 (   0.00%)     7413.00 ( -12.92%)
Min       alloc-odr4-128      6896.00 (   0.00%)     7898.00 ( -14.53%)
Min       alloc-odr4-256      7303.00 (   0.00%)     8163.00 ( -11.78%)
Min       alloc-odr4-512     10179.00 (   0.00%)    11985.00 ( -17.74%)
Min       alloc-odr4-1024    11000.00 (   0.00%)    12165.00 ( -10.59%)
Min       free-odr4-1          820.00 (   0.00%)     1230.00 ( -50.00%)
Min       free-odr4-2          511.00 (   0.00%)      952.00 ( -86.30%)
Min       free-odr4-4          347.00 (   0.00%)      434.00 ( -25.07%)
Min       free-odr4-8          286.00 (   0.00%)      399.00 ( -39.51%)
Min       free-odr4-16         250.00 (   0.00%)      405.00 ( -62.00%)
Min       free-odr4-32         294.00 (   0.00%)      405.00 ( -37.76%)
Min       free-odr4-64         333.00 (   0.00%)      363.00 (  -9.01%)
Min       free-odr4-128        340.00 (   0.00%)      412.00 ( -21.18%)
Min       free-odr4-256        339.00 (   0.00%)      329.00 (   2.95%)
Min       free-odr4-512        361.00 (   0.00%)      409.00 ( -13.30%)
Min       free-odr4-1024       300.00 (   0.00%)      361.00 ( -20.33%)
Stddev    alloc-odr4-1           7.29 (   0.00%)       90.78 (-1146.00%)
Stddev    alloc-odr4-2           3.87 (   0.00%)       51.30 (-1225.75%)
Stddev    alloc-odr4-4           3.20 (   0.00%)       50.90 (-1491.24%)
Stddev    alloc-odr4-8           4.67 (   0.00%)       52.23 (-1019.35%)
Stddev    alloc-odr4-16          5.72 (   0.00%)       27.53 (-381.04%)
Stddev    alloc-odr4-32          6.25 (   0.00%)      641.23 (-10154.46%)
Stddev    alloc-odr4-64          2.06 (   0.00%)      386.99 (-18714.22%)
Stddev    alloc-odr4-128        14.36 (   0.00%)       52.39 (-264.77%)
Stddev    alloc-odr4-256        32.42 (   0.00%)      326.19 (-906.05%)
Stddev    alloc-odr4-512        65.58 (   0.00%)      184.49 (-181.31%)
Stddev    alloc-odr4-1024        8.88 (   0.00%)      153.01 (-1622.67%)
Stddev    free-odr4-1            2.29 (   0.00%)      152.27 (-6549.85%)
Stddev    free-odr4-2           10.99 (   0.00%)       73.10 (-564.89%)
Stddev    free-odr4-4            1.99 (   0.00%)       28.40 (-1324.45%)
Stddev    free-odr4-8            2.51 (   0.00%)       52.93 (-2007.64%)
Stddev    free-odr4-16           2.85 (   0.00%)       26.04 (-814.88%)
Stddev    free-odr4-32           4.04 (   0.00%)       27.05 (-569.79%)
Stddev    free-odr4-64           2.10 (   0.00%)       48.07 (-2185.66%)
Stddev    free-odr4-128          2.63 (   0.00%)       26.23 (-897.86%)
Stddev    free-odr4-256          6.29 (   0.00%)       37.04 (-488.71%)
Stddev    free-odr4-512          2.56 (   0.00%)       10.65 (-315.28%)
Stddev    free-odr4-1024         0.95 (   0.00%)        6.46 (-582.22%)
Max       alloc-odr4-1        8564.00 (   0.00%)     9099.00 (  -6.25%)
Max       alloc-odr4-2        6511.00 (   0.00%)     6844.00 (  -5.11%)
Max       alloc-odr4-4        5549.00 (   0.00%)     6038.00 (  -8.81%)
Max       alloc-odr4-8        5045.00 (   0.00%)     5551.00 ( -10.03%)
Max       alloc-odr4-16       4800.00 (   0.00%)     5257.00 (  -9.52%)
Max       alloc-odr4-32       5861.00 (   0.00%)     8115.00 ( -38.46%)
Max       alloc-odr4-64       6571.00 (   0.00%)     8292.00 ( -26.19%)
Max       alloc-odr4-128      6930.00 (   0.00%)     8081.00 ( -16.61%)
Max       alloc-odr4-256      7372.00 (   0.00%)     9150.00 ( -24.12%)
Max       alloc-odr4-512     10333.00 (   0.00%)    12636.00 ( -22.29%)
Max       alloc-odr4-1024    11035.00 (   0.00%)    12590.00 ( -14.09%)
Max       free-odr4-1          828.00 (   0.00%)     1724.00 (-108.21%)
Max       free-odr4-2          543.00 (   0.00%)     1192.00 (-119.52%)
Max       free-odr4-4          354.00 (   0.00%)      519.00 ( -46.61%)
Max       free-odr4-8          293.00 (   0.00%)      617.00 (-110.58%)
Max       free-odr4-16         260.00 (   0.00%)      483.00 ( -85.77%)
Max       free-odr4-32         308.00 (   0.00%)      488.00 ( -58.44%)
Max       free-odr4-64         341.00 (   0.00%)      505.00 ( -48.09%)
Max       free-odr4-128        346.00 (   0.00%)      497.00 ( -43.64%)
Max       free-odr4-256        353.00 (   0.00%)      463.00 ( -31.16%)
Max       free-odr4-512        367.00 (   0.00%)      442.00 ( -20.44%)
Max       free-odr4-1024       303.00 (   0.00%)      381.00 ( -25.74%)

With patch test results:

                                         patched                patched
                                      no-monitor                monitor
Min       alloc-odr4-1        8488.00 (   0.00%)     8514.00 (  -0.31%)
Min       alloc-odr4-2        6551.00 (   0.00%)     6527.00 (   0.37%)
Min       alloc-odr4-4        5536.00 (   0.00%)     5591.00 (  -0.99%)
Min       alloc-odr4-8        5008.00 (   0.00%)     5098.00 (  -1.80%)
Min       alloc-odr4-16       4760.00 (   0.00%)     4857.00 (  -2.04%)
Min       alloc-odr4-32       5827.00 (   0.00%)     5919.00 (  -1.58%)
Min       alloc-odr4-64       6561.00 (   0.00%)     6680.00 (  -1.81%)
Min       alloc-odr4-128      6898.00 (   0.00%)     7014.00 (  -1.68%)
Min       alloc-odr4-256      7311.00 (   0.00%)     7464.00 (  -2.09%)
Min       alloc-odr4-512     10181.00 (   0.00%)    10286.00 (  -1.03%)
Min       alloc-odr4-1024    11205.00 (   0.00%)    11725.00 (  -4.64%)
Min       free-odr4-1          789.00 (   0.00%)      867.00 (  -9.89%)
Min       free-odr4-2          490.00 (   0.00%)      526.00 (  -7.35%)
Min       free-odr4-4          350.00 (   0.00%)      360.00 (  -2.86%)
Min       free-odr4-8          272.00 (   0.00%)      287.00 (  -5.51%)
Min       free-odr4-16         247.00 (   0.00%)      254.00 (  -2.83%)
Min       free-odr4-32         298.00 (   0.00%)      304.00 (  -2.01%)
Min       free-odr4-64         334.00 (   0.00%)      325.00 (   2.69%)
Min       free-odr4-128        334.00 (   0.00%)      329.00 (   1.50%)
Min       free-odr4-256        336.00 (   0.00%)      336.00 (   0.00%)
Min       free-odr4-512        360.00 (   0.00%)      342.00 (   5.00%)
Min       free-odr4-1024       327.00 (   0.00%)      355.00 (  -8.56%)
Stddev    alloc-odr4-1           5.19 (   0.00%)       45.38 (-775.09%)
Stddev    alloc-odr4-2           6.99 (   0.00%)       37.63 (-437.98%)
Stddev    alloc-odr4-4           3.91 (   0.00%)       17.85 (-356.28%)
Stddev    alloc-odr4-8           5.15 (   0.00%)        9.34 ( -81.47%)
Stddev    alloc-odr4-16          3.83 (   0.00%)        5.34 ( -39.34%)
Stddev    alloc-odr4-32          1.96 (   0.00%)       10.28 (-425.09%)
Stddev    alloc-odr4-64          1.32 (   0.00%)      333.30 (-25141.39%)
Stddev    alloc-odr4-128         2.06 (   0.00%)        7.37 (-258.28%)
Stddev    alloc-odr4-256        15.56 (   0.00%)      113.48 (-629.25%)
Stddev    alloc-odr4-512        61.25 (   0.00%)      165.09 (-169.53%)
Stddev    alloc-odr4-1024       18.89 (   0.00%)        2.93 (  84.51%)
Stddev    free-odr4-1            4.45 (   0.00%)       40.12 (-800.98%)
Stddev    free-odr4-2            1.50 (   0.00%)       29.30 (-1850.31%)
Stddev    free-odr4-4            1.27 (   0.00%)       19.49 (-1439.40%)
Stddev    free-odr4-8            0.97 (   0.00%)        8.93 (-823.07%)
Stddev    free-odr4-16           8.38 (   0.00%)        4.51 (  46.21%)
Stddev    free-odr4-32           3.18 (   0.00%)        6.59 (-107.42%)
Stddev    free-odr4-64           2.40 (   0.00%)        3.09 ( -28.50%)
Stddev    free-odr4-128          1.55 (   0.00%)        2.53 ( -62.92%)
Stddev    free-odr4-256          0.41 (   0.00%)        2.80 (-585.57%)
Stddev    free-odr4-512          1.60 (   0.00%)        4.84 (-202.08%)
Stddev    free-odr4-1024         0.66 (   0.00%)        1.19 ( -80.68%)
Max       alloc-odr4-1        8505.00 (   0.00%)     8676.00 (  -2.01%)
Max       alloc-odr4-2        6572.00 (   0.00%)     6651.00 (  -1.20%)
Max       alloc-odr4-4        5552.00 (   0.00%)     5646.00 (  -1.69%)
Max       alloc-odr4-8        5024.00 (   0.00%)     5131.00 (  -2.13%)
Max       alloc-odr4-16       4774.00 (   0.00%)     4875.00 (  -2.12%)
Max       alloc-odr4-32       5834.00 (   0.00%)     5950.00 (  -1.99%)
Max       alloc-odr4-64       6565.00 (   0.00%)     7434.00 ( -13.24%)
Max       alloc-odr4-128      6907.00 (   0.00%)     7034.00 (  -1.84%)
Max       alloc-odr4-256      7347.00 (   0.00%)     7843.00 (  -6.75%)
Max       alloc-odr4-512     10315.00 (   0.00%)    10866.00 (  -5.34%)
Max       alloc-odr4-1024    11278.00 (   0.00%)    11733.00 (  -4.03%)
Max       free-odr4-1          803.00 (   0.00%)     1009.00 ( -25.65%)
Max       free-odr4-2          495.00 (   0.00%)      607.00 ( -22.63%)
Max       free-odr4-4          354.00 (   0.00%)      417.00 ( -17.80%)
Max       free-odr4-8          275.00 (   0.00%)      313.00 ( -13.82%)
Max       free-odr4-16         273.00 (   0.00%)      272.00 (   0.37%)
Max       free-odr4-32         309.00 (   0.00%)      324.00 (  -4.85%)
Max       free-odr4-64         340.00 (   0.00%)      335.00 (   1.47%)
Max       free-odr4-128        340.00 (   0.00%)      338.00 (   0.59%)
Max       free-odr4-256        338.00 (   0.00%)      346.00 (  -2.37%)
Max       free-odr4-512        364.00 (   0.00%)      359.00 (   1.37%)
Max       free-odr4-1024       329.00 (   0.00%)      359.00 (  -9.12%)
---
 mm/page_alloc.c | 10 ++++++----
 mm/vmstat.c     | 30 +++++++-----------------------
 2 files changed, 13 insertions(+), 27 deletions(-)
mm/page_alloc.c +6 -4
index 9431073e7255..a90f2bf735f6 100644
@@ -818,7 +818,8 @@ static inline void __add_to_free_list(struct page *page, struct zone *zone,
 	else
 		list_add(&page->buddy_list, &area->free_list[migratetype]);
 	area->nr_free++;
-	area->mt_nr_free[migratetype]++;
+	WRITE_ONCE(area->mt_nr_free[migratetype],
+		area->mt_nr_free[migratetype] + 1);
 
 	if (order >= pageblock_order && !is_migrate_isolate(migratetype))
 		__mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, nr_pages);
@@ -841,8 +842,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone,
 		     get_pageblock_migratetype(page), old_mt, nr_pages);
 
 	list_move_tail(&page->buddy_list, &area->free_list[new_mt]);
-	area->mt_nr_free[old_mt]--;
-	area->mt_nr_free[new_mt]++;
+	WRITE_ONCE(area->mt_nr_free[old_mt], area->mt_nr_free[old_mt] - 1);
+	WRITE_ONCE(area->mt_nr_free[new_mt], area->mt_nr_free[new_mt] + 1);
 
 	account_freepages(zone, -nr_pages, old_mt);
 	account_freepages(zone, nr_pages, new_mt);
@@ -873,7 +874,8 @@ static inline void __del_page_from_free_list(struct page *page, struct zone *zon
 	__ClearPageBuddy(page);
 	set_page_private(page, 0);
 	area->nr_free--;
-	area->mt_nr_free[migratetype]--;
+	WRITE_ONCE(area->mt_nr_free[migratetype],
+		area->mt_nr_free[migratetype] - 1);
 
 	if (order >= pageblock_order && !is_migrate_isolate(migratetype))
 		__mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, -nr_pages);
mm/vmstat.c +7 -23
index bb09c032eecf..9334bbbe1e16 100644
@@ -1590,32 +1590,16 @@ static void pagetypeinfo_showfree_print(struct seq_file *m,
 					zone->name,
 					migratetype_names[mtype]);
 		for (order = 0; order < NR_PAGE_ORDERS; ++order) {
-			unsigned long freecount = 0;
-			struct free_area *area;
-			struct list_head *curr;
+			unsigned long freecount;
 			bool overflow = false;
 
-			area = &(zone->free_area[order]);
-
-			list_for_each(curr, &area->free_list[mtype]) {
-				/*
-				 * Cap the free_list iteration because it might
-				 * be really large and we are under a spinlock
-				 * so a long time spent here could trigger a
-				 * hard lockup detector. Anyway this is a
-				 * debugging tool so knowing there is a handful
-				 * of pages of this order should be more than
-				 * sufficient.
-				 */
-				if (++freecount >= 100000) {
-					overflow = true;
-					break;
-				}
+			/* Keep the same output format for user-space tools compatibility */
+			freecount = READ_ONCE(zone->free_area[order].mt_nr_free[mtype]);
+			if (freecount >= 100000) {
+				overflow = true;
+				freecount = 100000;
 			}
 			seq_printf(m, "%s%6lu ", overflow ? ">" : "", freecount);
-			spin_unlock_irq(&zone->lock);
-			cond_resched();
-			spin_lock_irq(&zone->lock);
 		}
 		seq_putc(m, '\n');
 	}
@@ -1633,7 +1617,7 @@ static void pagetypeinfo_showfree(struct seq_file *m, void *arg)
 		seq_printf(m, "%6d ", order);
 	seq_putc(m, '\n');
 
-	walk_zones_in_node(m, pgdat, true, false, pagetypeinfo_showfree_print);
+	walk_zones_in_node(m, pgdat, true, true, pagetypeinfo_showfree_print);
 }
 
 static void pagetypeinfo_showblockcount_print(struct seq_file *m,
Signature · 1 line
2.43.0
#4 lore raw

Re: [PATCH 2/3] mm/vmstat: get fragmentation statistics from per-migragetype count

Hi, Hongru

Quoted 26 lines
mm/page_alloc.c +4 -3
index 9431073e7255..a90f2bf735f6 100644
@@ -818,7 +818,8 @@ static inline void __add_to_free_list(struct page *page, struct zone *zone,
 	else
 		list_add(&page->buddy_list, &area->free_list[migratetype]);
 	area->nr_free++;
-	area->mt_nr_free[migratetype]++;
+	WRITE_ONCE(area->mt_nr_free[migratetype],
+		area->mt_nr_free[migratetype] + 1);
 
 	if (order >= pageblock_order && !is_migrate_isolate(migratetype))
 		__mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, nr_pages);
@@ -841,8 +842,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone,
 		     get_pageblock_migratetype(page), old_mt, nr_pages);
 
 	list_move_tail(&page->buddy_list, &area->free_list[new_mt]);
-	area->mt_nr_free[old_mt]--;
-	area->mt_nr_free[new_mt]++;
+	WRITE_ONCE(area->mt_nr_free[old_mt], area->mt_nr_free[old_mt] - 1);
+	WRITE_ONCE(area->mt_nr_free[new_mt], area->mt_nr_free[new_mt] + 1);
 
 	account_freepages(zone, -nr_pages, old_mt);
 	account_freepages(zone, nr_pages, new_mt);

mm/page_alloc.c static inline void __del_page_from_free_list(struct page *page, struct zone *zon · line 878

Quoted 6 lines
mm/page_alloc.c +2 -1
@@ -873,7 +874,8 @@ static inline void __del_page_from_free_list(struct page *page, struct zone *zon
 	__ClearPageBuddy(page);
 	set_page_private(page, 0);
 	area->nr_free--;
-	area->mt_nr_free[migratetype]--;
+	WRITE_ONCE(area->mt_nr_free[migratetype],
+		area->mt_nr_free[migratetype] - 1);

It doesn't seem like a good idea to use WRITE_ONCE on the hot path.

Quoted 25 lines
mm/page_alloc.c +0 -0
 	if (order >= pageblock_order && !is_migrate_isolate(migratetype))
 		__mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, -nr_pages);
mm/vmstat.c +1 -11
index bb09c032eecf..9334bbbe1e16 100644
@@ -1590,32 +1590,16 @@ static void pagetypeinfo_showfree_print(struct seq_file *m,
 					zone->name,
 					migratetype_names[mtype]);
 		for (order = 0; order < NR_PAGE_ORDERS; ++order) {
-			unsigned long freecount = 0;
-			struct free_area *area;
-			struct list_head *curr;
+			unsigned long freecount;
 			bool overflow = false;
 
-			area = &(zone->free_area[order]);
-
-			list_for_each(curr, &area->free_list[mtype]) {
-				/*
-				 * Cap the free_list iteration because it might
-				 * be really large and we are under a spinlock
-				 * so a long time spent here could trigger a
-				 * hard lockup detector. Anyway this is a

mm/vmstat.c static void pagetypeinfo_showfree_print(struct seq_file *m, · line 1597

Quoted 10 lines
mm/vmstat.c +2 -8
-				 * debugging tool so knowing there is a handful
-				 * of pages of this order should be more than
-				 * sufficient.
-				 */
-				if (++freecount >= 100000) {
-					overflow = true;
-					break;
-				}
+			/* Keep the same output format for user-space tools compatibility */
+			freecount = READ_ONCE(zone->free_area[order].mt_nr_free[mtype]);

I think it might be better for using an array of size NR_PAGE_ORDERS to store the free count for each order. Like the code below.

unsigned long freecount[NR_PAGE_ORDERS]

spin_lock_irq(&zone->lock)

	for_each_order
		freecount[order] = zone->free_area[order].mt_nr_free[mtype]

spin_unlock_irq(&zone->lock)

for_each_order
print freecount[order]

Quoted 18 lines
mm/vmstat.c +4 -4
+			if (freecount >= 100000) {
+				overflow = true;
+				freecount = 100000;
 			}
 			seq_printf(m, "%s%6lu ", overflow ? ">" : "", freecount);
-			spin_unlock_irq(&zone->lock);
-			cond_resched();
-			spin_lock_irq(&zone->lock);
 		}
 		seq_putc(m, '\n');
 	}
@@ -1633,7 +1617,7 @@ static void pagetypeinfo_showfree(struct seq_file *m, void *arg)
 		seq_printf(m, "%6d ", order);
 	seq_putc(m, '\n');
 
-	walk_zones_in_node(m, pgdat, true, false, pagetypeinfo_showfree_print);
+	walk_zones_in_node(m, pgdat, true, true, pagetypeinfo_showfree_print);
 }
#5 lore raw

Re: [PATCH 2/3] mm/vmstat: get fragmentation statistics from per-migragetype count

Quoted 29 lines
Quoted 29 lines
      if (order >= pageblock_order && !is_migrate_isolate(migratetype))
              __mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, -nr_pages);
mm/vmstat.c +1 -16
index bb09c032eecf..9334bbbe1e16 100644
@@ -1590,32 +1590,16 @@ static void pagetypeinfo_showfree_print(struct seq_file *m,
                                      zone->name,
                                      migratetype_names[mtype]);
              for (order = 0; order < NR_PAGE_ORDERS; ++order) {
-                     unsigned long freecount = 0;
-                     struct free_area *area;
-                     struct list_head *curr;
+                     unsigned long freecount;
                      bool overflow = false;
 
-                     area = &(zone->free_area[order]);
-
-                     list_for_each(curr, &area->free_list[mtype]) {
-                             /*
-                              * Cap the free_list iteration because it might
-                              * be really large and we are under a spinlock
-                              * so a long time spent here could trigger a
-                              * hard lockup detector. Anyway this is a
-                              * debugging tool so knowing there is a handful
-                              * of pages of this order should be more than
-                              * sufficient.
-                              */
-                             if (++freecount >= 100000) {
Quoted 8 lines

mm/vmstat.c static void pagetypeinfo_showfree_print(struct seq_file *m, · line 1597

Quoted 5 lines
mm/vmstat.c +2 -3
-                                     overflow = true;
-                                     break;
-                             }
+                     /* Keep the same output format for user-space tools compatibility */
+                     freecount = READ_ONCE(zone->free_area[order].mt_nr_free[mtype]);

I think it might be better for using an array of size NR_PAGE_ORDERS to store the free count for each order. Like the code below.

Right. If we want the freecount to accurately reflect the current system state, we still need to take the zone lock.

Multiple independent WRITE_ONCE and READ_ONCE operations do not guarantee correctness. They may ensure single-copy atomicity per access, but not for the overall result.

Thanks
Barry

#6 lore raw

Re: [PATCH 2/3] mm/vmstat: get fragmentation statistics from per-migragetype count

Quoted 39 lines · Barry Song
Quoted 30 lines
Quoted 30 lines
      if (order >= pageblock_order && !is_migrate_isolate(migratetype))
              __mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, -nr_pages);
mm/vmstat.c +1 -16
index bb09c032eecf..9334bbbe1e16 100644
@@ -1590,32 +1590,16 @@ static void pagetypeinfo_showfree_print(struct seq_file *m,
                                      zone->name,
                                      migratetype_names[mtype]);
              for (order = 0; order < NR_PAGE_ORDERS; ++order) {
-                     unsigned long freecount = 0;
-                     struct free_area *area;
-                     struct list_head *curr;
+                     unsigned long freecount;
                      bool overflow = false;
 
-                     area = &(zone->free_area[order]);
-
-                     list_for_each(curr, &area->free_list[mtype]) {
-                             /*
-                              * Cap the free_list iteration because it might
-                              * be really large and we are under a spinlock
-                              * so a long time spent here could trigger a
-                              * hard lockup detector. Anyway this is a
-                              * debugging tool so knowing there is a handful
-                              * of pages of this order should be more than
-                              * sufficient.
-                              */
-                             if (++freecount >= 100000) {
Quoted 8 lines

mm/vmstat.c static void pagetypeinfo_showfree_print(struct seq_file *m, · line 1597

Quoted 5 lines
mm/vmstat.c +2 -3
-                                     overflow = true;
-                                     break;
-                             }
+                     /* Keep the same output format for user-space tools compatibility */
+                     freecount = READ_ONCE(zone->free_area[order].mt_nr_free[mtype]);

I think it might be better for using an array of size NR_PAGE_ORDERS to store the free count for each order. Like the code below.

Quoted 6 lines

Right. If we want the freecount to accurately reflect the current system state, we still need to take the zone lock.

Multiple independent WRITE_ONCE and READ_ONCE operations do not guarantee correctness. They may ensure single-copy atomicity per access, but not for the overall result.

On second thought, the original code releases and re-acquires the spinlock for each order, so cross-variable consistency may not be a real issue. Adding data_race() to silence KCSAN warnings should be sufficient? I mean something like the following.

@@ -843,8 +842,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone,

                     get_pageblock_migratetype(page), old_mt, nr_pages);

        list_move_tail(&page->buddy_list, &area->free_list[new_mt]);
-       WRITE_ONCE(area->mt_nr_free[old_mt], area->mt_nr_free[old_mt] - 1);
-       WRITE_ONCE(area->mt_nr_free[new_mt], area->mt_nr_free[new_mt] + 1);
+       area->mt_nr_free[old_mt]--;
+       area->mt_nr_free[new_mt]++;
        account_freepages(zone, -nr_pages, old_mt);
        account_freepages(zone, nr_pages, new_mt);
@@ -875,8 +874,7 @@ static inline void
__del_page_from_free_list(struct page *page, struct zone *zon
        __ClearPageBuddy(page);
        set_page_private(page, 0);
        area->nr_free--;

- WRITE_ONCE(area->mt_nr_free[migratetype],
- area->mt_nr_free[migratetype] - 1);
+ area->mt_nr_free[migratetype]--;

        if (order >= pageblock_order && !is_migrate_isolate(migratetype))
                __mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, -nr_pages);
mm/vmstat.c +0 -0
index 7e1e931eb209..d74004eb8c4d 100644
@@ -1599,7 +1599,7 @@ static void pagetypeinfo_showfree_print(struct

seq_file *m,

                        bool overflow = false;

                        /* Keep the same output format for user-space

tools compatibility */
- freecount =
READ_ONCE(zone->free_area[order].mt_nr_free[mtype]);
+ freecount =

data_race(zone->free_area[order].mt_nr_free[mtype]);
                        if (freecount >= 100000) {
                                overflow = true;
                                freecount = 100000;

Thanks
Barry

#7 lore raw

Re: [PATCH 2/3] mm/vmstat: get fragmentation statistics from per-migragetype count

In reply to Barry Song
Quoted 2 lines

Right. If we want the freecount to accurately reflect the current system state, we still need to take the zone lock.

Yeah, as I mentioned in patch (2/3), this implementation has accuracy limitation:

"Accuracy. Both implementations have accuracy limitations. The previous implementation required acquiring and releasing the zone lock for counting each order and migratetype, making it potentially inaccurate. Under high memory pressure, accuracy would further degrade due to zone lock contention or fragmentation. The new implementation collects data within a short time window, which helps maintain relatively small errors, and is unaffected by memory pressure. Furthermore, user-space memory management components inherently experience decision latency - by the time they process the collected data and execute actions, the memory state has already changed. This means that even perfectly accurate data at collection time becomes stale by decision time. Considering these factors, the accuracy trade-off introduced by the new implementation should be acceptable for practical use cases, offering a balance between performance and accuracy requirements."

Additional data:
1. average latency of pagetypeinfo_showfree_print() over 1,000,000 times is 4.67 us

2. average latency is 125 ns, if seq_printf() is taken out of the loop

Example code:

+unsigned long total_lat = 0;
+unsigned long total_count = 0;
+
static void pagetypeinfo_showfree_print(struct seq_file *m,
pg_data_t *pgdat, struct zone *zone)

 {
        int order, mtype;
+       ktime_t start;
+       u64 lat;
+       unsigned long freecounts[NR_PAGE_ORDERS][MIGRATE_TYPES]; /* ignore potential stack overflow */

+
+ start = ktime_get();
+ for (order = 0; order < NR_PAGE_ORDERS; ++order)
+ for (mtype = 0; mtype < MIGRATE_TYPES; mtype++)
+ freecounts[order][mtype] = READ_ONCE(zone->free_area[order].mt_nr_free[mtype]);
+

+       lat = ktime_to_ns(ktime_sub(ktime_get(), start));
+       total_count++;
+       total_lat += lat;
        for (mtype = 0; mtype < MIGRATE_TYPES; mtype++) {
                seq_printf(m, "Node %4d, zone %8s, type %12s ",
@@ -1594,7 +1609,7 @@ static void pagetypeinfo_showfree_print(struct seq_file *m,
                        bool overflow = false;
 
                        /* Keep the same output format for user-space tools compatibility */
-                       freecount = READ_ONCE(zone->free_area[order].mt_nr_free[mtype]);
+                       freecount = freecounts[order][mtype];
                        if (freecount >= 100000) {
                                overflow = true;
                                freecount = 100000;

@@ -1692,6 +1707,13 @@ static void pagetypeinfo_showmixedcount(struct seq_file *m, pg_data_t *pgdat)
#endif /* CONFIG_PAGE_OWNER */
}

I think both are small time window (if IRQ is disabled, latency is more deterministic).

Quoted 3 lines

Multiple independent WRITE_ONCE and READ_ONCE operations do not guarantee correctness. They may ensure single-copy atomicity per access, but not for the overall result.

I know this does not guarantee correctness of the overall result. READ_ONCE() and WRITE_ONCE() in this patch are used to avoid potential store tearing and read tearing caused by compiler optimizations.

In fact, I have already noticed /proc/buddyinfo, which collects data under zone lock and uses data_race to avoid KCSAN reports. But I'm wondering if we could remove its zone lock as well, for the same reasons as /proc/pagetypeinfo.

#8 lore raw

Re: [PATCH 2/3] mm/vmstat: get fragmentation statistics from per-migragetype count

Quoted 68 lines · Hongru Zhang
Quoted 3 lines

Right. If we want the freecount to accurately reflect the current system state, we still need to take the zone lock.

Yeah, as I mentioned in patch (2/3), this implementation has accuracy limitation:

"Accuracy. Both implementations have accuracy limitations. The previous implementation required acquiring and releasing the zone lock for counting each order and migratetype, making it potentially inaccurate. Under high memory pressure, accuracy would further degrade due to zone lock contention or fragmentation. The new implementation collects data within a short time window, which helps maintain relatively small errors, and is unaffected by memory pressure. Furthermore, user-space memory management components inherently experience decision latency - by the time they process the collected data and execute actions, the memory state has already changed. This means that even perfectly accurate data at collection time becomes stale by decision time. Considering these factors, the accuracy trade-off introduced by the new implementation should be acceptable for practical use cases, offering a balance between performance and accuracy requirements."

Additional data:
1. average latency of pagetypeinfo_showfree_print() over 1,000,000 times is 4.67 us

2. average latency is 125 ns, if seq_printf() is taken out of the loop

Example code:

(quoted diff) +16 -1
+unsigned long total_lat = 0;
+unsigned long total_count = 0;
+
 static void pagetypeinfo_showfree_print(struct seq_file *m,
                                        pg_data_t *pgdat, struct zone *zone)
 {
        int order, mtype;
+       ktime_t start;
+       u64 lat;
+       unsigned long freecounts[NR_PAGE_ORDERS][MIGRATE_TYPES]; /* ignore potential stack overflow */
+
+       start = ktime_get();
+       for (order = 0; order < NR_PAGE_ORDERS; ++order)
+               for (mtype = 0; mtype < MIGRATE_TYPES; mtype++)
+                       freecounts[order][mtype] = READ_ONCE(zone->free_area[order].mt_nr_free[mtype]);
+
+       lat = ktime_to_ns(ktime_sub(ktime_get(), start));
+       total_count++;
+       total_lat += lat;
 
        for (mtype = 0; mtype < MIGRATE_TYPES; mtype++) {
                seq_printf(m, "Node %4d, zone %8s, type %12s ",
@@ -1594,7 +1609,7 @@ static void pagetypeinfo_showfree_print(struct seq_file *m,
                        bool overflow = false;
 
                        /* Keep the same output format for user-space tools compatibility */
-                       freecount = READ_ONCE(zone->free_area[order].mt_nr_free[mtype]);
+                       freecount = freecounts[order][mtype];
                        if (freecount >= 100000) {
                                overflow = true;
                                freecount = 100000;
@@ -1692,6 +1707,13 @@ static void pagetypeinfo_showmixedcount(struct seq_file *m, pg_data_t *pgdat)
 #endif /* CONFIG_PAGE_OWNER */
 }
 

I think both are small time window (if IRQ is disabled, latency is more deterministic).

Quoted 7 lines
Quoted 3 lines

Multiple independent WRITE_ONCE and READ_ONCE operations do not guarantee correctness. They may ensure single-copy atomicity per access, but not for the overall result.

I know this does not guarantee correctness of the overall result. READ_ONCE() and WRITE_ONCE() in this patch are used to avoid potential store tearing and read tearing caused by compiler optimizations.

Yes, I realized that correctness might not be a major concern, so I sent a follow-up email [1] after replying to you.

Quoted 5 lines

In fact, I have already noticed /proc/buddyinfo, which collects data under zone lock and uses data_race to avoid KCSAN reports. But I'm wondering if we could remove its zone lock as well, for the same reasons as /proc/pagetypeinfo.

That might be correct. However, if it doesn’t significantly affect performance and buddyinfo is accessed much less frequently than the buddy list, we may just leave it as is.

[1] https://lore.kernel.org/linux-mm/CAGsJ_4wUQdQyB_3y0Buf3uG34hvgpMAP3qHHwJM3=R01RJOuvw@mail.gmail.com/

Thanks
Barry

#9 lore raw

PATCH 3/3 mm: optimize free_area_empty() check using per-migratetype counts

In reply to Hongru Zhang

From: Hongru Zhang <[email protected]>

Use per-migratetype counts instead of list_empty() helps reduce a few cpu instructions.

---
 mm/internal.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/internal.h +1 -1
index 1561fc2ff5b8..7759f8fdf445 100644
@@ -954,7 +954,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,
 
 static inline bool free_area_empty(struct free_area *area, int migratetype)
 {
-	return list_empty(&area->free_list[migratetype]);
+	return !READ_ONCE(area->mt_nr_free[migratetype]);
 }
 
 /* mm/util.c */
Signature · 1 line
2.43.0
#10 lore raw

Re: [PATCH 3/3] mm: optimize free_area_empty() check using per-migratetype counts

Quoted 16 lines · Hongru Zhang

From: Hongru Zhang <[email protected]>

Use per-migratetype counts instead of list_empty() helps reduce a few cpu instructions.

---
 mm/internal.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/internal.h +0 -0
index 1561fc2ff5b8..7759f8fdf445 100644

mm/internal.h int find_suitable_fallback(struct free_area *area, unsigned int order, · line 957

Quoted 5 lines
mm/internal.h +1 -1
@@ -954,7 +954,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,
 
 static inline bool free_area_empty(struct free_area *area, int migratetype)
 {
-       return list_empty(&area->free_list[migratetype]);
+       return !READ_ONCE(area->mt_nr_free[migratetype]);

I'm not quite sure about this. Since the counter is written and read more frequently, cache coherence traffic may actually be higher than for the list head.

I'd prefer to drop this unless there is real data showing it performs better.

Thanks
Barry

#11 lore raw

Re: [PATCH 3/3] mm: optimize free_area_empty() check using per-migratetype counts

Quoted 28 lines · Barry Song
Quoted 16 lines · Hongru Zhang

From: Hongru Zhang <[email protected]>

Use per-migratetype counts instead of list_empty() helps reduce a few cpu instructions.

---
 mm/internal.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/internal.h +0 -0
index 1561fc2ff5b8..7759f8fdf445 100644

mm/internal.h int find_suitable_fallback(struct free_area *area, unsigned int order, · line 957

Quoted 5 lines
mm/internal.h +1 -1
@@ -954,7 +954,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,
 
 static inline bool free_area_empty(struct free_area *area, int migratetype)
 {
-       return list_empty(&area->free_list[migratetype]);
+       return !READ_ONCE(area->mt_nr_free[migratetype]);

I'm not quite sure about this. Since the counter is written and read more frequently, cache coherence traffic may actually be higher than for the list head.

Quoted 1 line

I'd prefer to drop this unless there is real data showing it performs better.

If the goal is to optimize free_area list checks and list_add, a reasonable approach is to organize the data structure to reduce false sharing between different mt and order entries.

struct mt_free_area {
        struct list_head        free_list;
        unsigned long           nr_free;
} ____cacheline_aligned;
struct free_area {
        struct mt_free_area     mt_free_area[MIGRATE_TYPES];
};

However, without supporting data, it’s unclear if the space increase is justified :-)

Thanks
Barry

#12 lore raw

Re: [PATCH 3/3] mm: optimize free_area_empty() check using per-migratetype counts

Quoted 40 lines · Barry Song
Quoted 28 lines · Hongru Zhang
Quoted 16 lines

From: Hongru Zhang <[email protected]>

Use per-migratetype counts instead of list_empty() helps reduce a few cpu instructions.

---
 mm/internal.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/internal.h +0 -0
index 1561fc2ff5b8..7759f8fdf445 100644

mm/internal.h int find_suitable_fallback(struct free_area *area, unsigned int order, · line 957

Quoted 5 lines
mm/internal.h +1 -1
@@ -954,7 +954,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,
 
 static inline bool free_area_empty(struct free_area *area, int migratetype)
 {
-       return list_empty(&area->free_list[migratetype]);
+       return !READ_ONCE(area->mt_nr_free[migratetype]);

I'm not quite sure about this. Since the counter is written and read more frequently, cache coherence traffic may actually be higher than for the list head.

Quoted 1 line

I'd prefer to drop this unless there is real data showing it performs better.

If the goal is to optimize free_area list checks and list_add, a reasonable approach is to organize the data structure to reduce false sharing between different mt and order entries.

struct mt_free_area {
        struct list_head        free_list;
        unsigned long           nr_free;
} ____cacheline_aligned;
Quoted 6 lines
struct free_area {
        struct mt_free_area     mt_free_area[MIGRATE_TYPES];
};

However, without supporting data, it’s unclear if the space increase is justified :-)

I designed a test model to trigger more false sharing and collected data under it to see which layout performs better.

Test model
- Based on the microbench that was removed from mmtests
Commit: beeaeb89 ("pagealloc: Remove bit-rotted benchmark")
- Goal: Generate concurrent kernel page alloc/free activity across multiple orders and migratetypes to observe cacheline sharing and contention in the buddy free_area
- Mechanism: A systemtap module exposes a write-only /proc/mmtests-pagealloc-micro. Writing a 64-bit encoded value triggers repeated page alloc/free in kernel space
- bits 7:0 -> mt (0=UNMOVABLE, 1=MOVABLE, 2=RECLAIMABLE)
- bits 15:8 -> order
- bits 63:16 -> batch
- Workload distribution:
- order = cpu % 4 (orders 0/1/2/3)
- mt = cpu % 3 (UNMOVABLE/MOVABLE/RECLAIMABLE)
- cpu0 and cpu1 are not used for the test
- Sampling:
- load stap
- determine encoded value according to cpu id and bind it to the cpu
- after a short delay, runs 'perf mem record' for 100s
- unload stap
- Test tool:
- https://gist.github.com/zhr250/72e56f87ac703e833b11b5341d616cb0
- Data analysis tool:
- https://gist.github.com/zhr250/f4a385ffa9fae2993d22748f31e18588

CPU topo info of my machine:
Package L#0

    NUMANode L#0 (P#0 15GB)
    L3 L#0 (25MB)
      L2 L#0 (1280KB) + L1d L#0 (48KB) + L1i L#0 (32KB) + Core L#0
        PU L#0 (P#0)
        PU L#1 (P#1)
      L2 L#1 (1280KB) + L1d L#1 (48KB) + L1i L#1 (32KB) + Core L#1
        PU L#2 (P#2)
        PU L#3 (P#3)
      L2 L#2 (1280KB) + L1d L#2 (48KB) + L1i L#2 (32KB) + Core L#2
        PU L#4 (P#4)
        PU L#5 (P#5)
      L2 L#3 (1280KB) + L1d L#3 (48KB) + L1i L#3 (32KB) + Core L#3
        PU L#6 (P#6)
        PU L#7 (P#7)
      L2 L#4 (1280KB) + L1d L#4 (48KB) + L1i L#4 (32KB) + Core L#4
        PU L#8 (P#8)
        PU L#9 (P#9)
      L2 L#5 (1280KB) + L1d L#5 (48KB) + L1i L#5 (32KB) + Core L#5
        PU L#10 (P#10)
        PU L#11 (P#11)
      L2 L#6 (1280KB) + L1d L#6 (48KB) + L1i L#6 (32KB) + Core L#6
        PU L#12 (P#12)
        PU L#13 (P#13)
      L2 L#7 (1280KB) + L1d L#7 (48KB) + L1i L#7 (32KB) + Core L#7
        PU L#14 (P#14)
        PU L#15 (P#15)
      L2 L#8 (2048KB)
        L1d L#8 (32KB) + L1i L#8 (64KB) + Core L#8 + PU L#16 (P#16)
        L1d L#9 (32KB) + L1i L#9 (64KB) + Core L#9 + PU L#17 (P#17)
        L1d L#10 (32KB) + L1i L#10 (64KB) + Core L#10 + PU L#18 (P#18)
        L1d L#11 (32KB) + L1i L#11 (64KB) + Core L#11 + PU L#19 (P#19)

Actual (order, mt) distribution on my machine:
order=0, mt=0: cpu12
order=0, mt=1: cpu4, cpu16
order=0, mt=2: cpu8
order=1, mt=0: cpu9
order=1, mt=1: cpu13
order=1, mt=2: cpu5, cpu17
order=2, mt=0: cpu6, cpu18
order=2, mt=1: cpu10
order=2, mt=2: cpu2, cpu14
order=3, mt=0: cpu3, cpu15
order=3, mt=1: cpu7, cpu19
order=3, mt=2: cpu11

Different migratetype/order combinations are placed on CPUs that do not share L1/L2 caches to maximize cacheline contention. For our test goal, I think this distribution is relatively reasonable. I ran 10 rounds for each kernel and found the data to be stable.

Layouts tested (capturing load/store samples in free_area[0..MAX_PAGE_ORDER]):
- vanilla kernel:

    struct free_area
    {
            struct list_head free_list[MIGRATE_TYPES];
            unsigned long    nr_free;
    };
- patched kernel:
    struct free_area {
            struct list_head free_list[MIGRATE_TYPES];
            unsigned long    nr_free;
   +	    unsigned long    mt_nr_free[MIGRATE_TYPES];
    };

- mtlist kernel:
+struct mt_free_list {
+ struct list_head list;
+ unsigned long nr_free;
+};
+

    struct free_area {
   -        struct list_head    free_list[MIGRATE_TYPES];
   +        struct mt_free_list	mt_free_list[MIGRATE_TYPES];
            unsigned long       nr_free;
    };

summary:

+---------+-----------------+-----------------+------------------------+---------------+---------------------+---------------+
|  Kernel | inrange samples |     HitM (%)    | L1 hit inc LFB/MAB (%) |   L2 hit (%)  | L3 hit inc HitM (%) |  RAM hit (%)  |
+---------+-----------------+-----------------+------------------------+---------------+---------------------+---------------+
| vanilla |     192,468     | 45,421 (23.60%) |     94,486 (49.09%)    | 1,952 (1.01%) |   91,240 (47.41%)   | 4,790 (2.49%) |
+---------+-----------------+-----------------+------------------------+---------------+---------------------+---------------+
| patched |     227,196     | 27,293 (12.01%) |    165,238 (72.73%)    | 1,194 (0.53%) |   54,609 (24.04%)   | 6,155 (2.71%) |
+---------+-----------------+-----------------+------------------------+---------------+---------------------+---------------+
|  mtlist |     240,694     | 50,911 (21.15%) |    132,827 (55.19%)    | 3,165 (1.31%) |   98,556 (40.95%)   | 6,146 (2.55%) |
+---------+-----------------+-----------------+------------------------+---------------+---------------------+---------------+

Detailed data:
- https://gist.github.com/zhr250/2ccf8902080ecaf85477d9c051e72a96

For both L1 hit and HitM, the patched kernel is the best among the three.

In this test model, I also collected memory allocation counts. The patched kernel delivers the best performance — about 7.00% higher than vanilla and 4.93% higher than mtlist.

Detailed data:
https://gist.github.com/zhr250/4439523b7ca3c18f4a2d2c97b24c4965

#13 lore raw

Re: [PATCH 3/3] mm: optimize free_area_empty() check using per-migratetype counts

In reply to Barry Song
Quoted 26 lines
Quoted 16 lines

From: Hongru Zhang <[email protected]>

Use per-migratetype counts instead of list_empty() helps reduce a few cpu instructions.

---
 mm/internal.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/internal.h +0 -0
index 1561fc2ff5b8..7759f8fdf445 100644

mm/internal.h int find_suitable_fallback(struct free_area *area, unsigned int order, · line 957

Quoted 5 lines
mm/internal.h +1 -1
@@ -954,7 +954,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,
 
 static inline bool free_area_empty(struct free_area *area, int migratetype)
 {
-       return list_empty(&area->free_list[migratetype]);
+       return !READ_ONCE(area->mt_nr_free[migratetype]);

I'm not quite sure about this. Since the counter is written and read more frequently, cache coherence traffic may actually be higher than for the list head.

Quoted 1 line

I'd prefer to drop this unless there is real data showing it performs better.

Under the model described in email [1], with this patch (3/3), memory allocation performance is 0.36% better than without it.

Detailed data:
- https://gist.github.com/zhr250/4439523b7ca3c18f4a2d2c97b24c4965

Reference:
[1] https://lore.kernel.org/linux-mm/[email protected]/

reviewer
#14 lore raw

Re: [PATCH 3/3] mm: optimize free_area_empty() check using per-migratetype counts

Quoted 27 lines · Hongru Zhang
Quoted 27 lines
Quoted 17 lines

From: Hongru Zhang <[email protected]>

Use per-migratetype counts instead of list_empty() helps reduce a few cpu instructions.

---
 mm/internal.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/internal.h +0 -0
index 1561fc2ff5b8..7759f8fdf445 100644

mm/internal.h int find_suitable_fallback(struct free_area *area, unsigned int order, · line 957

Quoted 5 lines
mm/internal.h +1 -1
@@ -954,7 +954,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,
 
 static inline bool free_area_empty(struct free_area *area, int migratetype)
 {
-       return list_empty(&area->free_list[migratetype]);
+       return !READ_ONCE(area->mt_nr_free[migratetype]);

I'm not quite sure about this. Since the counter is written and read more frequently, cache coherence traffic may actually be higher than for the list head.

Quoted 4 lines
Quoted 1 line

I'd prefer to drop this unless there is real data showing it performs better.

Under the model described in email [1], with this patch (3/3), memory allocation performance is 0.36% better than without it.

I'm fine with the data, though my actual suggestion was to place each mt_free_area in a separate cacheline to avoid false sharing. That one could be separated if you see an obvious performance improvement.

For this patchset, I feel the main issue is convincing people of the value of increasing counters in the alloc/free hotpath. As Johannes commented [1], the new counters should provide more use cases than pagetypeinfo, or as Zi Yan commented [2], clearly demonstrate the value that userspace derives from pagetypeinfo.

Right now, pagetypeinfo is considered a debug interface, so its implementation is quite poor.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Quoted 6 lines

Detailed data:
- https://gist.github.com/zhr250/4439523b7ca3c18f4a2d2c97b24c4965

Reference:
[1] https://lore.kernel.org/linux-mm/[email protected]/

Thanks
Barry

#15 lore raw
In reply to Hongru Zhang

Just a general plea :) could we please try not to send larger series like this so late.

We're at the last day before the merge window, this is better sent during 6.19-rc1 or if now as an RFC.

Thanks, Lorenzo

Quoted 204 lines · Hongru Zhang

On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.

Under high load scenarios, reading /proc/pagetypeinfo causes memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention

Changes:
1. Add per-migratetype counts to the buddy allocator to track free page block counts for each migratetype and order
2. Optimize /proc/pagetypeinfo access by utilizing these per-migratetype counts instead of iterating through free lists under zone lock

Performance testing shows following improvements:
1. /proc/pagetypeinfo access latency reduced

    +-----------------------+----------+------------+
    |                       | no-patch | with-patch |
    +-----------------------+----------+------------+
    |    Just after boot    | 700.9 us |  268.6 us  |
    +-----------------------+----------+------------+
    | After building kernel |  28.7 ms |  269.8 us  |
    +-----------------------+----------+------------+

2. When /proc/pagetypeinfo is accessed concurrently, memory allocation and free performance degradation is reduced compared to the previous implementation

Test setup:
- Using config-pagealloc-micro
- Monitor set to proc-pagetypeinfo, update frequency set to 10ms
- PAGEALLOC_ORDER_MIN=4, PAGEALLOC_ORDER_MAX=4

Without patch test results:

                                         vanilla                vanilla
                                       no-monitor               monitor
Min       alloc-odr4-1        8539.00 (   0.00%)     8762.00 (  -2.61%)
Min       alloc-odr4-2        6501.00 (   0.00%)     6683.00 (  -2.80%)
Min       alloc-odr4-4        5537.00 (   0.00%)     5873.00 (  -6.07%)
Min       alloc-odr4-8        5030.00 (   0.00%)     5361.00 (  -6.58%)
Min       alloc-odr4-16       4782.00 (   0.00%)     5162.00 (  -7.95%)
Min       alloc-odr4-32       5838.00 (   0.00%)     6499.00 ( -11.32%)
Min       alloc-odr4-64       6565.00 (   0.00%)     7413.00 ( -12.92%)
Min       alloc-odr4-128      6896.00 (   0.00%)     7898.00 ( -14.53%)
Min       alloc-odr4-256      7303.00 (   0.00%)     8163.00 ( -11.78%)
Min       alloc-odr4-512     10179.00 (   0.00%)    11985.00 ( -17.74%)
Min       alloc-odr4-1024    11000.00 (   0.00%)    12165.00 ( -10.59%)
Min       free-odr4-1          820.00 (   0.00%)     1230.00 ( -50.00%)
Min       free-odr4-2          511.00 (   0.00%)      952.00 ( -86.30%)
Min       free-odr4-4          347.00 (   0.00%)      434.00 ( -25.07%)
Min       free-odr4-8          286.00 (   0.00%)      399.00 ( -39.51%)
Min       free-odr4-16         250.00 (   0.00%)      405.00 ( -62.00%)
Min       free-odr4-32         294.00 (   0.00%)      405.00 ( -37.76%)
Min       free-odr4-64         333.00 (   0.00%)      363.00 (  -9.01%)
Min       free-odr4-128        340.00 (   0.00%)      412.00 ( -21.18%)
Min       free-odr4-256        339.00 (   0.00%)      329.00 (   2.95%)
Min       free-odr4-512        361.00 (   0.00%)      409.00 ( -13.30%)
Min       free-odr4-1024       300.00 (   0.00%)      361.00 ( -20.33%)
Stddev    alloc-odr4-1           7.29 (   0.00%)       90.78 (-1146.00%)
Stddev    alloc-odr4-2           3.87 (   0.00%)       51.30 (-1225.75%)
Stddev    alloc-odr4-4           3.20 (   0.00%)       50.90 (-1491.24%)
Stddev    alloc-odr4-8           4.67 (   0.00%)       52.23 (-1019.35%)
Stddev    alloc-odr4-16          5.72 (   0.00%)       27.53 (-381.04%)
Stddev    alloc-odr4-32          6.25 (   0.00%)      641.23 (-10154.46%)
Stddev    alloc-odr4-64          2.06 (   0.00%)      386.99 (-18714.22%)
Stddev    alloc-odr4-128        14.36 (   0.00%)       52.39 (-264.77%)
Stddev    alloc-odr4-256        32.42 (   0.00%)      326.19 (-906.05%)
Stddev    alloc-odr4-512        65.58 (   0.00%)      184.49 (-181.31%)
Stddev    alloc-odr4-1024        8.88 (   0.00%)      153.01 (-1622.67%)
Stddev    free-odr4-1            2.29 (   0.00%)      152.27 (-6549.85%)
Stddev    free-odr4-2           10.99 (   0.00%)       73.10 (-564.89%)
Stddev    free-odr4-4            1.99 (   0.00%)       28.40 (-1324.45%)
Stddev    free-odr4-8            2.51 (   0.00%)       52.93 (-2007.64%)
Stddev    free-odr4-16           2.85 (   0.00%)       26.04 (-814.88%)
Stddev    free-odr4-32           4.04 (   0.00%)       27.05 (-569.79%)
Stddev    free-odr4-64           2.10 (   0.00%)       48.07 (-2185.66%)
Stddev    free-odr4-128          2.63 (   0.00%)       26.23 (-897.86%)
Stddev    free-odr4-256          6.29 (   0.00%)       37.04 (-488.71%)
Stddev    free-odr4-512          2.56 (   0.00%)       10.65 (-315.28%)
Stddev    free-odr4-1024         0.95 (   0.00%)        6.46 (-582.22%)
Max       alloc-odr4-1        8564.00 (   0.00%)     9099.00 (  -6.25%)
Max       alloc-odr4-2        6511.00 (   0.00%)     6844.00 (  -5.11%)
Max       alloc-odr4-4        5549.00 (   0.00%)     6038.00 (  -8.81%)
Max       alloc-odr4-8        5045.00 (   0.00%)     5551.00 ( -10.03%)
Max       alloc-odr4-16       4800.00 (   0.00%)     5257.00 (  -9.52%)
Max       alloc-odr4-32       5861.00 (   0.00%)     8115.00 ( -38.46%)
Max       alloc-odr4-64       6571.00 (   0.00%)     8292.00 ( -26.19%)
Max       alloc-odr4-128      6930.00 (   0.00%)     8081.00 ( -16.61%)
Max       alloc-odr4-256      7372.00 (   0.00%)     9150.00 ( -24.12%)
Max       alloc-odr4-512     10333.00 (   0.00%)    12636.00 ( -22.29%)
Max       alloc-odr4-1024    11035.00 (   0.00%)    12590.00 ( -14.09%)
Max       free-odr4-1          828.00 (   0.00%)     1724.00 (-108.21%)
Max       free-odr4-2          543.00 (   0.00%)     1192.00 (-119.52%)
Max       free-odr4-4          354.00 (   0.00%)      519.00 ( -46.61%)
Max       free-odr4-8          293.00 (   0.00%)      617.00 (-110.58%)
Max       free-odr4-16         260.00 (   0.00%)      483.00 ( -85.77%)
Max       free-odr4-32         308.00 (   0.00%)      488.00 ( -58.44%)
Max       free-odr4-64         341.00 (   0.00%)      505.00 ( -48.09%)
Max       free-odr4-128        346.00 (   0.00%)      497.00 ( -43.64%)
Max       free-odr4-256        353.00 (   0.00%)      463.00 ( -31.16%)
Max       free-odr4-512        367.00 (   0.00%)      442.00 ( -20.44%)
Max       free-odr4-1024       303.00 (   0.00%)      381.00 ( -25.74%)

With patch test results:

                                         patched                patched
                                      no-monitor                monitor
Min       alloc-odr4-1        8488.00 (   0.00%)     8514.00 (  -0.31%)
Min       alloc-odr4-2        6551.00 (   0.00%)     6527.00 (   0.37%)
Min       alloc-odr4-4        5536.00 (   0.00%)     5591.00 (  -0.99%)
Min       alloc-odr4-8        5008.00 (   0.00%)     5098.00 (  -1.80%)
Min       alloc-odr4-16       4760.00 (   0.00%)     4857.00 (  -2.04%)
Min       alloc-odr4-32       5827.00 (   0.00%)     5919.00 (  -1.58%)
Min       alloc-odr4-64       6561.00 (   0.00%)     6680.00 (  -1.81%)
Min       alloc-odr4-128      6898.00 (   0.00%)     7014.00 (  -1.68%)
Min       alloc-odr4-256      7311.00 (   0.00%)     7464.00 (  -2.09%)
Min       alloc-odr4-512     10181.00 (   0.00%)    10286.00 (  -1.03%)
Min       alloc-odr4-1024    11205.00 (   0.00%)    11725.00 (  -4.64%)
Min       free-odr4-1          789.00 (   0.00%)      867.00 (  -9.89%)
Min       free-odr4-2          490.00 (   0.00%)      526.00 (  -7.35%)
Min       free-odr4-4          350.00 (   0.00%)      360.00 (  -2.86%)
Min       free-odr4-8          272.00 (   0.00%)      287.00 (  -5.51%)
Min       free-odr4-16         247.00 (   0.00%)      254.00 (  -2.83%)
Min       free-odr4-32         298.00 (   0.00%)      304.00 (  -2.01%)
Min       free-odr4-64         334.00 (   0.00%)      325.00 (   2.69%)
Min       free-odr4-128        334.00 (   0.00%)      329.00 (   1.50%)
Min       free-odr4-256        336.00 (   0.00%)      336.00 (   0.00%)
Min       free-odr4-512        360.00 (   0.00%)      342.00 (   5.00%)
Min       free-odr4-1024       327.00 (   0.00%)      355.00 (  -8.56%)
Stddev    alloc-odr4-1           5.19 (   0.00%)       45.38 (-775.09%)
Stddev    alloc-odr4-2           6.99 (   0.00%)       37.63 (-437.98%)
Stddev    alloc-odr4-4           3.91 (   0.00%)       17.85 (-356.28%)
Stddev    alloc-odr4-8           5.15 (   0.00%)        9.34 ( -81.47%)
Stddev    alloc-odr4-16          3.83 (   0.00%)        5.34 ( -39.34%)
Stddev    alloc-odr4-32          1.96 (   0.00%)       10.28 (-425.09%)
Stddev    alloc-odr4-64          1.32 (   0.00%)      333.30 (-25141.39%)
Stddev    alloc-odr4-128         2.06 (   0.00%)        7.37 (-258.28%)
Stddev    alloc-odr4-256        15.56 (   0.00%)      113.48 (-629.25%)
Stddev    alloc-odr4-512        61.25 (   0.00%)      165.09 (-169.53%)
Stddev    alloc-odr4-1024       18.89 (   0.00%)        2.93 (  84.51%)
Stddev    free-odr4-1            4.45 (   0.00%)       40.12 (-800.98%)
Stddev    free-odr4-2            1.50 (   0.00%)       29.30 (-1850.31%)
Stddev    free-odr4-4            1.27 (   0.00%)       19.49 (-1439.40%)
Stddev    free-odr4-8            0.97 (   0.00%)        8.93 (-823.07%)
Stddev    free-odr4-16           8.38 (   0.00%)        4.51 (  46.21%)
Stddev    free-odr4-32           3.18 (   0.00%)        6.59 (-107.42%)
Stddev    free-odr4-64           2.40 (   0.00%)        3.09 ( -28.50%)
Stddev    free-odr4-128          1.55 (   0.00%)        2.53 ( -62.92%)
Stddev    free-odr4-256          0.41 (   0.00%)        2.80 (-585.57%)
Stddev    free-odr4-512          1.60 (   0.00%)        4.84 (-202.08%)
Stddev    free-odr4-1024         0.66 (   0.00%)        1.19 ( -80.68%)
Max       alloc-odr4-1        8505.00 (   0.00%)     8676.00 (  -2.01%)
Max       alloc-odr4-2        6572.00 (   0.00%)     6651.00 (  -1.20%)
Max       alloc-odr4-4        5552.00 (   0.00%)     5646.00 (  -1.69%)
Max       alloc-odr4-8        5024.00 (   0.00%)     5131.00 (  -2.13%)
Max       alloc-odr4-16       4774.00 (   0.00%)     4875.00 (  -2.12%)
Max       alloc-odr4-32       5834.00 (   0.00%)     5950.00 (  -1.99%)
Max       alloc-odr4-64       6565.00 (   0.00%)     7434.00 ( -13.24%)
Max       alloc-odr4-128      6907.00 (   0.00%)     7034.00 (  -1.84%)
Max       alloc-odr4-256      7347.00 (   0.00%)     7843.00 (  -6.75%)
Max       alloc-odr4-512     10315.00 (   0.00%)    10866.00 (  -5.34%)
Max       alloc-odr4-1024    11278.00 (   0.00%)    11733.00 (  -4.03%)
Max       free-odr4-1          803.00 (   0.00%)     1009.00 ( -25.65%)
Max       free-odr4-2          495.00 (   0.00%)      607.00 ( -22.63%)
Max       free-odr4-4          354.00 (   0.00%)      417.00 ( -17.80%)
Max       free-odr4-8          275.00 (   0.00%)      313.00 ( -13.82%)
Max       free-odr4-16         273.00 (   0.00%)      272.00 (   0.37%)
Max       free-odr4-32         309.00 (   0.00%)      324.00 (  -4.85%)
Max       free-odr4-64         340.00 (   0.00%)      335.00 (   1.47%)
Max       free-odr4-128        340.00 (   0.00%)      338.00 (   0.59%)
Max       free-odr4-256        338.00 (   0.00%)      346.00 (  -2.37%)
Max       free-odr4-512        364.00 (   0.00%)      359.00 (   1.37%)
Max       free-odr4-1024       329.00 (   0.00%)      359.00 (  -9.12%)

The main overhead is a slight increase in latency on the memory allocation and free paths due to additional per-migratetype counting, with theoretically minimal impact on overall performance.

This patch series is based on v6.18-rc7

Hongru Zhang (3):
mm/page_alloc: add per-migratetype counts to buddy allocator mm/vmstat: get fragmentation statistics from per-migragetype count
mm: optimize free_area_empty() check using per-migratetype counts

 include/linux/mmzone.h |  1 +
 mm/internal.h          |  2 +-
 mm/mm_init.c           |  1 +
 mm/page_alloc.c        |  9 ++++++++-
 mm/vmstat.c            | 30 +++++++-----------------------
 5 files changed, 18 insertions(+), 25 deletions(-)

--
2.43.0

#16 lore raw
Quoted 7 lines

Just a general plea :) could we please try not to send larger series like this so late.

We're at the last day before the merge window, this is better sent during 6.19-rc1 or if now as an RFC.

Thanks, Lorenzo

Hi Lorenzo,

Thank you for your feedback and sorry for the late submission. You're right - this series should have been sent earlier. Apologize for not following the proper submission timing guidelines. I'll make sure to follow the community norms and submit similar work well in advance in the future.

Thanks again for your patience and guidance.

#17 lore raw
Quoted 10 lines · Hongru Zhang
Quoted 7 lines

Just a general plea :) could we please try not to send larger series like this so late.

We're at the last day before the merge window, this is better sent during 6.19-rc1 or if now as an RFC.

Thanks, Lorenzo

Hi Lorenzo,

Quoted 7 lines

Thank you for your feedback and sorry for the late submission. You're right - this series should have been sent earlier. Apologize for not following the proper submission timing guidelines. I'll make sure to follow the community norms and submit similar work well in advance in the future.

Thanks again for your patience and guidance.

Hi Hongru,

Sorry I don't mean to be critical here and you weren't to know :) rather just in general - a plea for how we do things in mm.

Your series is very much appreciated, you didn't do anything wrong at all - this is just essentially an admin thing :P

We will absolutely review your series it's just about timing. And of course I'm just sort of making a point here, reviewers can choose to review as and when they want! :)

Cheers, Lorenzo

#18 lore raw
In reply to Hongru Zhang
Quoted 4 lines · Hongru Zhang

On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.

Hm /proc/buddyinfo could be enough to determine fragmentation? Also we have in-kernel proactive compaction these days.

Quoted 11 lines

Under high load scenarios, reading /proc/pagetypeinfo causes memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention

It could be argued that not capturing /proc/pagetypeinfo (often) would help. I wonder if we can find also other benefits from the counters in the kernel itself.

Adding these migratetype counters is something that wouldn't be even possible in the past, until the freelist migratetype hygiene was merged. So now it should be AFAIK possible, but it's still some overhead in relatively hot paths. I wonder if we even considered this before in the context of migratetype hygiene? Couldn't find anything quickly.

Quoted 184 lines

Changes:
1. Add per-migratetype counts to the buddy allocator to track free page block counts for each migratetype and order
2. Optimize /proc/pagetypeinfo access by utilizing these per-migratetype counts instead of iterating through free lists under zone lock

Performance testing shows following improvements:
1. /proc/pagetypeinfo access latency reduced

    +-----------------------+----------+------------+
    |                       | no-patch | with-patch |
    +-----------------------+----------+------------+
    |    Just after boot    | 700.9 us |  268.6 us  |
    +-----------------------+----------+------------+
    | After building kernel |  28.7 ms |  269.8 us  |
    +-----------------------+----------+------------+

2. When /proc/pagetypeinfo is accessed concurrently, memory allocation and free performance degradation is reduced compared to the previous implementation

Test setup:
- Using config-pagealloc-micro
- Monitor set to proc-pagetypeinfo, update frequency set to 10ms
- PAGEALLOC_ORDER_MIN=4, PAGEALLOC_ORDER_MAX=4

Without patch test results:

                                         vanilla                vanilla
                                       no-monitor               monitor
Min       alloc-odr4-1        8539.00 (   0.00%)     8762.00 (  -2.61%)
Min       alloc-odr4-2        6501.00 (   0.00%)     6683.00 (  -2.80%)
Min       alloc-odr4-4        5537.00 (   0.00%)     5873.00 (  -6.07%)
Min       alloc-odr4-8        5030.00 (   0.00%)     5361.00 (  -6.58%)
Min       alloc-odr4-16       4782.00 (   0.00%)     5162.00 (  -7.95%)
Min       alloc-odr4-32       5838.00 (   0.00%)     6499.00 ( -11.32%)
Min       alloc-odr4-64       6565.00 (   0.00%)     7413.00 ( -12.92%)
Min       alloc-odr4-128      6896.00 (   0.00%)     7898.00 ( -14.53%)
Min       alloc-odr4-256      7303.00 (   0.00%)     8163.00 ( -11.78%)
Min       alloc-odr4-512     10179.00 (   0.00%)    11985.00 ( -17.74%)
Min       alloc-odr4-1024    11000.00 (   0.00%)    12165.00 ( -10.59%)
Min       free-odr4-1          820.00 (   0.00%)     1230.00 ( -50.00%)
Min       free-odr4-2          511.00 (   0.00%)      952.00 ( -86.30%)
Min       free-odr4-4          347.00 (   0.00%)      434.00 ( -25.07%)
Min       free-odr4-8          286.00 (   0.00%)      399.00 ( -39.51%)
Min       free-odr4-16         250.00 (   0.00%)      405.00 ( -62.00%)
Min       free-odr4-32         294.00 (   0.00%)      405.00 ( -37.76%)
Min       free-odr4-64         333.00 (   0.00%)      363.00 (  -9.01%)
Min       free-odr4-128        340.00 (   0.00%)      412.00 ( -21.18%)
Min       free-odr4-256        339.00 (   0.00%)      329.00 (   2.95%)
Min       free-odr4-512        361.00 (   0.00%)      409.00 ( -13.30%)
Min       free-odr4-1024       300.00 (   0.00%)      361.00 ( -20.33%)
Stddev    alloc-odr4-1           7.29 (   0.00%)       90.78 (-1146.00%)
Stddev    alloc-odr4-2           3.87 (   0.00%)       51.30 (-1225.75%)
Stddev    alloc-odr4-4           3.20 (   0.00%)       50.90 (-1491.24%)
Stddev    alloc-odr4-8           4.67 (   0.00%)       52.23 (-1019.35%)
Stddev    alloc-odr4-16          5.72 (   0.00%)       27.53 (-381.04%)
Stddev    alloc-odr4-32          6.25 (   0.00%)      641.23 (-10154.46%)
Stddev    alloc-odr4-64          2.06 (   0.00%)      386.99 (-18714.22%)
Stddev    alloc-odr4-128        14.36 (   0.00%)       52.39 (-264.77%)
Stddev    alloc-odr4-256        32.42 (   0.00%)      326.19 (-906.05%)
Stddev    alloc-odr4-512        65.58 (   0.00%)      184.49 (-181.31%)
Stddev    alloc-odr4-1024        8.88 (   0.00%)      153.01 (-1622.67%)
Stddev    free-odr4-1            2.29 (   0.00%)      152.27 (-6549.85%)
Stddev    free-odr4-2           10.99 (   0.00%)       73.10 (-564.89%)
Stddev    free-odr4-4            1.99 (   0.00%)       28.40 (-1324.45%)
Stddev    free-odr4-8            2.51 (   0.00%)       52.93 (-2007.64%)
Stddev    free-odr4-16           2.85 (   0.00%)       26.04 (-814.88%)
Stddev    free-odr4-32           4.04 (   0.00%)       27.05 (-569.79%)
Stddev    free-odr4-64           2.10 (   0.00%)       48.07 (-2185.66%)
Stddev    free-odr4-128          2.63 (   0.00%)       26.23 (-897.86%)
Stddev    free-odr4-256          6.29 (   0.00%)       37.04 (-488.71%)
Stddev    free-odr4-512          2.56 (   0.00%)       10.65 (-315.28%)
Stddev    free-odr4-1024         0.95 (   0.00%)        6.46 (-582.22%)
Max       alloc-odr4-1        8564.00 (   0.00%)     9099.00 (  -6.25%)
Max       alloc-odr4-2        6511.00 (   0.00%)     6844.00 (  -5.11%)
Max       alloc-odr4-4        5549.00 (   0.00%)     6038.00 (  -8.81%)
Max       alloc-odr4-8        5045.00 (   0.00%)     5551.00 ( -10.03%)
Max       alloc-odr4-16       4800.00 (   0.00%)     5257.00 (  -9.52%)
Max       alloc-odr4-32       5861.00 (   0.00%)     8115.00 ( -38.46%)
Max       alloc-odr4-64       6571.00 (   0.00%)     8292.00 ( -26.19%)
Max       alloc-odr4-128      6930.00 (   0.00%)     8081.00 ( -16.61%)
Max       alloc-odr4-256      7372.00 (   0.00%)     9150.00 ( -24.12%)
Max       alloc-odr4-512     10333.00 (   0.00%)    12636.00 ( -22.29%)
Max       alloc-odr4-1024    11035.00 (   0.00%)    12590.00 ( -14.09%)
Max       free-odr4-1          828.00 (   0.00%)     1724.00 (-108.21%)
Max       free-odr4-2          543.00 (   0.00%)     1192.00 (-119.52%)
Max       free-odr4-4          354.00 (   0.00%)      519.00 ( -46.61%)
Max       free-odr4-8          293.00 (   0.00%)      617.00 (-110.58%)
Max       free-odr4-16         260.00 (   0.00%)      483.00 ( -85.77%)
Max       free-odr4-32         308.00 (   0.00%)      488.00 ( -58.44%)
Max       free-odr4-64         341.00 (   0.00%)      505.00 ( -48.09%)
Max       free-odr4-128        346.00 (   0.00%)      497.00 ( -43.64%)
Max       free-odr4-256        353.00 (   0.00%)      463.00 ( -31.16%)
Max       free-odr4-512        367.00 (   0.00%)      442.00 ( -20.44%)
Max       free-odr4-1024       303.00 (   0.00%)      381.00 ( -25.74%)

With patch test results:

                                         patched                patched
                                      no-monitor                monitor
Min       alloc-odr4-1        8488.00 (   0.00%)     8514.00 (  -0.31%)
Min       alloc-odr4-2        6551.00 (   0.00%)     6527.00 (   0.37%)
Min       alloc-odr4-4        5536.00 (   0.00%)     5591.00 (  -0.99%)
Min       alloc-odr4-8        5008.00 (   0.00%)     5098.00 (  -1.80%)
Min       alloc-odr4-16       4760.00 (   0.00%)     4857.00 (  -2.04%)
Min       alloc-odr4-32       5827.00 (   0.00%)     5919.00 (  -1.58%)
Min       alloc-odr4-64       6561.00 (   0.00%)     6680.00 (  -1.81%)
Min       alloc-odr4-128      6898.00 (   0.00%)     7014.00 (  -1.68%)
Min       alloc-odr4-256      7311.00 (   0.00%)     7464.00 (  -2.09%)
Min       alloc-odr4-512     10181.00 (   0.00%)    10286.00 (  -1.03%)
Min       alloc-odr4-1024    11205.00 (   0.00%)    11725.00 (  -4.64%)
Min       free-odr4-1          789.00 (   0.00%)      867.00 (  -9.89%)
Min       free-odr4-2          490.00 (   0.00%)      526.00 (  -7.35%)
Min       free-odr4-4          350.00 (   0.00%)      360.00 (  -2.86%)
Min       free-odr4-8          272.00 (   0.00%)      287.00 (  -5.51%)
Min       free-odr4-16         247.00 (   0.00%)      254.00 (  -2.83%)
Min       free-odr4-32         298.00 (   0.00%)      304.00 (  -2.01%)
Min       free-odr4-64         334.00 (   0.00%)      325.00 (   2.69%)
Min       free-odr4-128        334.00 (   0.00%)      329.00 (   1.50%)
Min       free-odr4-256        336.00 (   0.00%)      336.00 (   0.00%)
Min       free-odr4-512        360.00 (   0.00%)      342.00 (   5.00%)
Min       free-odr4-1024       327.00 (   0.00%)      355.00 (  -8.56%)
Stddev    alloc-odr4-1           5.19 (   0.00%)       45.38 (-775.09%)
Stddev    alloc-odr4-2           6.99 (   0.00%)       37.63 (-437.98%)
Stddev    alloc-odr4-4           3.91 (   0.00%)       17.85 (-356.28%)
Stddev    alloc-odr4-8           5.15 (   0.00%)        9.34 ( -81.47%)
Stddev    alloc-odr4-16          3.83 (   0.00%)        5.34 ( -39.34%)
Stddev    alloc-odr4-32          1.96 (   0.00%)       10.28 (-425.09%)
Stddev    alloc-odr4-64          1.32 (   0.00%)      333.30 (-25141.39%)
Stddev    alloc-odr4-128         2.06 (   0.00%)        7.37 (-258.28%)
Stddev    alloc-odr4-256        15.56 (   0.00%)      113.48 (-629.25%)
Stddev    alloc-odr4-512        61.25 (   0.00%)      165.09 (-169.53%)
Stddev    alloc-odr4-1024       18.89 (   0.00%)        2.93 (  84.51%)
Stddev    free-odr4-1            4.45 (   0.00%)       40.12 (-800.98%)
Stddev    free-odr4-2            1.50 (   0.00%)       29.30 (-1850.31%)
Stddev    free-odr4-4            1.27 (   0.00%)       19.49 (-1439.40%)
Stddev    free-odr4-8            0.97 (   0.00%)        8.93 (-823.07%)
Stddev    free-odr4-16           8.38 (   0.00%)        4.51 (  46.21%)
Stddev    free-odr4-32           3.18 (   0.00%)        6.59 (-107.42%)
Stddev    free-odr4-64           2.40 (   0.00%)        3.09 ( -28.50%)
Stddev    free-odr4-128          1.55 (   0.00%)        2.53 ( -62.92%)
Stddev    free-odr4-256          0.41 (   0.00%)        2.80 (-585.57%)
Stddev    free-odr4-512          1.60 (   0.00%)        4.84 (-202.08%)
Stddev    free-odr4-1024         0.66 (   0.00%)        1.19 ( -80.68%)
Max       alloc-odr4-1        8505.00 (   0.00%)     8676.00 (  -2.01%)
Max       alloc-odr4-2        6572.00 (   0.00%)     6651.00 (  -1.20%)
Max       alloc-odr4-4        5552.00 (   0.00%)     5646.00 (  -1.69%)
Max       alloc-odr4-8        5024.00 (   0.00%)     5131.00 (  -2.13%)
Max       alloc-odr4-16       4774.00 (   0.00%)     4875.00 (  -2.12%)
Max       alloc-odr4-32       5834.00 (   0.00%)     5950.00 (  -1.99%)
Max       alloc-odr4-64       6565.00 (   0.00%)     7434.00 ( -13.24%)
Max       alloc-odr4-128      6907.00 (   0.00%)     7034.00 (  -1.84%)
Max       alloc-odr4-256      7347.00 (   0.00%)     7843.00 (  -6.75%)
Max       alloc-odr4-512     10315.00 (   0.00%)    10866.00 (  -5.34%)
Max       alloc-odr4-1024    11278.00 (   0.00%)    11733.00 (  -4.03%)
Max       free-odr4-1          803.00 (   0.00%)     1009.00 ( -25.65%)
Max       free-odr4-2          495.00 (   0.00%)      607.00 ( -22.63%)
Max       free-odr4-4          354.00 (   0.00%)      417.00 ( -17.80%)
Max       free-odr4-8          275.00 (   0.00%)      313.00 ( -13.82%)
Max       free-odr4-16         273.00 (   0.00%)      272.00 (   0.37%)
Max       free-odr4-32         309.00 (   0.00%)      324.00 (  -4.85%)
Max       free-odr4-64         340.00 (   0.00%)      335.00 (   1.47%)
Max       free-odr4-128        340.00 (   0.00%)      338.00 (   0.59%)
Max       free-odr4-256        338.00 (   0.00%)      346.00 (  -2.37%)
Max       free-odr4-512        364.00 (   0.00%)      359.00 (   1.37%)
Max       free-odr4-1024       329.00 (   0.00%)      359.00 (  -9.12%)

The main overhead is a slight increase in latency on the memory allocation and free paths due to additional per-migratetype counting, with theoretically minimal impact on overall performance.

This patch series is based on v6.18-rc7

Hongru Zhang (3):
mm/page_alloc: add per-migratetype counts to buddy allocator mm/vmstat: get fragmentation statistics from per-migragetype count
mm: optimize free_area_empty() check using per-migratetype counts

 include/linux/mmzone.h |  1 +
 mm/internal.h          |  2 +-
 mm/mm_init.c           |  1 +
 mm/page_alloc.c        |  9 ++++++++-
 mm/vmstat.c            | 30 +++++++-----------------------
 5 files changed, 18 insertions(+), 25 deletions(-)
#19 lore raw
Quoted 21 lines · Vlastimil Babka
Quoted 4 lines · Hongru Zhang

On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.

Hm /proc/buddyinfo could be enough to determine fragmentation? Also we have in-kernel proactive compaction these days.

Quoted 11 lines

Under high load scenarios, reading /proc/pagetypeinfo causes memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention

Quoted 3 lines

It could be argued that not capturing /proc/pagetypeinfo (often) would help. I wonder if we can find also other benefits from the counters in the kernel itself.

In earlier iterations of the huge allocator patches, I played around with using these for compaction_suitable():

https://lore.kernel.org/linux-mm/[email protected]/

ISTR it cut down compaction numbers, because it would avoid runs where free pages are mostly in unsuitable targets (free_unmovable). But this was also in a series that used compaction_suitable() to stop kswapd, which in hindsight was a mistake; it would need re-evaluating by itself.

I also found these counters useful to have in OOM/allocfail dumps to see if allocator packing or compaction could have done better.

#20 lore raw
In reply to Vlastimil Babka
Quoted 7 lines
Quoted 4 lines

On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.

Hm /proc/buddyinfo could be enough to determine fragmentation? Also we have in-kernel proactive compaction these days.

In fact, besides /proc/pagetypeinfo, other system resource information is also collected at appropriate times, and resource usage throughout the process lifecycle is appropriately tracked as well. User-space management components integrate this information together to make decisions and perform proper actions.

Quoted 12 lines
Quoted 11 lines

Under high load scenarios, reading /proc/pagetypeinfo causes memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention

Quoted 3 lines

It could be argued that not capturing /proc/pagetypeinfo (often) would help. I wonder if we can find also other benefits from the counters in the kernel itself.

Collecting system and app resource statistics and making decisions based on this information is a common practice among Android device manufacturers.

Currently, there should be over a billion Android phones being used daily worldwide. The diversity of hardware configurations across Android devices makes it difficult for kernel mechanisms alone to maintain good performance across all usage scenarios.

First, hardware capabilities vary greatly - flagship phones may have up to 24GB of memory, while low-end devices may have as little as 4GB. CPU, storage, battery, and passive cooling capabilities vary significantly due to market positioning and cost factors. Hardware resources seem always inadequate.

Second, usage scenarios also differ - some people use devices in hot environments while others in cold environments; some enjoy high-definition gaming while others simply browse the web.

Third, user habits vary as well. Some people rarely restart their phones except when the battery dies or the system crashes; others restart daily, like me. Some users never actively close apps, only switching them to the background, resulting in dozens of apps running in the background and keeping system resources consumed (especially memory). Yet others just use a few apps, closing unused apps rather than leaving them in the background.

Despite the above challenges, Android device manufacturers hope to ensure a good user experience (no UI jank) across all situations.

Even at 60 Hz frame refresh rate (90 Hz, 120 Hz also supported now), all work from user input to render and display should be done within 16.7 ms. To achieve this goal, the management components perform tasks such as:
- Track system resource status: what system has
(system resource awareness)
- Learn and predict app resource demands: what app needs (resource demand awareness)
- Monitor app launch, exit, and foreground-background switches: least important app gives back resource to system to serve most important one, usually the foreground app
(user intent awareness)

Tracking system resources seems necessary for Android devices, not optional. So the related paths are not that cold on Android devices.

All the above are from workload perspective. From the kernel perspective, regardless of when or how frequently user-space tools read statistical information, they should not affect the kernel's own efficiency significantly. That's why I submit this patch series to make the read side of /proc/pagetypeinfo lock-free. But this does introduce overhead in hot path, I would greatly appreciate if we can discuss how to improve it here.

Quoted 5 lines

Adding these migratetype counters is something that wouldn't be even possible in the past, until the freelist migratetype hygiene was merged. So now it should be AFAIK possible, but it's still some overhead in relatively hot paths. I wonder if we even considered this before in the context of migratetype hygiene? Couldn't find anything quickly.

Yes, I wrote the code on old kernel initially, at that time, I reused set_pcppage_migratetype (also renamed) to cache the exact migratetype list that the page block is on. After the freelist migratetype hygiene patches were merged, I removed that logic.

reviewer
#21 lore raw
Quoted 71 lines · Hongru Zhang
Quoted 7 lines
Quoted 4 lines

On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.

Hm /proc/buddyinfo could be enough to determine fragmentation? Also we have in-kernel proactive compaction these days.

In fact, besides /proc/pagetypeinfo, other system resource information is also collected at appropriate times, and resource usage throughout the process lifecycle is appropriately tracked as well. User-space management components integrate this information together to make decisions and perform proper actions.

Quoted 12 lines
Quoted 11 lines

Under high load scenarios, reading /proc/pagetypeinfo causes memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention

Quoted 3 lines

It could be argued that not capturing /proc/pagetypeinfo (often) would help. I wonder if we can find also other benefits from the counters in the kernel itself.

Collecting system and app resource statistics and making decisions based on this information is a common practice among Android device manufacturers.

Currently, there should be over a billion Android phones being used daily worldwide. The diversity of hardware configurations across Android devices makes it difficult for kernel mechanisms alone to maintain good performance across all usage scenarios.

First, hardware capabilities vary greatly - flagship phones may have up to 24GB of memory, while low-end devices may have as little as 4GB. CPU, storage, battery, and passive cooling capabilities vary significantly due to market positioning and cost factors. Hardware resources seem always inadequate.

Second, usage scenarios also differ - some people use devices in hot environments while others in cold environments; some enjoy high-definition gaming while others simply browse the web.

Third, user habits vary as well. Some people rarely restart their phones except when the battery dies or the system crashes; others restart daily, like me. Some users never actively close apps, only switching them to the background, resulting in dozens of apps running in the background and keeping system resources consumed (especially memory). Yet others just use a few apps, closing unused apps rather than leaving them in the background.

Despite the above challenges, Android device manufacturers hope to ensure a good user experience (no UI jank) across all situations.

Even at 60 Hz frame refresh rate (90 Hz, 120 Hz also supported now), all work from user input to render and display should be done within 16.7 ms. To achieve this goal, the management components perform tasks such as:
- Track system resource status: what system has
(system resource awareness)
- Learn and predict app resource demands: what app needs (resource demand awareness)
- Monitor app launch, exit, and foreground-background switches: least important app gives back resource to system to serve most important one, usually the foreground app
(user intent awareness)

Quoted 2 lines

Tracking system resources seems necessary for Android devices, not optional. So the related paths are not that cold on Android devices.

These are all good background information. But how does userspace monitor utilize pageblock migratetype information? Can you give a concrete example?

Something like when free_movable is low, background apps is killed to provide more free pages? Or is userspace monitor even trying to attribute different pageblock usage to each app by monitoring /proc/pagetypeinfo before and after an app launch?

Thanks.

Quoted 14 lines

All the above are from workload perspective. From the kernel perspective, regardless of when or how frequently user-space tools read statistical information, they should not affect the kernel's own efficiency significantly. That's why I submit this patch series to make the read side of /proc/pagetypeinfo lock-free. But this does introduce overhead in hot path, I would greatly appreciate if we can discuss how to improve it here.

Quoted 5 lines

Adding these migratetype counters is something that wouldn't be even possible in the past, until the freelist migratetype hygiene was merged. So now it should be AFAIK possible, but it's still some overhead in relatively hot paths. I wonder if we even considered this before in the context of migratetype hygiene? Couldn't find anything quickly.

Quoted 4 lines

Yes, I wrote the code on old kernel initially, at that time, I reused set_pcppage_migratetype (also renamed) to cache the exact migratetype list that the page block is on. After the freelist migratetype hygiene patches were merged, I removed that logic.

Best Regards,
Yan, Zi

#22 lore raw
Quoted 21 lines
Quoted 15 lines

Despite the above challenges, Android device manufacturers hope to ensure a good user experience (no UI jank) across all situations.

Even at 60 Hz frame refresh rate (90 Hz, 120 Hz also supported now), all work from user input to render and display should be done within 16.7 ms. To achieve this goal, the management components perform tasks such as:
- Track system resource status: what system has
(system resource awareness)
- Learn and predict app resource demands: what app needs (resource demand awareness)
- Monitor app launch, exit, and foreground-background switches: least important app gives back resource to system to serve most important one, usually the foreground app
(user intent awareness)

Quoted 2 lines

Tracking system resources seems necessary for Android devices, not optional. So the related paths are not that cold on Android devices.

These are all good background information. But how does userspace monitor utilize pageblock migratetype information? Can you give a concrete example?

Quoted 6 lines

Something like when free_movable is low, background apps is killed to provide more free pages? Or is userspace monitor even trying to attribute different pageblock usage to each app by monitoring /proc/pagetypeinfo before and after an app launch?

Thanks.

AOSP:
https://android.googlesource.com/platform/frameworks/base/+/refs/heads/main/core/java/com/android/internal/app/procstats/ProcessStats.java#:~:text=public%20void%20updateFragmentation()

We have proprietary algorithms, but they are confidential. I cannot describe them publicly, sorry about that.

reviewer
#23 lore raw
In reply to Hongru Zhang
Quoted 6 lines · Hongru Zhang

On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.

Quoted 11 lines

Under high load scenarios, reading /proc/pagetypeinfo causes memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention

Do you have an idea how long each seq_printf call takes?

Assuming seq_printf is costly, printing while holding zone->lock may be suboptimal. A further optimization might be:

mm/vmstat.c +58 -0
index 2370c6fb1fcd..f501ca2840a6 100644
@@ -1570,6 +1570,7 @@ static int frag_show(struct seq_file *m, void *arg)
 	return 0;
 }
 
+#if 0
 static void pagetypeinfo_showfree_print(struct seq_file *m,
 					pg_data_t *pgdat, struct zone *zone)
 {
@@ -1611,6 +1612,63 @@ static void pagetypeinfo_showfree_print(struct seq_file *m,
 		seq_putc(m, '\n');
 	}
 }
+#endif
+
+static void pagetypeinfo_showfree_print(struct seq_file *m,
+					pg_data_t *pgdat,
+					struct zone *zone)
+{
+	unsigned long freecounts[MIGRATE_TYPES][NR_PAGE_ORDERS] = { 0 };
+	bool overflow[MIGRATE_TYPES][NR_PAGE_ORDERS] = { 0 };
+	int order, mtype;
+
+	for (mtype = 0; mtype < MIGRATE_TYPES; mtype++) {
+		for (order = 0; order < NR_PAGE_ORDERS; ++order) {
+			struct free_area *area;
+			struct list_head *curr;
+			unsigned long freecount = 0;
+
+			area = &zone->free_area[order];
+
+			list_for_each(curr, &area->free_list[mtype]) {
+				/*
+				 * Cap the free_list iteration because it might
+				 * be really large and we are under a spinlock
+				 * so a long time spent here could trigger a
+				 * hard lockup detector. Anyway this is a
+				 * debugging tool so knowing there is a handful
+				 * of pages of this order should be more than
+				 * sufficient.
+				 */
+				if (++freecount >= 100000) {
+					overflow[mtype][order] = true;
+					break;
+				}
+			}
+			freecounts[mtype][order] = freecount;
+			spin_unlock_irq(&zone->lock);
+			cond_resched();
+			spin_lock_irq(&zone->lock);
+		}
+	}
+
+	/* printing completely outside the lock */
+	spin_unlock_irq(&zone->lock);
+	for (mtype = 0; mtype < MIGRATE_TYPES; mtype++) {
+		seq_printf(m, "Node %4d, zone %8s, type %12s ",
+			   pgdat->node_id,
+			   zone->name,
+			   migratetype_names[mtype]);
+
+		for (order = 0; order < NR_PAGE_ORDERS; ++order) {
+			seq_printf(m, "%s%6lu ",
+				   overflow[mtype][order] ? ">" : "",
+				   freecounts[mtype][order]);
+		}
+		seq_putc(m, '\n');
+	}
+	spin_lock_irq(&zone->lock);
+}
 
 /* Print out the free pages at each order for each migratetype */
 static void pagetypeinfo_showfree(struct seq_file *m, void *arg)

Thanks
Barry

#24 lore raw
Quoted 90 lines · Hongru Zhang
Quoted 6 lines

On mobile devices, some user-space memory management components check memory pressure and fragmentation status periodically or via PSI, and take actions such as killing processes or performing memory compaction based on this information.

Quoted 11 lines

Under high load scenarios, reading /proc/pagetypeinfo causes memory management components or memory allocation/free paths to be blocked for extended periods waiting for the zone lock, leading to the following issues:
1. Long interrupt-disabled spinlocks - occasionally exceeding 10ms on Qcom 8750 platforms, reducing system real-time performance
2. Memory management components being blocked for extended periods, preventing rapid acquisition of memory fragmentation information for critical memory management decisions and actions
3. Increased latency in memory allocation and free paths due to prolonged zone lock contention

Do you have an idea how long each seq_printf call takes?

Assuming seq_printf is costly, printing while holding zone->lock may be suboptimal. A further optimization might be:

mm/vmstat.c +51 -0
index 2370c6fb1fcd..f501ca2840a6 100644
@@ -1570,6 +1570,7 @@ static int frag_show(struct seq_file *m, void *arg)
 	return 0;
 }
 
+#if 0
 static void pagetypeinfo_showfree_print(struct seq_file *m,
 					pg_data_t *pgdat, struct zone *zone)
 {
@@ -1611,6 +1612,63 @@ static void pagetypeinfo_showfree_print(struct seq_file *m,
 		seq_putc(m, '\n');
 	}
 }
+#endif
+
+static void pagetypeinfo_showfree_print(struct seq_file *m,
+					pg_data_t *pgdat,
+					struct zone *zone)
+{
+	unsigned long freecounts[MIGRATE_TYPES][NR_PAGE_ORDERS] = { 0 };
+	bool overflow[MIGRATE_TYPES][NR_PAGE_ORDERS] = { 0 };
+	int order, mtype;
+
+	for (mtype = 0; mtype < MIGRATE_TYPES; mtype++) {
+		for (order = 0; order < NR_PAGE_ORDERS; ++order) {
+			struct free_area *area;
+			struct list_head *curr;
+			unsigned long freecount = 0;
+
+			area = &zone->free_area[order];
+
+			list_for_each(curr, &area->free_list[mtype]) {
+				/*
+				 * Cap the free_list iteration because it might
+				 * be really large and we are under a spinlock
+				 * so a long time spent here could trigger a
+				 * hard lockup detector. Anyway this is a
+				 * debugging tool so knowing there is a handful
+				 * of pages of this order should be more than
+				 * sufficient.
+				 */
+				if (++freecount >= 100000) {
+					overflow[mtype][order] = true;
+					break;
+				}
+			}
+			freecounts[mtype][order] = freecount;
+			spin_unlock_irq(&zone->lock);
+			cond_resched();
+			spin_lock_irq(&zone->lock);
+		}
+	}
+
+	/* printing completely outside the lock */
+	spin_unlock_irq(&zone->lock);
+	for (mtype = 0; mtype < MIGRATE_TYPES; mtype++) {
+		seq_printf(m, "Node %4d, zone %8s, type %12s ",
+			   pgdat->node_id,
+			   zone->name,
+			   migratetype_names[mtype]);
+
+		for (order = 0; order < NR_PAGE_ORDERS; ++order) {
+			seq_printf(m, "%s%6lu ",

mm/vmstat.c static void pagetypeinfo_showfree_print(struct seq_file *m, · line 1674

Quoted 10 lines
mm/vmstat.c +7 -0
+				   overflow[mtype][order] ? ">" : "",
+				   freecounts[mtype][order]);
+		}
+		seq_putc(m, '\n');
+	}
+	spin_lock_irq(&zone->lock);
+}
 
 /* Print out the free pages at each order for each migratetype */
 static void pagetypeinfo_showfree(struct seq_file *m, void *arg)

Yes. I included the data in [1]. This seq_printf loop costs about 5 us on my machine. I'll look into optimizing this in the next revision.

[1] https://lore.kernel.org/linux-mm/[email protected]/