mm/page_alloc: keep track of free highatomic
Subsystems (4)
PATCH v2 1/1 mm/page_alloc: keep track of free highatomic
OOM kills due to vastly overestimated free highatomic reserves were observed:
... invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0 ...
Node 0 Normal free:1482936kB boost:0kB min:410416kB low:739404kB high:1068392kB reserved_highatomic:1073152KB ...
Node 0 Normal: 1292*4kB (ME) 1920*8kB (E) 383*16kB (UE) 220*32kB (ME) 340*64kB (E) 2155*128kB (UE) 3243*256kB (UE) 615*512kB (U) 1*1024kB (M) 0*2048kB 0*4096kB = 1477408kB
The second line above shows that the OOM kill was due to the following condition:
free (1482936kB) - reserved_highatomic (1073152kB) = 409784KB < min (410416kB)
And the third line shows there were no free pages in any MIGRATE_HIGHATOMIC pageblocks, which otherwise would show up as type 'H'. Therefore __zone_watermark_unusable_free() underestimated the usable free memory by over 1GB, which resulted in the unnecessary OOM kill above.
The comments in __zone_watermark_unusable_free() warns about the potential risk, i.e.,
If the caller does not have rights to reserves below the min watermark then subtract the high-atomic reserves. This will over-estimate the size of the atomic reserve but it avoids a search.
However, it is possible to keep track of free pages in reserved highatomic pageblocks with a new per-zone counter nr_free_highatomic protected by the zone lock, to avoid a search when calculating the usable free memory. And the cost would be minimal, i.e., simple arithmetics in the highatomic alloc/free/move paths.
--- include/linux/mmzone.h | 1 + mm/page_alloc.c | 23 ++++++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-)
include/linux/mmzone.h +1 -0
@@ -825,6 +825,7 @@ struct zone {unsigned long watermark_boost;unsigned long nr_reserved_highatomic; +unsigned long nr_free_highatomic;/* * We don't know if the memory that we're going to allocate will be
mm/page_alloc.c +20 -3
@@ -644,6 +644,18 @@ static inline void account_freepages(struct zone *zone, int nr_pages,__mod_zone_page_state (zone,NR_FREE_CMA_PAGES , nr_pages); } +static void account_highatomic_freepages (struct zone *zone,unsigned int order, +int old_mt,int new_mt) +{ +int delta =1 << order; + +if (is_migrate_highatomic (old_mt)) +WRITE_ONCE (zone->nr_free_highatomic, zone->nr_free_highatomic - delta); + +if (is_migrate_highatomic (new_mt)) +WRITE_ONCE (zone->nr_free_highatomic, zone->nr_free_highatomic + delta); +} +/* Used for pages not on another list */ static inline void __add_to_free_list(struct page *page, struct zone *zone, unsigned int order, int migratetype,
@@ -660,6 +672,8 @@ static inline void __add_to_free_list(struct page *page, struct zone *zone,else list_add (&page->buddy_list, &area->free_list[migratetype]); area->nr_free++; + +account_highatomic_freepages (zone, order,-1 , migratetype); }/*
@@ -681,6 +695,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone,account_freepages (zone, -(1 << order), old_mt);account_freepages (zone,1 << order, new_mt); + +account_highatomic_freepages (zone, order, old_mt, new_mt); } static inline void __del_page_from_free_list(struct page *page, struct zone *zone,
@@ -698,6 +714,8 @@ static inline void __del_page_from_free_list(struct page *page, struct zone *zon__ClearPageBuddy (page);set_page_private (page,0 ); zone->free_area[order].nr_free--; + +account_highatomic_freepages (zone, order, migratetype,-1 ); } static inline void del_page_from_free_list(struct page *page, struct zone *zone,
@@ -3119,11 +3137,10 @@ static inline long __zone_watermark_unusable_free(struct zone *z,/* * If the caller does not have rights to reserves below the min -* watermark then subtract the high-atomic reserves. This will -* over-estimate the size of the atomic reserve but it avoids a search. +* watermark then subtract the free pages reserved for highatomic. */ if (likely (!(alloc_flags &ALLOC_RESERVES ))) - unusable_free += z->nr_reserved_highatomic; + unusable_free +=READ_ONCE (z->nr_free_highatomic);#ifdef CONFIG_CMA /* If allocation can't use CMA areas don't use free CMA pages */
Signature · 1 line
2.47.0.163.g1226f6d8fa-goog
Andrew Morton
maintainerQuoted 6 lines · On Fri, 25 Oct 2024 21:36:25 -0600 Yu Zhao <[email protected]> wrote:
OOM kills due to vastly overestimated free highatomic reserves were observed:
... invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0 ...
Node 0 Normal free:1482936kB boost:0kB min:410416kB low:739404kB high:1068392kB reserved_highatomic:1073152KB ...
Node 0 Normal: 1292*4kB (ME) 1920*8kB (E) 383*16kB (UE) 220*32kB (ME) 340*64kB (E) 2155*128kB (UE) 3243*256kB (UE) 615*512kB (U) 1*1024kB (M) 0*2048kB 0*4096kB = 1477408kB
Under what circumstances?
Quoted 18 lines
The second line above shows that the OOM kill was due to the following condition:
free (1482936kB) -
reserved_highatomic(1073152kB) = 409784KB < min (410416kB)And the third line shows there were no free pages in any
MIGRATE_HIGHATOMICpageblocks, which otherwise would show up as type 'H'. Therefore__zone_watermark_unusable_free()underestimated the usable free memory by over 1GB, which resulted in the unnecessary OOM kill above.The comments in
__zone_watermark_unusable_free()warns about the potential risk, i.e.,If the caller does not have rights to reserves below the min watermark then subtract the high-atomic reserves. This will over-estimate the size of the atomic reserve but it avoids a search.
Quoted 5 lines
However, it is possible to keep track of free pages in reserved highatomic pageblocks with a new per-zone counter
nr_free_highatomicprotected by the zone lock, to avoid a search when calculating the usable free memory. And the cost would be minimal, i.e., simple arithmetics in the highatomic alloc/free/move paths.
Is a -stable backport needed?
If so, is a Fixes: target identifiable?
Quoted 11 lines · Andrew Morton
Quoted 6 lines · On Fri, 25 Oct 2024 21:36:25 -0600 Yu Zhao <[email protected]> wrote:
OOM kills due to vastly overestimated free highatomic reserves were observed:
... invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0 ...
Node 0 Normal free:1482936kB boost:0kB min:410416kB low:739404kB high:1068392kB reserved_highatomic:1073152KB ...
Node 0 Normal: 1292*4kB (ME) 1920*8kB (E) 383*16kB (UE) 220*32kB (ME) 340*64kB (E) 2155*128kB (UE) 3243*256kB (UE) 615*512kB (U) 1*1024kB (M) 0*2048kB 0*4096kB = 1477408kBUnder what circumstances?
Link wrote a repro, so he'd be the best person to answer this question.
Link, please help. Thanks.
Quoted 24 lines
Quoted 18 lines
The second line above shows that the OOM kill was due to the following condition:
free (1482936kB) -
reserved_highatomic(1073152kB) = 409784KB < min (410416kB)And the third line shows there were no free pages in any
MIGRATE_HIGHATOMICpageblocks, which otherwise would show up as type 'H'. Therefore__zone_watermark_unusable_free()underestimated the usable free memory by over 1GB, which resulted in the unnecessary OOM kill above.The comments in
__zone_watermark_unusable_free()warns about the potential risk, i.e.,If the caller does not have rights to reserves below the min watermark then subtract the high-atomic reserves. This will over-estimate the size of the atomic reserve but it avoids a search.
Quoted 5 lines
However, it is possible to keep track of free pages in reserved highatomic pageblocks with a new per-zone counter
nr_free_highatomicprotected by the zone lock, to avoid a search when calculating the usable free memory. And the cost would be minimal, i.e., simple arithmetics in the highatomic alloc/free/move paths.
Quoted 3 lines
Is a -stable backport needed?
If so, is a Fixes: target identifiable?
The code has been there for many years, and we only recently noticed the problem from Link's repro. So it doesn't look like a stable material.
Quoted 10 lines · Yu Zhao
Quoted 4 lines · Andrew Morton
Is a -stable backport needed?
If so, is a Fixes: target identifiable?
The code has been there for many years, and we only recently noticed the problem from Link's repro. So it doesn't look like a stable material.
The stable backportability would be limited as there's a prerequisity on the "mm: page_alloc: freelist migratetype hygiene" patchset from half a year ago
- patch applicability wise and importantly functionality wise. Otherwise the counter could drift easily.
However we could perhaps mark it for a 6.12 stable backport as that's presumably the upcoming LTS.
Quoted 9 lines · Vlastimil Babka
Quoted 7 lines · Yu Zhao
Quoted 4 lines · Andrew Morton
Is a -stable backport needed?
If so, is a Fixes: target identifiable?
Quoted 7 lines
Quoted 3 lines
The code has been there for many years, and we only recently noticed the problem from Link's repro. So it doesn't look like a stable material.
The stable backportability would be limited as there's a prerequisity on the "mm:
page_alloc: freelist migratetype hygiene" patchset from half a year ago
- patch applicability wise and importantly functionality wise.
I don't see any dependency to that series.
Quoted 2 lines
Otherwise the
counter could drift easily.
Care to elaborate how that could happen?
Quoted 2 lines
However we could perhaps mark it for a 6.12 stable backport as that's presumably the upcoming LTS.
Right.
Quoted 25 lines · Yu Zhao
OOM kills due to vastly overestimated free highatomic reserves were observed:
... invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0 ...
Node 0 Normal free:1482936kB boost:0kB min:410416kB low:739404kB high:1068392kB reserved_highatomic:1073152KB ...
Node 0 Normal: 1292*4kB (ME) 1920*8kB (E) 383*16kB (UE) 220*32kB (ME) 340*64kB (E) 2155*128kB (UE) 3243*256kB (UE) 615*512kB (U) 1*1024kB (M) 0*2048kB 0*4096kB = 1477408kBThe second line above shows that the OOM kill was due to the following condition:
free (1482936kB) -
reserved_highatomic(1073152kB) = 409784KB < min (410416kB)And the third line shows there were no free pages in any
MIGRATE_HIGHATOMICpageblocks, which otherwise would show up as type 'H'. Therefore__zone_watermark_unusable_free()underestimated the usable free memory by over 1GB, which resulted in the unnecessary OOM kill above.The comments in
__zone_watermark_unusable_free()warns about the potential risk, i.e.,If the caller does not have rights to reserves below the min watermark then subtract the high-atomic reserves. This will over-estimate the size of the atomic reserve but it avoids a search.
Quoted 8 lines
However, it is possible to keep track of free pages in reserved highatomic pageblocks with a new per-zone counter
nr_free_highatomicprotected by the zone lock, to avoid a search when calculating the usable free memory. And the cost would be minimal, i.e., simple arithmetics in the highatomic alloc/free/move paths.Reported-by Link Lin <[email protected]>Signed-off-by Yu Zhao <[email protected]>
Quoted 21 lines · Yu Zhao
OOM kills due to vastly overestimated free highatomic reserves were observed:
... invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0 ...
Node 0 Normal free:1482936kB boost:0kB min:410416kB low:739404kB high:1068392kB reserved_highatomic:1073152KB ...
Node 0 Normal: 1292*4kB (ME) 1920*8kB (E) 383*16kB (UE) 220*32kB (ME) 340*64kB (E) 2155*128kB (UE) 3243*256kB (UE) 615*512kB (U) 1*1024kB (M) 0*2048kB 0*4096kB = 1477408kBThe second line above shows that the OOM kill was due to the following condition:
free (1482936kB) -
reserved_highatomic(1073152kB) = 409784KB < min (410416kB)And the third line shows there were no free pages in any
MIGRATE_HIGHATOMICpageblocks, which otherwise would show up as type 'H'. Therefore__zone_watermark_unusable_free()underestimated the usable free memory by over 1GB, which resulted in the unnecessary OOM kill above.The comments in
__zone_watermark_unusable_free()warns about the potential risk, i.e.,
Quoted 7 lines
If the caller does not have rights to reserves below the min watermark then subtract the high-atomic reserves. This will over-estimate the size of the atomic reserve but it avoids a search.
However, it is possible to keep track of free pages in reserved highatomic pageblocks with a new per-zone counter
nr_free_highatomicprotected by the zone lock, to avoid a search when calculating the
It's only possible to track this reliably since the "mm: page_alloc:
freelist migratetype hygiene" patchset was merged, which explains why nr_reserved_highatomic was used until now, even if it's imprecise. But I agree it should be a good solution to the problem now.
Quoted 5 lines
usable free memory. And the cost would be minimal, i.e., simple arithmetics in the highatomic alloc/free/move paths.
Reported-by Link Lin <[email protected]>Signed-off-by Yu Zhao <[email protected]>
However, implementation wise that patchset also centralized everything to account_freepages() which currently handles NR_FREE_PAGES and NR_FREE_CMA_PAGES, and your patch differs from that approach. Can't you simply add a NR_FREE_HIGHATOMIC update there instead of creating account_highatomic_freepages() which is hooked to 3 different places? And for consistency indeed add a zone_stat_item counter instead a struct zone field? Then it becomes visible in e.g. /proc/zoneinfo and /proc/vmstat too. Bonus points for printing it in the oom/alloc failure reports as well (wherever we print NR_FREE_CMA_PAGES).
Thanks!
Vlastimil
Quoted 81 lines
--- include/linux/mmzone.h | 1 + mm/page_alloc.c | 23 ++++++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-)include/linux/mmzone.h +1
-0@@ -825,6 +825,7 @@ struct zone {unsigned long watermark_boost;unsigned long nr_reserved_highatomic; +unsigned long nr_free_highatomic;/* * We don't know if the memory that we're going to allocate will be mm/page_alloc.c +20
-3@@ -644,6 +644,18 @@ static inline void account_freepages(struct zone *zone, int nr_pages,__mod_zone_page_state (zone,NR_FREE_CMA_PAGES , nr_pages); } +static void account_highatomic_freepages (struct zone *zone,unsigned int order, +int old_mt,int new_mt) +{ +int delta =1 << order; + +if (is_migrate_highatomic (old_mt)) +WRITE_ONCE (zone->nr_free_highatomic, zone->nr_free_highatomic - delta); + +if (is_migrate_highatomic (new_mt)) +WRITE_ONCE (zone->nr_free_highatomic, zone->nr_free_highatomic + delta); +} +/* Used for pages not on another list */ static inline void __add_to_free_list(struct page *page, struct zone *zone, unsigned int order, int migratetype,@@ -660,6 +672,8 @@ static inline void __add_to_free_list(struct page *page, struct zone *zone,else list_add (&page->buddy_list, &area->free_list[migratetype]); area->nr_free++; + +account_highatomic_freepages (zone, order,-1 , migratetype); }/* @@ -681,6 +695,8 @@ static inline void move_to_free_list(struct page *page, struct zone *zone,account_freepages (zone, -(1 << order), old_mt);account_freepages (zone,1 << order, new_mt); + +account_highatomic_freepages (zone, order, old_mt, new_mt); } static inline void __del_page_from_free_list(struct page *page, struct zone *zone,@@ -698,6 +714,8 @@ static inline void __del_page_from_free_list(struct page *page, struct zone *zon__ClearPageBuddy (page);set_page_private (page,0 ); zone->free_area[order].nr_free--; + +account_highatomic_freepages (zone, order, migratetype,-1 ); } static inline void del_page_from_free_list(struct page *page, struct zone *zone,@@ -3119,11 +3137,10 @@ static inline long __zone_watermark_unusable_free(struct zone *z,/* * If the caller does not have rights to reserves below the min -* watermark then subtract the high-atomic reserves. This will -* over-estimate the size of the atomic reserve but it avoids a search. +* watermark then subtract the free pages reserved for highatomic. */ if (likely (!(alloc_flags &ALLOC_RESERVES ))) - unusable_free += z->nr_reserved_highatomic; + unusable_free +=READ_ONCE (z->nr_free_highatomic);#ifdef CONFIG_CMA /* If allocation can't use CMA areas don't use free CMA pages */
Quoted 27 lines · Vlastimil Babka
Quoted 25 lines · Yu Zhao
OOM kills due to vastly overestimated free highatomic reserves were observed:
... invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0 ...
Node 0 Normal free:1482936kB boost:0kB min:410416kB low:739404kB high:1068392kB reserved_highatomic:1073152KB ...
Node 0 Normal: 1292*4kB (ME) 1920*8kB (E) 383*16kB (UE) 220*32kB (ME) 340*64kB (E) 2155*128kB (UE) 3243*256kB (UE) 615*512kB (U) 1*1024kB (M) 0*2048kB 0*4096kB = 1477408kBThe second line above shows that the OOM kill was due to the following condition:
free (1482936kB) -
reserved_highatomic(1073152kB) = 409784KB < min (410416kB)And the third line shows there were no free pages in any
MIGRATE_HIGHATOMICpageblocks, which otherwise would show up as type 'H'. Therefore__zone_watermark_unusable_free()underestimated the usable free memory by over 1GB, which resulted in the unnecessary OOM kill above.The comments in
__zone_watermark_unusable_free()warns about the potential risk, i.e.,If the caller does not have rights to reserves below the min watermark then subtract the high-atomic reserves. This will over-estimate the size of the atomic reserve but it avoids a search.
Quoted 7 lines
Quoted 3 lines
However, it is possible to keep track of free pages in reserved highatomic pageblocks with a new per-zone counter
nr_free_highatomicprotected by the zone lock, to avoid a search when calculating theIt's only possible to track this reliably since the "mm:
page_alloc:
freelist migratetype hygiene" patchset was merged, which explains whynr_reserved_highatomicwas used until now, even if it's imprecise.
I just refreshed my memory by quickly going through the discussion around that series and didn't find anything that helps me understand the above. More pointers please?
Quoted 32 lines · Yu Zhao
Quoted 31 lines · Vlastimil Babka
Quoted 28 lines · Yu Zhao
OOM kills due to vastly overestimated free highatomic reserves were observed:
... invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0 ...
Node 0 Normal free:1482936kB boost:0kB min:410416kB low:739404kB high:1068392kB reserved_highatomic:1073152KB ...
Node 0 Normal: 1292*4kB (ME) 1920*8kB (E) 383*16kB (UE) 220*32kB (ME) 340*64kB (E) 2155*128kB (UE) 3243*256kB (UE) 615*512kB (U) 1*1024kB (M) 0*2048kB 0*4096kB = 1477408kBThe second line above shows that the OOM kill was due to the following condition:
free (1482936kB) -
reserved_highatomic(1073152kB) = 409784KB < min (410416kB)And the third line shows there were no free pages in any
MIGRATE_HIGHATOMICpageblocks, which otherwise would show up as type 'H'. Therefore__zone_watermark_unusable_free()underestimated the usable free memory by over 1GB, which resulted in the unnecessary OOM kill above.The comments in
__zone_watermark_unusable_free()warns about the potential risk, i.e.,If the caller does not have rights to reserves below the min watermark then subtract the high-atomic reserves. This will over-estimate the size of the atomic reserve but it avoids a search.
However, it is possible to keep track of free pages in reserved highatomic pageblocks with a new per-zone counter
nr_free_highatomicprotected by the zone lock, to avoid a search when calculating the
Quoted 7 lines
Quoted 3 lines
It's only possible to track this reliably since the "mm:
page_alloc:
freelist migratetype hygiene" patchset was merged, which explains whynr_reserved_highatomicwas used until now, even if it's imprecise.I just refreshed my memory by quickly going through the discussion around that series and didn't find anything that helps me understand the above. More pointers please?
For example:
- a page is on pcplist in MIGRATE_MOVABLE list
- we reserve its pageblock as highatomic, which does nothing to the page on the pcplist
- page above is flushed from pcplist to zone freelist, but it remembers it was MIGRATE_MOVABLE, merges with another buddy/buddies from the now-highatomic list, the resulting order-X page ends up on the movable freelist despite being in highatomic pageblock. The counter of free highatomic is now wrong wrt the freelist reality
The series has addressed various scenarios like that, where page can end up on the wrong freelist.
Quoted 44 lines · Vlastimil Babka
Quoted 32 lines · Yu Zhao
Quoted 31 lines · Vlastimil Babka
Quoted 28 lines · Yu Zhao
OOM kills due to vastly overestimated free highatomic reserves were observed:
... invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0 ...
Node 0 Normal free:1482936kB boost:0kB min:410416kB low:739404kB high:1068392kB reserved_highatomic:1073152KB ...
Node 0 Normal: 1292*4kB (ME) 1920*8kB (E) 383*16kB (UE) 220*32kB (ME) 340*64kB (E) 2155*128kB (UE) 3243*256kB (UE) 615*512kB (U) 1*1024kB (M) 0*2048kB 0*4096kB = 1477408kBThe second line above shows that the OOM kill was due to the following condition:
free (1482936kB) -
reserved_highatomic(1073152kB) = 409784KB < min (410416kB)And the third line shows there were no free pages in any
MIGRATE_HIGHATOMICpageblocks, which otherwise would show up as type 'H'. Therefore__zone_watermark_unusable_free()underestimated the usable free memory by over 1GB, which resulted in the unnecessary OOM kill above.The comments in
__zone_watermark_unusable_free()warns about the potential risk, i.e.,If the caller does not have rights to reserves below the min watermark then subtract the high-atomic reserves. This will over-estimate the size of the atomic reserve but it avoids a search.
However, it is possible to keep track of free pages in reserved highatomic pageblocks with a new per-zone counter
nr_free_highatomicprotected by the zone lock, to avoid a search when calculating theQuoted 7 lines
Quoted 3 lines
It's only possible to track this reliably since the "mm:
page_alloc:
freelist migratetype hygiene" patchset was merged, which explains whynr_reserved_highatomicwas used until now, even if it's imprecise.I just refreshed my memory by quickly going through the discussion around that series and didn't find anything that helps me understand the above. More pointers please?
For example:
Quoted 8 lines
- a page is on pcplist in
MIGRATE_MOVABLElist
- we reserve its pageblock as highatomic, which does nothing to the page on the pcplist
- page above is flushed from pcplist to zone freelist, but it remembers it wasMIGRATE_MOVABLE, merges with another buddy/buddies from the now-highatomic list, the resulting order-X page ends up on the movable freelist despite being in highatomic pageblock. The counter of free highatomic is now wrong wrt the freelist reality
This is the part I don't follow: how is it wrong w.r.t. the freelist reality? The new nr_free_highatomic should reflect how many pages are exactly on free_list[MIGRATE_HIGHATOMIC], because it's updated accordingly.
(My current understanding is that, in this case, the reservation itself is messed up, i.e., under-reserved.)
Quoted 2 lines
The series has addressed various scenarios like that, where page can end up on the wrong freelist.
Quoted 17 lines · Yu Zhao
Quoted 7 lines · Vlastimil Babka
Quoted 2 lines · Yu Zhao
Quoted 1 line · Vlastimil Babka
For example:
Quoted 8 lines
- a page is on pcplist in
MIGRATE_MOVABLElist
- we reserve its pageblock as highatomic, which does nothing to the page on the pcplist
- page above is flushed from pcplist to zone freelist, but it remembers it wasMIGRATE_MOVABLE, merges with another buddy/buddies from the now-highatomic list, the resulting order-X page ends up on the movable freelist despite being in highatomic pageblock. The counter of free highatomic is now wrong wrt the freelist reality
Quoted 4 lines
This is the part I don't follow: how is it wrong w.r.t. the freelist reality? The new
nr_free_highatomicshould reflect how many pages are exactly on free_list[MIGRATE_HIGHATOMIC], because it's updated accordingly.
You'd have to try implementing your change in the kernel without that migratetype hygiene series, and see how it would either not work, or you'd end up implementing the series as part of that.
Quoted 5 lines
(My current understanding is that, in this case, the reservation itself is messed up, i.e., under-reserved.)
Quoted 2 lines
The series has addressed various scenarios like that, where page can end up on the wrong freelist.
Quoted 19 lines · Vlastimil Babka
Quoted 17 lines · Yu Zhao
Quoted 15 lines · Vlastimil Babka
Quoted 2 lines · Yu Zhao
Quoted 1 line · Vlastimil Babka
For example:
- a page is on pcplist in
MIGRATE_MOVABLElist
- we reserve its pageblock as highatomic, which does nothing to the page on the pcplist
- page above is flushed from pcplist to zone freelist, but it remembers it wasMIGRATE_MOVABLE, merges with another buddy/buddies from the now-highatomic list, the resulting order-X page ends up on the movable freelist despite being in highatomic pageblock. The counter of free highatomic is now wrong wrt the freelist reality
Quoted 8 lines
Quoted 4 lines
This is the part I don't follow: how is it wrong w.r.t. the freelist reality? The new
nr_free_highatomicshould reflect how many pages are exactly on free_list[MIGRATE_HIGHATOMIC], because it's updated accordingly.You'd have to try implementing your change in the kernel without that migratetype hygiene series, and see how it would either not work, or you'd end up implementing the series as part of that.
A proper backport would need to track the MT of the free_list a page is deleted from, and I see no reason why in such a proper backport "the counter could drift easily" or "the counter of free highatomic is now wrong wrt the freelist reality". So I assume you actually mean this patch can't be backported cleanly? (Which I do agree.)
Quoted 29 lines · Yu Zhao
Quoted 19 lines · Vlastimil Babka
Quoted 17 lines · Yu Zhao
Quoted 15 lines · Vlastimil Babka
Quoted 2 lines · Yu Zhao
Quoted 1 line · Vlastimil Babka
For example:
- a page is on pcplist in
MIGRATE_MOVABLElist
- we reserve its pageblock as highatomic, which does nothing to the page on the pcplist
- page above is flushed from pcplist to zone freelist, but it remembers it wasMIGRATE_MOVABLE, merges with another buddy/buddies from the now-highatomic list, the resulting order-X page ends up on the movable freelist despite being in highatomic pageblock. The counter of free highatomic is now wrong wrt the freelist realityQuoted 8 lines
Quoted 4 lines
This is the part I don't follow: how is it wrong w.r.t. the freelist reality? The new
nr_free_highatomicshould reflect how many pages are exactly on free_list[MIGRATE_HIGHATOMIC], because it's updated accordingly.You'd have to try implementing your change in the kernel without that migratetype hygiene series, and see how it would either not work, or you'd end up implementing the series as part of that.
Quoted 5 lines
A proper backport would need to track the MT of the
free_lista page is deleted from, and I see no reason why in such a proper backport "the counter could drift easily" or "the counter of free highatomic is now wrong wrt the freelist reality". So I assume you actually mean this patch can't be backported cleanly? (Which I do agree.)
Yes you're right. But since we don't plan to backport it beyond 6.12, sorry for sidetracking the discussion unnecessarily. More importantly, is it possible to change the implementation as I suggested? [1] Hooking to __del_page_from_free_list() and __add_to_free_list() means extra work in every loop iteration in expand() and __free_one_page(). The migratetype hygiene should ensure it's not necessary to intercept every freelist add/move and hooking to account_freepages() should be sufficient and in line with the intended design.
Thanks,
Vlastimil
Quoted 37 lines · Vlastimil Babka
Quoted 29 lines · Yu Zhao
Quoted 19 lines · Vlastimil Babka
Quoted 17 lines · Yu Zhao
Quoted 15 lines · Vlastimil Babka
Quoted 2 lines · Yu Zhao
Quoted 1 line · Vlastimil Babka
For example:
- a page is on pcplist in
MIGRATE_MOVABLElist
- we reserve its pageblock as highatomic, which does nothing to the page on the pcplist
- page above is flushed from pcplist to zone freelist, but it remembers it wasMIGRATE_MOVABLE, merges with another buddy/buddies from the now-highatomic list, the resulting order-X page ends up on the movable freelist despite being in highatomic pageblock. The counter of free highatomic is now wrong wrt the freelist realityQuoted 8 lines
Quoted 4 lines
This is the part I don't follow: how is it wrong w.r.t. the freelist reality? The new
nr_free_highatomicshould reflect how many pages are exactly on free_list[MIGRATE_HIGHATOMIC], because it's updated accordingly.You'd have to try implementing your change in the kernel without that migratetype hygiene series, and see how it would either not work, or you'd end up implementing the series as part of that.
Quoted 5 lines
A proper backport would need to track the MT of the
free_lista page is deleted from, and I see no reason why in such a proper backport "the counter could drift easily" or "the counter of free highatomic is now wrong wrt the freelist reality". So I assume you actually mean this patch can't be backported cleanly? (Which I do agree.)
Quoted 3 lines
Yes you're right. But since we don't plan to backport it beyond 6.12, sorry for sidetracking the discussion unnecessarily. More importantly, is it possible to change the implementation as I suggested?
The only reason I didn't fold account_highatomic_freepages() into account_freepages() is because the former must be called under the zone lock, which is also how the latter is called but not as a requirement.
I understand where you come from when suggesting a new per-cpu counter for free highatomic. I have to disagree with that because 1) free highatomic is relatively small and drifting might defeat its purpose;
2) per-cpu memory is among the top kernel memory overhead in our fleet -- it really adds up. So I prefer not to use per-cpu counters unless necessary.
So if it's ok with you, I'll just fold account_highatomic_freepages() into account_freepages(), but keep the counter as per zone, not per cpu.
Quoted 6 lines
[1] Hooking
to__del_page_from_free_list()and__add_to_free_list()means extra work in every loop iteration inexpand()and__free_one_page(). The migratetype hygiene should ensure it's not necessary to intercept every freelist add/move and hooking toaccount_freepages()should be sufficient and in line with the intended design.
Agreed.
Quoted 10 lines · Yu Zhao
Quoted 4 lines · Vlastimil Babka
Yes you're right. But since we don't plan to backport it beyond 6.12, sorry for sidetracking the discussion unnecessarily. More importantly, is it possible to change the implementation as I suggested?
The only reason I didn't fold
account_highatomic_freepages()intoaccount_freepages()is because the former must be called under the zone lock, which is also how the latter is called but not as a requirement.
Ah, I guess we can document the requirement/add an lockdep assert. Using __mod_zone_page_state() already implies some context restrictions although not zone lock specifically.
Quoted 6 lines
I understand where you come from when suggesting a new per-cpu counter for free highatomic. I have to disagree with that because 1) free highatomic is relatively small and drifting might defeat its purpose;
2) per-cpu memory is among the top kernel memory overhead in our fleet -- it really adds up. So I prefer not to use per-cpu counters unless necessary.
OK, didn't think of these drawbacks.
Quoted 3 lines
So if it's ok with you, I'll just fold
account_highatomic_freepages()intoaccount_freepages(), but keep the counter as per zone, not per cpu.
OK, thanks!
Quoted 8 lines
Quoted 6 lines
[1] Hooking
to__del_page_from_free_list()and__add_to_free_list()means extra work in every loop iteration inexpand()and__free_one_page(). The migratetype hygiene should ensure it's not necessary to intercept every freelist add/move and hooking toaccount_freepages()should be sufficient and in line with the intended design.Agreed.