mm: count time in drain_all_pages during direct reclaim as memory pressure
Subsystems (2)
Suren Baghdasaryan
reviewerPATCH v3 1/1 mm: count time in drain_all_pages during direct reclaim as memory pressure
When page allocation in direct reclaim path fails, the system will make one attempt to shrink per-cpu page lists and free pages from high alloc reserves. Draining per-cpu pages into buddy allocator can be a very slow operation because it's done using workqueues and the task in direct reclaim waits for all of them to finish before proceeding. Currently this time is not accounted as psi memory stall.
While testing mobile devices under extreme memory pressure, when allocations are failing during direct reclaim, we notices that psi events which would be expected in such conditions were not triggered. After profiling these cases it was determined that the reason for missing psi events was that a big chunk of time spent in direct reclaim is not accounted as memory stall, therefore psi would not reach the levels at which an event is generated. Further investigation revealed that the bulk of that unaccounted time was spent inside drain_all_pages call.
A typical captured case when drain_all_pages path gets activated:
__alloc_pages_slowpath took 44.644.613ns
__perform_reclaim took 751.668ns (1.7%)
drain_all_pages took 43.887.167ns (98.3%)PSI in this case records the time spent in __perform_reclaim but ignores drain_all_pages, IOW it misses 98.3% of the time spent in __alloc_pages_slowpath.
Annotate __alloc_pages_direct_reclaim in its entirety so that delays from handling page allocation failure in the direct reclaim path are accounted as memory stall.
---
changes in v3:
- Moved psi_memstall_leave after the "out" label
mm/page_alloc.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
mm/page_alloc.c +6 -4
@@ -4595,13 +4595,12 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order, const struct alloc_context *ac) {unsigned int noreclaim_flag; -unsigned long pflags, progress; +unsigned long progress;cond_resched ();/* We now go into synchronous reclaim */ cpuset_memory_pressure_bump (); -psi_memstall_enter (&pflags);fs_reclaim_acquire (gfp_mask); noreclaim_flag =memalloc_noreclaim_save ();
@@ -4610,7 +4609,6 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,memalloc_noreclaim_restore (noreclaim_flag);fs_reclaim_release (gfp_mask); -psi_memstall_leave (&pflags);cond_resched ();
@@ -4624,11 +4622,13 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, unsigned long *did_some_progress) {struct page *page =NULL ; +unsigned long pflags;bool drained = false; +psi_memstall_enter (&pflags); *did_some_progress =__perform_reclaim (gfp_mask, order, ac);if (unlikely (!(*did_some_progress))) -return NULL ; + goto out; retry: page =get_page_from_freelist (gfp_mask, order, alloc_flags, ac);
@@ -4644,6 +4644,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, drained = true; goto retry; } +out: +psi_memstall_leave (&pflags);return page; }
Signature · 1 line
2.35.1.473.g83b2b277ed-goog
Quoted 27 lines · Suren Baghdasaryan
When page allocation in direct reclaim path fails, the system will make one attempt to shrink per-cpu page lists and free pages from high alloc reserves. Draining per-cpu pages into buddy allocator can be a very slow operation because it's done using workqueues and the task in direct reclaim waits for all of them to finish before proceeding. Currently this time is not accounted as psi memory stall.
While testing mobile devices under extreme memory pressure, when allocations are failing during direct reclaim, we notices that psi events which would be expected in such conditions were not triggered. After profiling these cases it was determined that the reason for missing psi events was that a big chunk of time spent in direct reclaim is not accounted as memory stall, therefore psi would not reach the levels at which an event is generated. Further investigation revealed that the bulk of that unaccounted time was spent inside
drain_all_pagescall.A typical captured case when
drain_all_pagespath gets activated:
__alloc_pages_slowpathtook 44.644.613ns__perform_reclaim took 751.668ns (1.7%) drain_all_pages took 43.887.167ns (98.3%)PSI in this case records the time spent in
__perform_reclaimbut ignoresdrain_all_pages, IOW it misses 98.3% of the time spent in__alloc_pages_slowpath.
Quoted 7 lines
Annotate
__alloc_pages_direct_reclaimin its entirety so that delays from handling page allocation failure in the direct reclaim path are accounted as memory stall.Reported-by Tim Murray <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Acked-by Johannes Weiner <[email protected]>
Quoted 17 lines · Suren Baghdasaryan
When page allocation in direct reclaim path fails, the system will make one attempt to shrink per-cpu page lists and free pages from high alloc reserves. Draining per-cpu pages into buddy allocator can be a very slow operation because it's done using workqueues and the task in direct reclaim waits for all of them to finish before proceeding. Currently this time is not accounted as psi memory stall.
While testing mobile devices under extreme memory pressure, when allocations are failing during direct reclaim, we notices that psi events which would be expected in such conditions were not triggered. After profiling these cases it was determined that the reason for missing psi events was that a big chunk of time spent in direct reclaim is not accounted as memory stall, therefore psi would not reach the levels at which an event is generated. Further investigation revealed that the bulk of that unaccounted time was spent inside
drain_all_pagescall.
Quoted 5 lines
A typical captured case when
drain_all_pagespath gets activated:__alloc_pages_slowpath took 44.644.613ns __perform_reclaim took 751.668ns (1.7%) drain_all_pages took 43.887.167ns (98.3%)
Although the draining is done in the slow path these numbers suggest that we should really reconsider the use of WQ both for draining and other purposes (like vmstats).
Quoted 11 lines
PSI in this case records the time spent in
__perform_reclaimbut ignoresdrain_all_pages, IOW it misses 98.3% of the time spent in__alloc_pages_slowpath.Annotate
__alloc_pages_direct_reclaimin its entirety so that delays from handling page allocation failure in the direct reclaim path are accounted as memory stall.Reported-by Tim Murray <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Acked-by Johannes Weiner <[email protected]>
Thanks!
Quoted 60 lines
---
changes in v3:
- Movedpsi_memstall_leaveafter the "out" labelmm/page_alloc.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)mm/page_alloc.c +6
-4@@ -4595,13 +4595,12 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order, const struct alloc_context *ac) {unsigned int noreclaim_flag; -unsigned long pflags, progress; +unsigned long progress;cond_resched ();/* We now go into synchronous reclaim */ cpuset_memory_pressure_bump (); -psi_memstall_enter (&pflags);fs_reclaim_acquire (gfp_mask); noreclaim_flag =memalloc_noreclaim_save ();@@ -4610,7 +4609,6 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,memalloc_noreclaim_restore (noreclaim_flag);fs_reclaim_release (gfp_mask); -psi_memstall_leave (&pflags);cond_resched ();@@ -4624,11 +4622,13 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, unsigned long *did_some_progress) {struct page *page =NULL ; +unsigned long pflags;bool drained = false; +psi_memstall_enter (&pflags); *did_some_progress =__perform_reclaim (gfp_mask, order, ac);if (unlikely (!(*did_some_progress))) -return NULL ; + goto out; retry: page =get_page_from_freelist (gfp_mask, order, alloc_flags, ac);@@ -4644,6 +4644,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, drained = true; goto retry; } +out: +psi_memstall_leave (&pflags);return page; }Signature · 1 line
2.35.1.473.g83b2b277ed-goog
Signature · 2 lines
Michal Hocko SUSE Labs
Suren Baghdasaryan
reviewerQuoted 21 lines · Michal Hocko' via kernel-team
Quoted 19 lines · Suren Baghdasaryan
When page allocation in direct reclaim path fails, the system will make one attempt to shrink per-cpu page lists and free pages from high alloc reserves. Draining per-cpu pages into buddy allocator can be a very slow operation because it's done using workqueues and the task in direct reclaim waits for all of them to finish before proceeding. Currently this time is not accounted as psi memory stall.
While testing mobile devices under extreme memory pressure, when allocations are failing during direct reclaim, we notices that psi events which would be expected in such conditions were not triggered. After profiling these cases it was determined that the reason for missing psi events was that a big chunk of time spent in direct reclaim is not accounted as memory stall, therefore psi would not reach the levels at which an event is generated. Further investigation revealed that the bulk of that unaccounted time was spent inside
drain_all_pagescall.A typical captured case when
drain_all_pagespath gets activated:
Quoted 7 lines
Quoted 3 lines
__alloc_pages_slowpath took 44.644.613ns __perform_reclaim took 751.668ns (1.7%) drain_all_pages took 43.887.167ns (98.3%)Although the draining is done in the slow path these numbers suggest that we should really reconsider the use of WQ both for draining and other purposes (like vmstats).
Yep, I'm testing the kthread_create_worker_on_cpu approach suggested by Petr. Will post it later today if nothing regresses.
Quoted 85 lines
Quoted 5 lines
PSI in this case records the time spent in
__perform_reclaimbut ignoresdrain_all_pages, IOW it misses 98.3% of the time spent in__alloc_pages_slowpath.Quoted 7 lines
Annotate
__alloc_pages_direct_reclaimin its entirety so that delays from handling page allocation failure in the direct reclaim path are accounted as memory stall.Reported-by Tim Murray <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Acked-by Johannes Weiner <[email protected]>Acked-by Michal Hocko <[email protected]>Thanks!
Quoted 59 lines
---
changes in v3:
- Movedpsi_memstall_leaveafter the "out" labelmm/page_alloc.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)mm/page_alloc.c +6
-5@@ -4595,13 +4595,12 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order, const struct alloc_context *ac) {unsigned int noreclaim_flag; -unsigned long pflags, progress; +unsigned long progress;cond_resched ();/* We now go into synchronous reclaim */ cpuset_memory_pressure_bump (); -psi_memstall_enter (&pflags);fs_reclaim_acquire (gfp_mask); noreclaim_flag =memalloc_noreclaim_save ();@@ -4610,7 +4609,6 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,memalloc_noreclaim_restore (noreclaim_flag);fs_reclaim_release (gfp_mask); -psi_memstall_leave (&pflags);cond_resched ();@@ -4624,11 +4622,13 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, unsigned long *did_some_progress) {struct page *page =NULL ; +unsigned long pflags;bool drained = false; +psi_memstall_enter (&pflags); *did_some_progress =__perform_reclaim (gfp_mask, order, ac);if (unlikely (!(*did_some_progress))) -return NULL ; + goto out; retry: page =get_page_from_freelist (gfp_mask, order, alloc_flags, ac);@@ -4644,6 +4644,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, drained = true; goto retry; } +out: +psi_memstall_leave (&pflags);return page; } --Quoted 1 line
2.35.1.473.g83b2b277ed-goog
--
Michal Hocko
SUSE Labs--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Suren Baghdasaryan
reviewerQuoted 28 lines · Suren Baghdasaryan
Quoted 25 lines · Michal Hocko' via kernel-team
Quoted 22 lines · Suren Baghdasaryan
When page allocation in direct reclaim path fails, the system will make one attempt to shrink per-cpu page lists and free pages from high alloc reserves. Draining per-cpu pages into buddy allocator can be a very slow operation because it's done using workqueues and the task in direct reclaim waits for all of them to finish before proceeding. Currently this time is not accounted as psi memory stall.
While testing mobile devices under extreme memory pressure, when allocations are failing during direct reclaim, we notices that psi events which would be expected in such conditions were not triggered. After profiling these cases it was determined that the reason for missing psi events was that a big chunk of time spent in direct reclaim is not accounted as memory stall, therefore psi would not reach the levels at which an event is generated. Further investigation revealed that the bulk of that unaccounted time was spent inside
drain_all_pagescall.A typical captured case when
drain_all_pagespath gets activated:__alloc_pages_slowpath took 44.644.613ns __perform_reclaim took 751.668ns (1.7%) drain_all_pages took 43.887.167ns (98.3%)
Quoted 6 lines
Quoted 3 lines
Although the draining is done in the slow path these numbers suggest that we should really reconsider the use of WQ both for draining and other purposes (like vmstats).
Yep, I'm testing the
kthread_create_worker_on_cpuapproach suggested by Petr. Will post it later today if nothing regresses.
An RFC for kthreads approach is posted at
https://lore.kernel.org/all/[email protected]/
Quoted 86 lines
Quoted 86 lines
Quoted 6 lines
PSI in this case records the time spent in
__perform_reclaimbut ignoresdrain_all_pages, IOW it misses 98.3% of the time spent in__alloc_pages_slowpath.Quoted 7 lines
Annotate
__alloc_pages_direct_reclaimin its entirety so that delays from handling page allocation failure in the direct reclaim path are accounted as memory stall.Reported-by Tim Murray <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Acked-by Johannes Weiner <[email protected]>Acked-by Michal Hocko <[email protected]>Thanks!
Quoted 59 lines
---
changes in v3:
- Movedpsi_memstall_leaveafter the "out" labelmm/page_alloc.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)mm/page_alloc.c +6
-5@@ -4595,13 +4595,12 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order, const struct alloc_context *ac) {unsigned int noreclaim_flag; -unsigned long pflags, progress; +unsigned long progress;cond_resched ();/* We now go into synchronous reclaim */ cpuset_memory_pressure_bump (); -psi_memstall_enter (&pflags);fs_reclaim_acquire (gfp_mask); noreclaim_flag =memalloc_noreclaim_save ();@@ -4610,7 +4609,6 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,memalloc_noreclaim_restore (noreclaim_flag);fs_reclaim_release (gfp_mask); -psi_memstall_leave (&pflags);cond_resched ();@@ -4624,11 +4622,13 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, unsigned long *did_some_progress) {struct page *page =NULL ; +unsigned long pflags;bool drained = false; +psi_memstall_enter (&pflags); *did_some_progress =__perform_reclaim (gfp_mask, order, ac);if (unlikely (!(*did_some_progress))) -return NULL ; + goto out; retry: page =get_page_from_freelist (gfp_mask, order, alloc_flags, ac);@@ -4644,6 +4644,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, drained = true; goto retry; } +out: +psi_memstall_leave (&pflags);return page; } --Quoted 1 line
2.35.1.473.g83b2b277ed-goog
--
Michal Hocko
SUSE Labs--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].