mm: vmscan: do not pass reclaimed slab to vmpressure
Subsystems (4)
PATCH v2 1/2 mm: vmscan: do not pass reclaimed slab to vmpressure
It is noticed that during a global reclaim the memory reclaimed via shrinking the slabs can sometimes result in reclaimed pages being greater than the scanned pages in shrink_node. When this is passed to vmpressure, the unsigned arithmetic results in the pressure value to be huge, thus resulting in a critical event being sent to root cgroup. While this can be fixed by underflow checks in vmpressure, adding reclaimed slab without a corresponding increment of nr_scanned results in incorrect vmpressure reporting. So do not consider reclaimed slab pages in vmpressure calculation.
--- mm/vmscan.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
mm/vmscan.c +5 -5
@@ -2594,16 +2594,16 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) sc->nr_scanned - nr_scanned, node_lru_pages); -if (reclaim_state) { - sc->nr_reclaimed += reclaim_state->reclaimed_slab; - reclaim_state->reclaimed_slab =0 ; - } -/* Record the subtree's reclaim efficiency */ vmpressure (sc->gfp_mask, sc->target_mem_cgroup, true, sc->nr_scanned - nr_scanned, sc->nr_reclaimed - nr_reclaimed); +if (reclaim_state) { + sc->nr_reclaimed += reclaim_state->reclaimed_slab; + reclaim_state->reclaimed_slab =0 ; + } +if (sc->nr_reclaimed - nr_reclaimed) reclaimable = true;
Signature · 8 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/2 mm: vmpressure: fix sending wrong events on underflow
At the end of a window period, if the reclaimed pages is greater than scanned, an unsigned underflow can result in a huge pressure value and thus a critical event. Reclaimed pages is found to go higher than scanned because of the addition of reclaimed slab pages to reclaimed in shrink_node without a corresponding increment to scanned pages. Minchan Kim mentioned that this can also happen in the case of a THP page where the scanned is 1 and reclaimed could be 512.
--- mm/vmpressure.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/vmpressure.c +4 -1
@@ -112,8 +112,10 @@ static enum vmpressure_levels vmpressure_calc_level(unsigned long scanned, unsigned long reclaimed) {unsigned long scale = scanned + reclaimed; -unsigned long pressure; +unsigned long pressure =0 ; +if (reclaimed >= scanned) + goto out;/* * We calculate the ratio (in percents) of how many pages were * scanned vs. reclaimed in a given time frame (window). Note that
@@ -124,6 +126,7 @@ static enum vmpressure_levels vmpressure_calc_level(unsigned long scanned, pressure = scale - (reclaimed * scale / scanned); pressure = pressure *100 / scale; +out:pr_debug ("%s: %3lu (s: %lu r: %lu)\n" , __func__, pressure, scanned, reclaimed);
Signature · 8 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/2] mm: vmpressure: fix sending wrong events on underflow
Quoted 11 lines · Vinayak Menon
At the end of a window period, if the reclaimed pages is greater than scanned, an unsigned underflow can result in a huge pressure value and thus a critical event. Reclaimed pages is found to go higher than scanned because of the addition of reclaimed slab pages to reclaimed in
shrink_nodewithout a corresponding increment to scanned pages. Minchan Kim mentioned that this can also happen in the case of a THP page where the scanned is 1 and reclaimed could be 512.Signed-off-by Vinayak Menon <[email protected]>
Thanks for the fix up!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Vinayak Menon
It is noticed that during a global reclaim the memory reclaimed via shrinking the slabs can sometimes result in reclaimed pages being greater than the scanned pages in
shrink_node. When this is passed to vmpressure, the unsigned arithmetic results in the pressure value to be huge, thus resulting in a critical event being sent to root cgroup. While this can be fixed by underflow checks in vmpressure, adding reclaimed slab without a corresponding increment ofnr_scannedresults in incorrect vmpressure reporting. So do not consider reclaimed slab pages in vmpressure calculation.
I belive we could enhance the description better.
problem
VM include nr_reclaimed of slab but not nr_scanned so pressure calculation can be underflow.
solution
do not consider reclaimed slab pages for vmpressure
why
Freeing a page by slab shrinking depends on each slab's object population so the cost model(i.e., scan:free) is not fair with LRU pages. Also, every shrinker doesn't account reclaimed pages. Lastly, this regression happens since 6b4f7799c6a5
Quoted 14 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/vmscan.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)mm/vmscan.c +0
-0@@ -2594,16 +2594,16 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) sc->nr_scanned - nr_scanned, node_lru_pages);
mm/vmscan.c static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) · line 2601
Quoted 10 lines
mm/vmscan.c +0
-5-if (reclaim_state) { - sc->nr_reclaimed += reclaim_state->reclaimed_slab; - reclaim_state->reclaimed_slab =0 ; - } -/* Record the subtree's reclaim efficiency */ vmpressure (sc->gfp_mask, sc->target_mem_cgroup, true, sc->nr_scanned - nr_scanned, sc->nr_reclaimed - nr_reclaimed);
Please add comment about "vmpressure excludes reclaimed pages via slab because blah blah blah" so upcoming patches doesn't make mistake again.
Thanks!
Quoted 17 lines
mm/vmscan.c +5
-0+if (reclaim_state) { + sc->nr_reclaimed += reclaim_state->reclaimed_slab; + reclaim_state->reclaimed_slab =0 ; + } +if (sc->nr_reclaimed - nr_reclaimed) reclaimable = true;Signature · 8 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 22 lines · Minchan Kim
Quoted 11 lines · Vinayak Menon
It is noticed that during a global reclaim the memory reclaimed via shrinking the slabs can sometimes result in reclaimed pages being greater than the scanned pages in
shrink_node. When this is passed to vmpressure, the unsigned arithmetic results in the pressure value to be huge, thus resulting in a critical event being sent to root cgroup. While this can be fixed by underflow checks in vmpressure, adding reclaimed slab without a corresponding increment ofnr_scannedresults in incorrect vmpressure reporting. So do not consider reclaimed slab pages in vmpressure calculation.I belive we could enhance the description better.
problem
VM include
nr_reclaimedof slab but notnr_scannedso pressure calculation can be underflow.solution
Quoted 9 lines
do not consider reclaimed slab pages for vmpressure
why
Freeing a page by slab shrinking depends on each slab's object population so the cost
model(i.e., scan:free)is not fair with LRU pages. Also, every shrinker doesn't account reclaimed pages. Lastly, this regression happens since6b4f7799c6a5
Done. Sending an updated one.
Quoted 28 lines
Quoted 14 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/vmscan.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)mm/vmscan.c +0
-0@@ -2594,16 +2594,16 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) sc->nr_scanned - nr_scanned, node_lru_pages);mm/vmscan.c static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) · line 2601
Quoted 10 lines
mm/vmscan.c +0
-5-if (reclaim_state) { - sc->nr_reclaimed += reclaim_state->reclaimed_slab; - reclaim_state->reclaimed_slab =0 ; - } -/* Record the subtree's reclaim efficiency */ vmpressure (sc->gfp_mask, sc->target_mem_cgroup, true, sc->nr_scanned - nr_scanned, sc->nr_reclaimed - nr_reclaimed);Please add comment about "vmpressure excludes reclaimed pages via slab because blah blah blah" so upcoming patches doesn't make mistake again.
Quoted 2 lines
Thanks!
Done. Thanks Minchan.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v3 1/2 mm: vmscan: do not pass reclaimed slab to vmpressure
In reply to Vinayak MenonDuring global reclaim, the nr_reclaimed passed to vmpressure includes the pages reclaimed from slab. But the corresponding scanned slab pages is not passed. This can cause total reclaimed pages to be greater than scanned, causing an unsigned underflow in vmpressure resulting in a critical event being sent to root cgroup. So do not consider reclaimed slab pages for vmpressure calculation. The reclaimed pages from slab can be excluded because the freeing of a page by slab shrinking depends on each slab's object population, making the cost model (i.e. scan:free) different from that of LRU. Also, not every shrinker accounts the pages it reclaims. This is a regression introduced by commit 6b4f7799c6a5 ("mm: vmscan: invoke slab shrinkers from shrink_zone()").
--- mm/vmscan.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
mm/vmscan.c +12 -5
@@ -2594,16 +2594,23 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) sc->nr_scanned - nr_scanned, node_lru_pages); +/* +* Record the subtree's reclaim efficiency. The reclaimed +* pages from slab is excluded here because the corresponding +* scanned pages is not accounted. Moreover, freeing a page +* by slab shrinking depends on each slab's object population, +* making the cost model (i.e. scan:free) different from that +* of LRU. +*/ +vmpressure (sc->gfp_mask, sc->target_mem_cgroup, true, + sc->nr_scanned - nr_scanned, + sc->nr_reclaimed - nr_reclaimed); +if (reclaim_state) { sc->nr_reclaimed += reclaim_state->reclaimed_slab; reclaim_state->reclaimed_slab =0 ; } -/* Record the subtree's reclaim efficiency */ -vmpressure (sc->gfp_mask, sc->target_mem_cgroup, true, - sc->nr_scanned - nr_scanned, - sc->nr_reclaimed - nr_reclaimed); -if (sc->nr_reclaimed - nr_reclaimed) reclaimable = true;
Signature · 8 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 13 lines · Vinayak Menon
During global reclaim, the
nr_reclaimedpassed to vmpressure includes the pages reclaimed from slab. But the corresponding scanned slab pages is not passed. This can cause total reclaimed pages to be greater than scanned, causing an unsigned underflow in vmpressure resulting in a critical event being sent to root cgroup. So do not consider reclaimed slab pages for vmpressure calculation. The reclaimed pages from slab can be excluded because the freeing of a page by slab shrinking depends on each slab's object population, making the cost model (i.e. scan:free) different from that of LRU. Also, not every shrinker accounts the pages it reclaims. This is a regression introduced by commit6b4f7799c6a5("mm: vmscan: invoke slab shrinkers fromshrink_zone()").
Quoted 1 line
Signed-off-by Vinayak Menon <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Michal Hocko
reviewerQuoted 10 lines · Vinayak Menon
During global reclaim, the
nr_reclaimedpassed to vmpressure includes the pages reclaimed from slab. But the corresponding scanned slab pages is not passed. This can cause total reclaimed pages to be greater than scanned, causing an unsigned underflow in vmpressure resulting in a critical event being sent to root cgroup. So do not consider reclaimed slab pages for vmpressure calculation. The reclaimed pages from slab can be excluded because the freeing of a page by slab shrinking depends on each slab's object population, making the cost model (i.e. scan:free) different from that of LRU.
This might be true but what happens if the slab reclaim contributes significantly to the overal reclaim? This would be quite rare but not impossible.
I am wondering why we cannot simply make cap nr_reclaimed to nr_scanned and be done with this all? Sure it will be imprecise but the same will be true with this approach.
Quoted 3 lines
Also, not every shrinker accounts the pages it reclaims. This is a regression introduced by commit
6b4f7799c6a5("mm: vmscan: invoke slab shrinkers fromshrink_zone()").
We usually refer to the culprit comment as
Quoted 42 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/vmscan.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)mm/vmscan.c +12
-5@@ -2594,16 +2594,23 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) sc->nr_scanned - nr_scanned, node_lru_pages); +/* +* Record the subtree's reclaim efficiency. The reclaimed +* pages from slab is excluded here because the corresponding +* scanned pages is not accounted. Moreover, freeing a page +* by slab shrinking depends on each slab's object population, +* making the cost model (i.e. scan:free) different from that +* of LRU. +*/ +vmpressure (sc->gfp_mask, sc->target_mem_cgroup, true, + sc->nr_scanned - nr_scanned, + sc->nr_reclaimed - nr_reclaimed); +if (reclaim_state) { sc->nr_reclaimed += reclaim_state->reclaimed_slab; reclaim_state->reclaimed_slab =0 ; } -/* Record the subtree's reclaim efficiency */ -vmpressure (sc->gfp_mask, sc->target_mem_cgroup, true, - sc->nr_scanned - nr_scanned, - sc->nr_reclaimed - nr_reclaimed); -if (sc->nr_reclaimed - nr_reclaimed) reclaimable = true;Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Michal Hocko
reviewerQuoted 12 lines · Michal Hocko
Quoted 10 lines · Vinayak Menon
During global reclaim, the
nr_reclaimedpassed to vmpressure includes the pages reclaimed from slab. But the corresponding scanned slab pages is not passed. This can cause total reclaimed pages to be greater than scanned, causing an unsigned underflow in vmpressure resulting in a critical event being sent to root cgroup. So do not consider reclaimed slab pages for vmpressure calculation. The reclaimed pages from slab can be excluded because the freeing of a page by slab shrinking depends on each slab's object population, making the cost model (i.e. scan:free) different from that of LRU.
Quoted 7 lines
This might be true but what happens if the slab reclaim contributes significantly to the overal reclaim? This would be quite rare but not impossible.
I am wondering why we cannot simply make cap
nr_reclaimedtonr_scannedand be done with this all? Sure it will be imprecise but the same will be true with this approach.
In other words something as "beautiful" as the following:
mm/vmpressure.c +9 -0
@@ -236,6 +236,15 @@ void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,return ;/* +* Due to accounting issues - e.g. THP contributing 1 to scanned but +* potentially much more to reclaimed or SLAB pages not contributing +* to scanned at all - we have to skew reclaimed to prevent from +* wrong pressure levels due to overflows. +*/ +if (reclaimed > scanned) + reclaimed = scanned; + +/* * If we got here with no pages scanned, then that is an indicator * that reclaimer was unable to find any shrinkable LRUs at the * current scanning depth. But it does not mean that we should
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 13 lines · Michal Hocko
Quoted 12 lines · Michal Hocko
Quoted 10 lines · Vinayak Menon
During global reclaim, the
nr_reclaimedpassed to vmpressure includes the pages reclaimed from slab. But the corresponding scanned slab pages is not passed. This can cause total reclaimed pages to be greater than scanned, causing an unsigned underflow in vmpressure resulting in a critical event being sent to root cgroup. So do not consider reclaimed slab pages for vmpressure calculation. The reclaimed pages from slab can be excluded because the freeing of a page by slab shrinking depends on each slab's object population, making the cost model (i.e. scan:free) different from that of LRU.
Quoted 7 lines
Quoted 7 lines
This might be true but what happens if the slab reclaim contributes significantly to the overal reclaim? This would be quite rare but not impossible.
I am wondering why we cannot simply make cap
nr_reclaimedtonr_scannedand be done with this all? Sure it will be imprecise but the same will be true with this approach.
Thinking of a case where 100 LRU pages were scanned and only 10 were reclaimed. Now, say slab reclaimed 100 pages and we have no idea how many were scanned. The actual vmpressure of 90 will now be 0 because of the addition on 100 slab pages. So underflow was not the only issue, but incorrect vmpressure. Even though the slab reclaimed is not accounted in vmpressure, the slab reclaimed
pages will have a feedback effect on the LRU pressure right ? i.e. the next LRU scan
will either be less or delayed if enough slab pages are reclaimed, in turn lowering the
vmpressure or delaying it ? If that is so, the current approach of neglecting slab reclaimed
will provide more accurate vmpressure than capping nr_reclaimed to nr_scanned ? Our internal tests on Android actually shows the problem. When vmpressure with slab
reclaimed added is used to kill tasks, it does not kick in at the right time.
Quoted 9 lines
In other words something as "beautiful" as the following:
mm/vmpressure.c +0
-0@@ -236,6 +236,15 @@ void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,return ;
mm/vmpressure.c void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree, · line 247
Quoted 10 lines
mm/vmpressure.c +9
-0/* +* Due to accounting issues - e.g. THP contributing 1 to scanned but +* potentially much more to reclaimed or SLAB pages not contributing +* to scanned at all - we have to skew reclaimed to prevent from +* wrong pressure levels due to overflows. +*/ +if (reclaimed > scanned) + reclaimed = scanned; + +/*
This underflow problem is fixed by a separate patch
https://lkml.org/lkml/2017/1/27/48
That patch performs this check only once at the end of a window period. Is that ok ?
Thanks,
Vinayak
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Michal Hocko
reviewerQuoted 22 lines · vinayak menon
Quoted 13 lines · Michal Hocko
Quoted 12 lines · Michal Hocko
Quoted 10 lines · Vinayak Menon
During global reclaim, the
nr_reclaimedpassed to vmpressure includes the pages reclaimed from slab. But the corresponding scanned slab pages is not passed. This can cause total reclaimed pages to be greater than scanned, causing an unsigned underflow in vmpressure resulting in a critical event being sent to root cgroup. So do not consider reclaimed slab pages for vmpressure calculation. The reclaimed pages from slab can be excluded because the freeing of a page by slab shrinking depends on each slab's object population, making the cost model (i.e. scan:free) different from that of LRU.Quoted 7 lines
Quoted 7 lines
This might be true but what happens if the slab reclaim contributes significantly to the overal reclaim? This would be quite rare but not impossible.
I am wondering why we cannot simply make cap
nr_reclaimedtonr_scannedand be done with this all? Sure it will be imprecise but the same will be true with this approach.
Quoted 5 lines
Thinking of a case where 100 LRU pages were scanned and only 10 were reclaimed. Now, say slab reclaimed 100 pages and we have no idea how many were scanned. The actual vmpressure of 90 will now be 0 because of the addition on 100 slab pages. So underflow was not the only issue, but incorrect vmpressure.
Is this actually a problem. The end result - enough pages being reclaimed should matter, no?
Quoted 5 lines
Even though the slab reclaimed is not accounted in vmpressure, the slab reclaimed pages will have a feedback effect on the LRU pressure right ? i.e. the next LRU scan will either be less or delayed if enough slab pages are reclaimed, in turn lowering the vmpressure or delaying it ?
Not sure what you mean but we can break out from the direct reclaim because we have fulfilled the reclaim target and that is why I think that it shouldn't be really harmful to consider them in the pressure calculation. After all we are making reclaim progress and that should be considered. reclaimed/scanned is a reasonable estimation but it has many issues because it doesn't really tell how hard it was to get that number of pages reclaimed. We might have to wait for writeback which is something completely different from a clean page cache. There are certainly different possible metrics.
Quoted 3 lines
If that is so, the
current approach of neglecting slab reclaimed will provide more accurate vmpressure than cappingnr_reclaimedtonr_scanned?
The problem I can see is that you can get serious vmpressure events while the reclaim manages to provide pages we are asking for and later decisions might be completely inappropriate.
Quoted 4 lines
Our
internal tests on Android actually shows the problem. When vmpressure with slab reclaimed added is used to kill tasks, it does not kick in at the right time.
With the skewed reclaimed? How that happens? Could you elaborate more?
Quoted 8 lines
Quoted 8 lines
In other words something as "beautiful" as the following:
mm/vmpressure.c +0
-0@@ -236,6 +236,15 @@ void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,return ;
Quoted 15 lines
mm/vmpressure.c void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree, · line 247
Quoted 10 lines
mm/vmpressure.c +9
-0/* +* Due to accounting issues - e.g. THP contributing 1 to scanned but +* potentially much more to reclaimed or SLAB pages not contributing +* to scanned at all - we have to skew reclaimed to prevent from +* wrong pressure levels due to overflows. +*/ +if (reclaimed > scanned) + reclaimed = scanned; + +/* This underflow problem is fixed by a separate patch
https://lkml.org/lkml/2017/1/27/48
That patch performs this check only once at the end of a window period. Is that ok ?
I have seen that patch but pushing that up into the vmpressure makes more sense to me because it is more obvious. Not something I would insist on, though.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 23 lines · Michal Hocko
Quoted 22 lines · vinayak menon
Quoted 20 lines · Michal Hocko
Quoted 19 lines · Michal Hocko
Quoted 10 lines · Vinayak Menon
During global reclaim, the
nr_reclaimedpassed to vmpressure includes the pages reclaimed from slab. But the corresponding scanned slab pages is not passed. This can cause total reclaimed pages to be greater than scanned, causing an unsigned underflow in vmpressure resulting in a critical event being sent to root cgroup. So do not consider reclaimed slab pages for vmpressure calculation. The reclaimed pages from slab can be excluded because the freeing of a page by slab shrinking depends on each slab's object population, making the cost model (i.e. scan:free) different from that of LRU.This might be true but what happens if the slab reclaim contributes significantly to the overal reclaim? This would be quite rare but not impossible.
I am wondering why we cannot simply make cap
nr_reclaimedtonr_scannedand be done with this all? Sure it will be imprecise but the same will be true with this approach.
Quoted 9 lines
Quoted 5 lines
Thinking of a case where 100 LRU pages were scanned and only 10 were reclaimed. Now, say slab reclaimed 100 pages and we have no idea how many were scanned. The actual vmpressure of 90 will now be 0 because of the addition on 100 slab pages. So underflow was not the only issue, but incorrect vmpressure.
Is this actually a problem. The end result - enough pages being reclaimed should matter, no?
But vmpressure is incorrect now, no ? Because the scanned slab pages is not included in nr_scanned (the cost). The 100 scanned and 10 reclaimed from LRU were a reasonable estimate as you said, and to that we are adding a reclaimed value alone without
scanned and thus making it incorrect ? Because the cost of slab reclaim is not accounted. But I agree that the vmpressure value would have been more correct if it could include both scanned and reclaimed from slab. And may be more correct
if we can include the scanned and reclaimed from all shrinkers which I think is not
the case right now (lowmemorykiller, zsmalloc etc). But as Minchan was pointing out, since the cost model for slab is different, would it be fine to just add reclaimed
from slab to vmpressure ?
Quoted 6 lines
Quoted 5 lines
Even though the slab reclaimed is not accounted in vmpressure, the slab reclaimed pages will have a feedback effect on the LRU pressure right ? i.e. the next LRU scan will either be less or delayed if enough slab pages are reclaimed, in turn lowering the vmpressure or delaying it ?
Quoted 10 lines
Not sure what you mean but we can break out from the direct reclaim because we have fulfilled the reclaim target and that is why I think that it shouldn't be really harmful to consider them in the pressure calculation. After all we are making reclaim progress and that should be considered. reclaimed/scanned is a reasonable estimation but it has many issues because it doesn't really tell how hard it was to get that number of pages reclaimed. We might have to wait for writeback which is something completely different from a clean page cache. There are certainly different possible metrics.
I see.
Quoted 8 lines
Quoted 3 lines
If that is so, the
current approach of neglecting slab reclaimed will provide more accurate vmpressure than cappingnr_reclaimedtonr_scanned?The problem I can see is that you can get serious vmpressure events while the reclaim manages to provide pages we are asking for and later decisions might be completely inappropriate.
Quoted 6 lines
Quoted 4 lines
Our
internal tests on Android actually shows the problem. When vmpressure with slab reclaimed added is used to kill tasks, it does not kick in at the right time.With the skewed reclaimed? How that happens? Could you elaborate more?
Yes. Because of the skewed reclaim. The observation is that the vmpressure critical events are received late. Because of adding slab reclaimed without corresponding scanned, the vmpressure values are diluted resulting in lesser number of critical events at the beginning, resulting in tasks not being chosen to
be killed. This increases the memory pressure and finally result in late critical events,
but by that time the task launch latencies are impacted.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Michal Hocko
reviewerQuoted 30 lines · vinayak menon
Quoted 29 lines · Michal Hocko
Quoted 27 lines · vinayak menon
Quoted 13 lines · Michal Hocko
Quoted 12 lines · Michal Hocko
Quoted 10 lines · Vinayak Menon
During global reclaim, the
nr_reclaimedpassed to vmpressure includes the pages reclaimed from slab. But the corresponding scanned slab pages is not passed. This can cause total reclaimed pages to be greater than scanned, causing an unsigned underflow in vmpressure resulting in a critical event being sent to root cgroup. So do not consider reclaimed slab pages for vmpressure calculation. The reclaimed pages from slab can be excluded because the freeing of a page by slab shrinking depends on each slab's object population, making the cost model (i.e. scan:free) different from that of LRU.Quoted 7 lines
Quoted 7 lines
This might be true but what happens if the slab reclaim contributes significantly to the overal reclaim? This would be quite rare but not impossible.
I am wondering why we cannot simply make cap
nr_reclaimedtonr_scannedand be done with this all? Sure it will be imprecise but the same will be true with this approach.Thinking of a case where 100 LRU pages were scanned and only 10 were reclaimed. Now, say slab reclaimed 100 pages and we have no idea how many were scanned. The actual vmpressure of 90 will now be 0 because of the addition on 100 slab pages. So underflow was not the only issue, but incorrect vmpressure.
Quoted 5 lines
Quoted 3 lines
Is this actually a problem. The end result - enough pages being reclaimed should matter, no?
But vmpressure is incorrect now, no ?
What does it mean incorrect? vmpressure is just an approximation that tells us how much we struggle to reclaim memory. If we are making a progress then we shouldn't reach higher levels.
Quoted 5 lines
Because the scanned slab pages
is not included innr_scanned(the cost). The 100 scanned and 10 reclaimed from LRU were a reasonable estimate as you said, and to that we are adding a reclaimed value alone without scanned and thus making it incorrect ? Because the cost of slab reclaim is not accounted.
there are other costs which are not included. E.g. stalling because of dirty pages etc...
Quoted 8 lines
But
I agree that the vmpressure value would have been more correct if it could include both scanned and reclaimed from slab. And may be more correct if we can include the scanned and reclaimed from all shrinkers which I think is not the case right now (lowmemorykiller, zsmalloc etc). But as Minchan was pointing out, since the cost model for slab is different, would it be fine to just add reclaimed from slab to vmpressure ?
Get back to your example. Do you really prefer seeing an alarm just because we had hard time reclaiming LRU pages which might be pinned due to reclaimable slab pages (e.g. fs metadata) when the slab reclaim can free enough of them?
vmpressure never had a good semantic, it is just an approximation that happened to work for some workloads which it was proposed for.
[...]
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Our
internal tests on Android actually shows the problem. When vmpressure with slab reclaimed added is used to kill tasks, it does not kick in at the right time.
Quoted 7 lines
Quoted 1 line
With the skewed reclaimed? How that happens? Could you elaborate more?
Yes. Because of the skewed reclaim. The observation is that the vmpressure critical events are received late. Because of adding slab reclaimed without corresponding scanned, the vmpressure values are diluted resulting in lesser number of critical events at the beginning, resulting in tasks not being chosen to be killed.
Why would you like to chose and kill a task when the slab reclaim can still make sufficient progres? Are you sure that the slab contribution to the stats makes all the above happening?
Quoted 3 lines
This increases the memory pressure and
finally result in late critical events, but by that time the task launch latencies are impacted.
I have seen vmpressure hitting critical events really quickly but that is mostly because the vmpressure uses only very simplistic approximation. Usually the reclaim goes well, until you hit to dirty or pinned pages. Then it can get really bad, so you can get from high effectiveness to 0 pretty quickly.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Michal Hocko
Why would you like to chose and kill a task when the slab reclaim can still make sufficient progres? Are you sure that the slab contribution to the stats makes all the above happening?
I agree that a task need not be killed if sufficient progress is made in reclaiming
memory say from slab. But here it looks like we have an impact because of just increasing the reclaimed without touching the scanned. It could be because of disimilar costs or not adding adding cost. I agree that vmpressure is only a reasonable
estimate which does not already include few other costs, but I am not sure whether it is ok
to add another element which further increases that disparity.
We noticed this problem when moving from 3.18 to 4.4 kernel version. With the same workload, the vmpressure events differ between 3.18 and 4.4 causing the above mentioned problem. And with this patch on 4.4 we get the same results as in 3,18. So the slab contribution to stats is making a difference.
Quoted 5 lines
Quoted 3 lines
This increases the memory pressure and
finally result in late critical events, but by that time the task launch latencies are impacted.
Quoted 8 lines
I have seen vmpressure hitting critical events really quickly but that is mostly because the vmpressure uses only very simplistic approximation. Usually the reclaim goes well, until you hit to dirty or pinned pages. Then it can get really bad, so you can get from high effectiveness to 0 pretty quickly.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Michal Hocko
reviewerQuoted 6 lines · vinayak menon
Quoted 5 lines · Michal Hocko
Why would you like to chose and kill a task when the slab reclaim can still make sufficient progres? Are you sure that the slab contribution to the stats makes all the above happening?
Quoted 13 lines
I agree that a task need not be killed if sufficient progress is made in reclaiming
memory say from slab. But here it looks like we have an impact because of just increasing the reclaimed without touching the scanned. It could be because of disimilar costs or not adding adding cost. I agree that vmpressure is only a reasonable
estimate which does not already include few other costs, but I am not sure whether it is ok
to add another element which further increases that disparity.
We noticed this problem when moving from 3.18 to 4.4 kernel version. With the same workload, the vmpressure events differ between 3.18 and 4.4 causing the above mentioned problem. And with this patch on 4.4 we get the same results as in 3,18. So the slab contribution to stats is making a difference.
Please document that in the changelog along with description of the workload that is affected. Ideally also add some data from /proc/vmstat so that we can see the reclaim activity.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 21 lines · Michal Hocko
Quoted 6 lines · vinayak menon
Quoted 5 lines · Michal Hocko
Why would you like to chose and kill a task when the slab reclaim can still make sufficient progres? Are you sure that the slab contribution to the stats makes all the above happening?
Quoted 13 lines
I agree that a task need not be killed if sufficient progress is made in reclaiming
memory say from slab. But here it looks like we have an impact because of just increasing the reclaimed without touching the scanned. It could be because of disimilar costs or not adding adding cost. I agree that vmpressure is only a reasonable
estimate which does not already include few other costs, but I am not sure whether it is ok
to add another element which further increases that disparity.
We noticed this problem when moving from 3.18 to 4.4 kernel version. With the same workload, the vmpressure events differ between 3.18 and 4.4 causing the above mentioned problem. And with this patch on 4.4 we get the same results as in 3,18. So the slab contribution to stats is making a difference.
Quoted 3 lines
Please document that in the changelog along with description of the workload that is affected. Ideally also add some data from
/proc/vmstatso that we can see the reclaim activity.
Sure, I will add these to the changelog.
Thanks,
Vinayak
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines · Michal Hocko
We usually refer to the culprit comment as
Fixes 6b4f7799c6a5 ("mm: vmscan: invoke slab shrinkers from shrink_zone()")
Thanks for pointing that out Michal. I see that added to the version of patch in mmotm.
Quoted 4 lines
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 12 lines · Michal Hocko
Quoted 10 lines · Vinayak Menon
During global reclaim, the
nr_reclaimedpassed to vmpressure includes the pages reclaimed from slab. But the corresponding scanned slab pages is not passed. This can cause total reclaimed pages to be greater than scanned, causing an unsigned underflow in vmpressure resulting in a critical event being sent to root cgroup. So do not consider reclaimed slab pages for vmpressure calculation. The reclaimed pages from slab can be excluded because the freeing of a page by slab shrinking depends on each slab's object population, making the cost model (i.e. scan:free) different from that of LRU.
Quoted 3 lines
This might be true but what happens if the slab reclaim contributes significantly to the overal reclaim? This would be quite rare but not impossible.
Of course, it is better for vmpressure to cover slab but it's not easy without page-based shrinking model, I think. It wold make vmpressure higher easily due to low reclaim efficiency compared to LRU pages. Yeah, vmpressure is not a perfect but no need to add more noises, either. It's regression since 6b4f7799c6a5 so I think this patch should go first and if someone want to cover slab really, he should spend a time to work it well. It's too much that Vinayak shuld make a effort for that.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>