mm: list_lru: hold nlru lock to avoid reading transient negative nr_items
Subsystems (2)
PATCH 1/1 mm: list_lru: hold nlru lock to avoid reading transient negative nr_items
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs shrinker_map bit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.
I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root
/ \
system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparent
dst->nr_items == 0
shrinker:
total_objects == 0
add src->nr_items to dst
set_bit
retrun SHRINK_EMPTY
clear_bit
list_lru_del()reparent again
dst->nr_items may go negative
due to current list_lru_del()
on CPU C
The second run of shrinker:
read nr_items without any synchronization, so it may see intermediate negative nr_items then total_objects may return 0 conincidently
keep the bit cleared
dst->nr_items != 0
skip set_bit
add scr->nr_item to dst
After this point dst->nr_item may never go zero, so reparenting will not set shrinker_map bit anymore. And since there is no task under user slice directly, so no new object will be added to its lru to set the shrinker map bit either. That bit is kept cleared forever.
How does list_lru_del() race with reparenting? It is because reparenting replaces childen's kmemcg_id to parent's without protecting from nlru->lock, so list_lru_del() may see parent's kmemcg_id but actually deleting items from child's lru, but dec'ing parent's nr_items, so the parent's nr_items may go negative as commit 2788cf0c401c268b4819c5407493a8769b7007aa ("memcg: reparent list_lrus and free kmemcg_id on css offline") says.
Can we move kmemcg_id replacement after reparenting? No, because the race with list_lru_del() may result in negative src->nr_items, but it will never be fixed. So the shrinker may never return SHRINK_EMPTY then keep the shrinker map bit set always. The shrinker will be always called for nonsense.
Can we synchronize list_lru_del() and reparenting? Yes, it could be done. But it seems we need introduce a new lock or use nlru->lock. But it sounds complicated to move kmemcg_id replacement code under nlru->lock. And list_lru_del() may be called quite often to exacerbate some hot path, i.e. dentry kill.
So, it sounds acceptable to synchronize reading nr_items to avoid seeing intermediate negative nr_items given the simplicity and it is typically just called by shrinkers when counting the freeable objects.
The patch is tested with some shrinker intensive workloads, no noticeable regression is soptted.
--- mm/list_lru.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
mm/list_lru.c +9 -2
@@ -178,10 +178,17 @@ unsigned long list_lru_count_one(struct list_lru *lru,struct list_lru_one *l;unsigned long count; -rcu_read_lock (); +/* +* Since list_lru_{add,del} may be called under an IRQ-safe lock, +* we have to use IRQ-safe primitives here to avoid deadlock. +* +* Hold the lock to prevent from seeing transient negative +* nr_items value. +*/ +spin_lock_irq (&nlru->lock); l =list_lru_from_memcg_idx (nlru,memcg_cache_id (memcg)); count =READ_ONCE (l->nr_items); -rcu_read_unlock (); +spin_unlock_irq (&nlru->lock);return count; }
Signature · 1 line
2.26.2
Quoted 77 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidentlykeep the bit cleared
dst->nr_items!= 0
skipset_bit
addscr->nr_itemto dstAfter this point
dst->nr_itemmay never go zero, so reparenting will not setshrinker_mapbit anymore. And since there is no task under user slice directly, so no new object will be added to its lru to set the shrinker map bit either. That bit is kept cleared forever.How does
list_lru_del()race with reparenting? It is because reparenting replaces childen'skmemcg_idto parent's without protecting fromnlru->lock, solist_lru_del()may see parent'skmemcg_idbut actually deleting items from child's lru, but dec'ing parent'snr_items, so the parent'snr_itemsmay go negative as commit2788cf0c401c268b4819c5407493a8769b7007aa("memcg: reparentlist_lrusand freekmemcg_idon css offline") says.Can we move
kmemcg_idreplacement after reparenting? No, because the race withlist_lru_del()may result in negativesrc->nr_items, but it will never be fixed. So the shrinker may never returnSHRINK_EMPTYthen keep the shrinker map bit set always. The shrinker will be always called for nonsense.Can we synchronize
list_lru_del()and reparenting? Yes, it could be done. But it seems we need introduce a new lock or usenlru->lock. But it sounds complicated to movekmemcg_idreplacement code undernlru->lock. Andlist_lru_del()may be called quite often to exacerbate some hot path, i.e. dentry kill.
Quoted 6 lines
So, it sounds acceptable to synchronize reading
nr_itemsto avoid seeing intermediate negativenr_itemsgiven the simplicity and it is typically just called by shrinkers when counting the freeable objects.The patch is tested with some shrinker intensive workloads, no noticeable regression is soptted.
Hi Yang!
It's really tricky, thank you for digging in! It's a perfect analysis!
I wonder though, if it's better to just always set the shrinker bit on reparenting if we do reparent some items? Then we'll avoid adding new synchronization to the hot path. What do you think?
--
@@ -534,7 +534,6 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid,
struct list_lru_node *nlru = &lru->node[nid];
int dst_idx = dst_memcg->kmemcg_id;
struct list_lru_one *src, *dst;
- bool set;
/*
* Since list_lru_{add,del} may be called under an IRQ-safe lock,
@@ -546,9 +545,8 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid,
dst = list_lru_from_memcg_idx(nlru, dst_idx);
list_splice_init(&src->list, &dst->list);
- set = (!dst->nr_items && src->nr_items);
dst->nr_items += src->nr_items;
- if (set)
+ if (src->nr_items)
memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru));
src->nr_items = 0;
--
Btw, it seems that the bug is quite old. I wonder why we haven't seen it before? Any ideas?
Thanks!
Quoted 37 lines
Cc Vladimir Davydov <[email protected]>Cc Kirill Tkhai <[email protected]>Cc Roman Gushchin <[email protected]>Cc Shakeel Butt <[email protected]>Signed-off-by Yang Shi <[email protected]>--- mm/list_lru.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)mm/list_lru.c +9
-2@@ -178,10 +178,17 @@ unsigned long list_lru_count_one(struct list_lru *lru,struct list_lru_one *l;unsigned long count; -rcu_read_lock (); +/* +* Since list_lru_{add,del} may be called under an IRQ-safe lock, +* we have to use IRQ-safe primitives here to avoid deadlock. +* +* Hold the lock to prevent from seeing transient negative +* nr_items value. +*/ +spin_lock_irq (&nlru->lock); l =list_lru_from_memcg_idx (nlru,memcg_cache_id (memcg)); count =READ_ONCE (l->nr_items); -rcu_read_unlock (); +spin_unlock_irq (&nlru->lock);return count; }Signature · 1 line
2.26.2
Quoted 86 lines · Roman Gushchin
Quoted 77 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidentlykeep the bit cleared
dst->nr_items!= 0
skipset_bit
addscr->nr_itemto dstAfter this point
dst->nr_itemmay never go zero, so reparenting will not setshrinker_mapbit anymore. And since there is no task under user slice directly, so no new object will be added to its lru to set the shrinker map bit either. That bit is kept cleared forever.How does
list_lru_del()race with reparenting? It is because reparenting replaces childen'skmemcg_idto parent's without protecting fromnlru->lock, solist_lru_del()may see parent'skmemcg_idbut actually deleting items from child's lru, but dec'ing parent'snr_items, so the parent'snr_itemsmay go negative as commit2788cf0c401c268b4819c5407493a8769b7007aa("memcg: reparentlist_lrusand freekmemcg_idon css offline") says.Can we move
kmemcg_idreplacement after reparenting? No, because the race withlist_lru_del()may result in negativesrc->nr_items, but it will never be fixed. So the shrinker may never returnSHRINK_EMPTYthen keep the shrinker map bit set always. The shrinker will be always called for nonsense.Can we synchronize
list_lru_del()and reparenting? Yes, it could be done. But it seems we need introduce a new lock or usenlru->lock. But it sounds complicated to movekmemcg_idreplacement code undernlru->lock. Andlist_lru_del()may be called quite often to exacerbate some hot path, i.e. dentry kill.Quoted 6 lines
So, it sounds acceptable to synchronize reading
nr_itemsto avoid seeing intermediate negativenr_itemsgiven the simplicity and it is typically just called by shrinkers when counting the freeable objects.The patch is tested with some shrinker intensive workloads, no noticeable regression is soptted.
Quoted 7 lines
Hi Yang!
It's really tricky, thank you for digging in! It's a perfect analysis!
I wonder though, if it's better to just always set the shrinker bit on reparenting if we do reparent some items? Then we'll avoid adding new synchronization to the hot path. What do you think?
Thanks a lot for the suggestion. I was thinking about the same approach too, but I thought src->nr_items may go zero due to concurrent list_lru_del() at the first place. But I just rethought the whole thing, it seems impossible that dst->nr_items goes negative and src->nr_items goes zero at the same time. list_lru_del() should just see either dst or src, it can't manipulate both lists simultaneously. So I think your suggestion should work. I will incarnate your suggestion in v2.
Quoted 25 lines
--
(quoted diff) +1
-4@@ -534,7 +534,6 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, struct list_lru_node *nlru = &lru->node[nid]; int dst_idx = dst_memcg->kmemcg_id; struct list_lru_one *src, *dst; - bool set; /* * Since list_lru_{add,del} may be called under an IRQ-safe lock,@@ -546,9 +545,8 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, dst = list_lru_from_memcg_idx(nlru, dst_idx); list_splice_init(&src->list, &dst->list); - set = (!dst->nr_items && src->nr_items); dst->nr_items += src->nr_items; - if (set) + if (src->nr_items) memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; --
Quoted 2 lines
Btw, it seems that the bug is quite old. I wonder why we haven't seen it before? Any ideas?
It is not new, but not that old from my point of view. The shrinker_map thing was introduced since v4.19, I bet pre-v4.19 kernel may still dominate in production environment. And, it needs some conditions (i.e. nr_inode + nr_dentry == 0 coincidently, and there is not task under dst memcg directly, etc) to trigger, so it seems unlikely to hit.
And the consequence may be not noticeable to the most people at all. We happened to see frequent OOMs on a couple of small machines (32G memory w/o swap, but most memory was consumed by anonymous pages) recently and they were already up for long time (almost 300 days), then the investigation leads to this race condition.
Quoted 42 lines
Thanks!
Quoted 37 lines
Cc Vladimir Davydov <[email protected]>Cc Kirill Tkhai <[email protected]>Cc Roman Gushchin <[email protected]>Cc Shakeel Butt <[email protected]>Signed-off-by Yang Shi <[email protected]>--- mm/list_lru.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)mm/list_lru.c +9
-3@@ -178,10 +178,17 @@ unsigned long list_lru_count_one(struct list_lru *lru,struct list_lru_one *l;unsigned long count; -rcu_read_lock (); +/* +* Since list_lru_{add,del} may be called under an IRQ-safe lock, +* we have to use IRQ-safe primitives here to avoid deadlock. +* +* Hold the lock to prevent from seeing transient negative +* nr_items value. +*/ +spin_lock_irq (&nlru->lock); l =list_lru_from_memcg_idx (nlru,memcg_cache_id (memcg)); count =READ_ONCE (l->nr_items); -rcu_read_unlock (); +spin_unlock_irq (&nlru->lock);return count; } --2.26.2
Quoted 60 lines · Yang Shi
Quoted 59 lines · Roman Gushchin
Quoted 57 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidentlykeep the bit cleared
dst->nr_items!= 0
skipset_bit
addscr->nr_itemto dstAfter this point
dst->nr_itemmay never go zero, so reparenting will not setshrinker_mapbit anymore. And since there is no task under user slice directly, so no new object will be added to its lru to set the shrinker map bit either. That bit is kept cleared forever.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
How does
list_lru_del()race with reparenting? It is because reparenting replaces childen'skmemcg_idto parent's without protecting fromnlru->lock, solist_lru_del()may see parent'skmemcg_idbut actually deleting items from child's lru, but dec'ing parent'snr_items, so the parent'snr_itemsmay go negative as commit2788cf0c401c268b4819c5407493a8769b7007aa("memcg: reparentlist_lrusand freekmemcg_idon css offline") says.
Also note that since the introduction of the slab reparenting, list_lru_from_kmem() can return the parent lru.
Quoted 28 lines
Quoted 20 lines
Quoted 13 lines
Can we move
kmemcg_idreplacement after reparenting? No, because the race withlist_lru_del()may result in negativesrc->nr_items, but it will never be fixed. So the shrinker may never returnSHRINK_EMPTYthen keep the shrinker map bit set always. The shrinker will be always called for nonsense.Can we synchronize
list_lru_del()and reparenting? Yes, it could be done. But it seems we need introduce a new lock or usenlru->lock. But it sounds complicated to movekmemcg_idreplacement code undernlru->lock. Andlist_lru_del()may be called quite often to exacerbate some hot path, i.e. dentry kill.Quoted 6 lines
So, it sounds acceptable to synchronize reading
nr_itemsto avoid seeing intermediate negativenr_itemsgiven the simplicity and it is typically just called by shrinkers when counting the freeable objects.The patch is tested with some shrinker intensive workloads, no noticeable regression is soptted.
Quoted 7 lines
Hi Yang!
It's really tricky, thank you for digging in! It's a perfect analysis!
I wonder though, if it's better to just always set the shrinker bit on reparenting if we do reparent some items? Then we'll avoid adding new synchronization to the hot path. What do you think?
Quoted 5 lines
Thanks a lot for the suggestion. I was thinking about the same approach too, but I thought
src->nr_itemsmay go zero due to concurrentlist_lru_del()at the first place. But I just rethought the whole thing, it seems impossible thatdst->nr_itemsgoes negative andsrc->nr_itemsgoes zero at the same time.
Even if it would be possible, it seems less scary: the next reparenting will likely set the bit. So we'll not get into the permanently bad state.
Quoted 40 lines
list_lru_del()should just
see either dst or src, it can't manipulate both lists simultaneously. So I think your suggestion should work. I will incarnate your suggestion in v2.Quoted 25 lines
--
(quoted diff) +1
-4@@ -534,7 +534,6 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, struct list_lru_node *nlru = &lru->node[nid]; int dst_idx = dst_memcg->kmemcg_id; struct list_lru_one *src, *dst; - bool set; /* * Since list_lru_{add,del} may be called under an IRQ-safe lock,@@ -546,9 +545,8 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, dst = list_lru_from_memcg_idx(nlru, dst_idx); list_splice_init(&src->list, &dst->list); - set = (!dst->nr_items && src->nr_items); dst->nr_items += src->nr_items; - if (set) + if (src->nr_items) memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; --Quoted 2 lines
Btw, it seems that the bug is quite old. I wonder why we haven't seen it before? Any ideas?
It is not new, but not that old from my point of view. The
shrinker_mapthing was introduced since v4.19, I bet pre-v4.19 kernel may still dominate in production environment. And, it needs some conditions (i.e.nr_inode+nr_dentry== 0 coincidently, and there is not task under dst memcg directly, etc) to trigger, so it seems unlikely to hit.
Quoted 5 lines
And the consequence may be not noticeable to the most people at all. We happened to see frequent OOMs on a couple of small machines (32G memory w/o swap, but most memory was consumed by anonymous pages) recently and they were already up for long time (almost 300 days), then the investigation leads to this race condition.
I agree that most users will unlikely notice it.
But https://www.spinics.net/lists/cgroups/msg27295.html looks very similar and can be caused by the same problem. Once you'll have v2, let's ask them to test it too.
Thanks!
Quoted 70 lines · Roman Gushchin
Quoted 60 lines · Yang Shi
Quoted 59 lines · Roman Gushchin
Quoted 57 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidentlykeep the bit cleared
dst->nr_items!= 0
skipset_bit
addscr->nr_itemto dstAfter this point
dst->nr_itemmay never go zero, so reparenting will not setshrinker_mapbit anymore. And since there is no task under user slice directly, so no new object will be added to its lru to set the shrinker map bit either. That bit is kept cleared forever.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
How does
list_lru_del()race with reparenting? It is because reparenting replaces childen'skmemcg_idto parent's without protecting fromnlru->lock, solist_lru_del()may see parent'skmemcg_idbut actually deleting items from child's lru, but dec'ing parent'snr_items, so the parent'snr_itemsmay go negative as commit2788cf0c401c268b4819c5407493a8769b7007aa("memcg: reparentlist_lrusand freekmemcg_idon css offline") says.
Quoted 2 lines
Also note that since the introduction of the slab reparenting,
list_lru_from_kmem()can return the parent lru.
Do you mean slab charge reparenting or lru reparenting? I think list_lru_from_kmem() can return the parent lru since lru reparenting.
Quoted 29 lines
Quoted 29 lines
Quoted 28 lines
Quoted 14 lines
Can we move
kmemcg_idreplacement after reparenting? No, because the race withlist_lru_del()may result in negativesrc->nr_items, but it will never be fixed. So the shrinker may never returnSHRINK_EMPTYthen keep the shrinker map bit set always. The shrinker will be always called for nonsense.Can we synchronize
list_lru_del()and reparenting? Yes, it could be done. But it seems we need introduce a new lock or usenlru->lock. But it sounds complicated to movekmemcg_idreplacement code undernlru->lock. Andlist_lru_del()may be called quite often to exacerbate some hot path, i.e. dentry kill.Quoted 6 lines
So, it sounds acceptable to synchronize reading
nr_itemsto avoid seeing intermediate negativenr_itemsgiven the simplicity and it is typically just called by shrinkers when counting the freeable objects.The patch is tested with some shrinker intensive workloads, no noticeable regression is soptted.
Hi Yang!
It's really tricky, thank you for digging in! It's a perfect analysis!
I wonder though, if it's better to just always set the shrinker bit on reparenting if we do reparent some items? Then we'll avoid adding new synchronization to the hot path. What do you think?
Quoted 8 lines
Quoted 5 lines
Thanks a lot for the suggestion. I was thinking about the same approach too, but I thought
src->nr_itemsmay go zero due to concurrentlist_lru_del()at the first place. But I just rethought the whole thing, it seems impossible thatdst->nr_itemsgoes negative andsrc->nr_itemsgoes zero at the same time.Even if it would be possible, it seems less scary: the next reparenting will likely set the bit. So we'll not get into the permanently bad state.
Unfortunately, no. Once the race happens, reparenting won't set the bit anymore since dst->nr_items won't go zero because the shrinker will not be called.
Quoted 47 lines
Quoted 41 lines
list_lru_del()should just
see either dst or src, it can't manipulate both lists simultaneously. So I think your suggestion should work. I will incarnate your suggestion in v2.Quoted 25 lines
--
(quoted diff) +1
-4@@ -534,7 +534,6 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, struct list_lru_node *nlru = &lru->node[nid]; int dst_idx = dst_memcg->kmemcg_id; struct list_lru_one *src, *dst; - bool set; /* * Since list_lru_{add,del} may be called under an IRQ-safe lock,@@ -546,9 +545,8 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, dst = list_lru_from_memcg_idx(nlru, dst_idx); list_splice_init(&src->list, &dst->list); - set = (!dst->nr_items && src->nr_items); dst->nr_items += src->nr_items; - if (set) + if (src->nr_items) memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; --Quoted 2 lines
Btw, it seems that the bug is quite old. I wonder why we haven't seen it before? Any ideas?
It is not new, but not that old from my point of view. The
shrinker_mapthing was introduced since v4.19, I bet pre-v4.19 kernel may still dominate in production environment. And, it needs some conditions (i.e.nr_inode+nr_dentry== 0 coincidently, and there is not task under dst memcg directly, etc) to trigger, so it seems unlikely to hit.Quoted 5 lines
And the consequence may be not noticeable to the most people at all. We happened to see frequent OOMs on a couple of small machines (32G memory w/o swap, but most memory was consumed by anonymous pages) recently and they were already up for long time (almost 300 days), then the investigation leads to this race condition.
Quoted 5 lines
I agree that most users will unlikely notice it.
But https://www.spinics.net/lists/cgroups/msg27295.html looks very similar and can be caused by the same problem. Once you'll have v2, let's ask them to test it too.
Yeah, may be the same root cause.
Quoted 2 lines
Thanks!
Quoted 71 lines · Yang Shi
Quoted 70 lines · Roman Gushchin
Quoted 67 lines · Yang Shi
Quoted 66 lines · Roman Gushchin
Quoted 64 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidentlykeep the bit cleared
dst->nr_items!= 0
skipset_bit
addscr->nr_itemto dstAfter this point
dst->nr_itemmay never go zero, so reparenting will not setshrinker_mapbit anymore. And since there is no task under user slice directly, so no new object will be added to its lru to set the shrinker map bit either. That bit is kept cleared forever.How does
list_lru_del()race with reparenting? It is because reparenting replaces childen'skmemcg_idto parent's without protecting fromnlru->lock, solist_lru_del()may see parent'skmemcg_idbut actually deleting items from child's lru, but dec'ing parent'snr_items, so the parent'snr_itemsmay go negative as commit2788cf0c401c268b4819c5407493a8769b7007aa("memcg: reparentlist_lrusand freekmemcg_idon css offline") says.
Quoted 5 lines
Quoted 2 lines
Also note that since the introduction of the slab reparenting,
list_lru_from_kmem()can return the parent lru.Do you mean slab charge reparenting or lru reparenting? I think
list_lru_from_kmem()can return the parent lru since lru reparenting.
objcg reparenting to be precise. It's actually kinda weird now, because there are two slightly different reparenting mechanisms. We might to wanna merge them in the future.
Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 22 lines
Quoted 15 lines
Can we move
kmemcg_idreplacement after reparenting? No, because the race withlist_lru_del()may result in negativesrc->nr_items, but it will never be fixed. So the shrinker may never returnSHRINK_EMPTYthen keep the shrinker map bit set always. The shrinker will be always called for nonsense.Can we synchronize
list_lru_del()and reparenting? Yes, it could be done. But it seems we need introduce a new lock or usenlru->lock. But it sounds complicated to movekmemcg_idreplacement code undernlru->lock. Andlist_lru_del()may be called quite often to exacerbate some hot path, i.e. dentry kill.Quoted 6 lines
So, it sounds acceptable to synchronize reading
nr_itemsto avoid seeing intermediate negativenr_itemsgiven the simplicity and it is typically just called by shrinkers when counting the freeable objects.The patch is tested with some shrinker intensive workloads, no noticeable regression is soptted.
Quoted 7 lines
Hi Yang!
It's really tricky, thank you for digging in! It's a perfect analysis!
I wonder though, if it's better to just always set the shrinker bit on reparenting if we do reparent some items? Then we'll avoid adding new synchronization to the hot path. What do you think?
Thanks a lot for the suggestion. I was thinking about the same approach too, but I thought
src->nr_itemsmay go zero due to concurrentlist_lru_del()at the first place. But I just rethought the whole thing, it seems impossible thatdst->nr_itemsgoes negative andsrc->nr_itemsgoes zero at the same time.
Quoted 6 lines
Quoted 2 lines
Even if it would be possible, it seems less scary: the next reparenting will likely set the bit. So we'll not get into the permanently bad state.
Unfortunately, no. Once the race happens, reparenting won't set the bit anymore since
dst->nr_itemswon't go zero because the shrinker will not be called.
I mean if we don't check dst->nr_items. Anyway, because it's an impossible case, no matter to discuss it :)
Quoted 50 lines · Yang Shi
Quoted 49 lines · Roman Gushchin
Quoted 47 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidently
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
keep the bit cleared dst->nr_items != 0 skip set_bit add scr->nr_item to dst
Btw, I think I have a simpler explanation:
A (0 objects)
|
B (N objects)
Let's say the reparenting races with the deletion of a single slab object. list_lru_del() can see parent's lru list and substract 1 from nr_items == 0, setting A's nr_items to -1 (the item is actually still in B's list).
memcg_drain_list_lru_node() will check !dst->nr_items && src->nr_items !-1 && N => 0 and not set the bit. But now we have (N-1) objects in A's list and the shrinker bit not set.
My proposed fix should resolve it. Alternatively, we maybe can check if dst->nr_items <= 0 and only then set the bit, but it seems to be an unnecessary optimization.
Thanks!
Quoted 63 lines · Roman Gushchin
Quoted 50 lines · Yang Shi
Quoted 49 lines · Roman Gushchin
Quoted 47 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidentlyQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
keep the bit cleared dst->nr_items != 0 skip set_bit add scr->nr_item to dstBtw, I think I have a simpler explanation:
A (0 objects)
|
B (N objects)
Quoted 7 lines
Let's say the reparenting races with the deletion of a single slab object.
list_lru_del()can see parent's lru list and substract 1 fromnr_items== 0, setting A'snr_itemsto -1 (the item is actually still in B's list).
memcg_drain_list_lru_node()will check !dst->nr_items &&src->nr_items!-1 && N => 0 and not set the bit. But now we have (N-1) objects in A's list and the shrinker bit not set.
Yes, this is the exact race I elaborated in the commit log.
Quoted 4 lines
My proposed fix should resolve it. Alternatively, we maybe can check if
dst->nr_items<= 0 and only then set the bit, but it seems to be an unnecessary optimization.
Yes, I think "src->nr_items != 0" is good enough.
Quoted 2 lines
Thanks!
Quoted 68 lines · Yang Shi
Quoted 67 lines · Roman Gushchin
Quoted 50 lines · Yang Shi
Quoted 49 lines · Roman Gushchin
Quoted 47 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidentlyQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
keep the bit cleared dst->nr_items != 0 skip set_bit add scr->nr_item to dstBtw, I think I have a simpler explanation:
A (0 objects)
|
B (N objects)Let's say the reparenting races with the deletion of a single slab object.
list_lru_del()can see parent's lru list and substract 1 fromnr_items== 0, setting A'snr_itemsto -1 (the item is actually still in B's list).
Quoted 5 lines
Quoted 3 lines
memcg_drain_list_lru_node()will check !dst->nr_items &&src->nr_items!-1 && N => 0 and not set the bit. But now we have (N-1) objects in A's list and the shrinker bit not set.Yes, this is the exact race I elaborated in the commit log.
Yes, the same problem for sure, I just think if we don't need to actually run into the shrinker code to mentally reproduce it, it's a bit easier model to follow.
Quoted 7 lines
Quoted 5 lines
My proposed fix should resolve it. Alternatively, we maybe can check if
dst->nr_items<= 0 and only then set the bit, but it seems to be an unnecessary optimization.Yes, I think "
src->nr_items!= 0" is good enough.
I agree.
Quoted 104 lines · Roman Gushchin
Quoted 77 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidentlykeep the bit cleared
dst->nr_items!= 0
skipset_bit
addscr->nr_itemto dstAfter this point
dst->nr_itemmay never go zero, so reparenting will not setshrinker_mapbit anymore. And since there is no task under user slice directly, so no new object will be added to its lru to set the shrinker map bit either. That bit is kept cleared forever.How does
list_lru_del()race with reparenting? It is because reparenting replaces childen'skmemcg_idto parent's without protecting fromnlru->lock, solist_lru_del()may see parent'skmemcg_idbut actually deleting items from child's lru, but dec'ing parent'snr_items, so the parent'snr_itemsmay go negative as commit2788cf0c401c268b4819c5407493a8769b7007aa("memcg: reparentlist_lrusand freekmemcg_idon css offline") says.Can we move
kmemcg_idreplacement after reparenting? No, because the race withlist_lru_del()may result in negativesrc->nr_items, but it will never be fixed. So the shrinker may never returnSHRINK_EMPTYthen keep the shrinker map bit set always. The shrinker will be always called for nonsense.Can we synchronize
list_lru_del()and reparenting? Yes, it could be done. But it seems we need introduce a new lock or usenlru->lock. But it sounds complicated to movekmemcg_idreplacement code undernlru->lock. Andlist_lru_del()may be called quite often to exacerbate some hot path, i.e. dentry kill.Quoted 6 lines
So, it sounds acceptable to synchronize reading
nr_itemsto avoid seeing intermediate negativenr_itemsgiven the simplicity and it is typically just called by shrinkers when counting the freeable objects.The patch is tested with some shrinker intensive workloads, no noticeable regression is soptted.
Hi Yang!
It's really tricky, thank you for digging in! It's a perfect analysis!
I wonder though, if it's better to just always set the shrinker bit on reparenting if we do reparent some items? Then we'll avoid adding new synchronization to the hot path. What do you think?
--
(quoted diff) +0
-1@@ -534,7 +534,6 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, struct list_lru_node *nlru = &lru->node[nid]; int dst_idx = dst_memcg->kmemcg_id; struct list_lru_one *src, *dst; - bool set; /* * Since list_lru_{add,del} may be called under an IRQ-safe lock,
Review static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, · line 551
Quoted 9 lines
(quoted diff) +1
-2@@ -546,9 +545,8 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, dst = list_lru_from_memcg_idx(nlru, dst_idx); list_splice_init(&src->list, &dst->list); - set = (!dst->nr_items && src->nr_items); dst->nr_items += src->nr_items; - if (set) + if (src->nr_items) memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0;
This looks like a good fix.
To make a code more clear, we may also want to group neighbouring lines under the same "if" branch in Yang's v2 resend.
Thanks,
Kirill
Quoted 118 lines · Kirill Tkhai
Quoted 104 lines · Roman Gushchin
Quoted 77 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidentlykeep the bit cleared
dst->nr_items!= 0
skipset_bit
addscr->nr_itemto dstAfter this point
dst->nr_itemmay never go zero, so reparenting will not setshrinker_mapbit anymore. And since there is no task under user slice directly, so no new object will be added to its lru to set the shrinker map bit either. That bit is kept cleared forever.How does
list_lru_del()race with reparenting? It is because reparenting replaces childen'skmemcg_idto parent's without protecting fromnlru->lock, solist_lru_del()may see parent'skmemcg_idbut actually deleting items from child's lru, but dec'ing parent'snr_items, so the parent'snr_itemsmay go negative as commit2788cf0c401c268b4819c5407493a8769b7007aa("memcg: reparentlist_lrusand freekmemcg_idon css offline") says.Can we move
kmemcg_idreplacement after reparenting? No, because the race withlist_lru_del()may result in negativesrc->nr_items, but it will never be fixed. So the shrinker may never returnSHRINK_EMPTYthen keep the shrinker map bit set always. The shrinker will be always called for nonsense.Can we synchronize
list_lru_del()and reparenting? Yes, it could be done. But it seems we need introduce a new lock or usenlru->lock. But it sounds complicated to movekmemcg_idreplacement code undernlru->lock. Andlist_lru_del()may be called quite often to exacerbate some hot path, i.e. dentry kill.Quoted 6 lines
So, it sounds acceptable to synchronize reading
nr_itemsto avoid seeing intermediate negativenr_itemsgiven the simplicity and it is typically just called by shrinkers when counting the freeable objects.The patch is tested with some shrinker intensive workloads, no noticeable regression is soptted.
Hi Yang!
It's really tricky, thank you for digging in! It's a perfect analysis!
I wonder though, if it's better to just always set the shrinker bit on reparenting if we do reparent some items? Then we'll avoid adding new synchronization to the hot path. What do you think?
--
(quoted diff) +0
-1@@ -534,7 +534,6 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, struct list_lru_node *nlru = &lru->node[nid]; int dst_idx = dst_memcg->kmemcg_id; struct list_lru_one *src, *dst; - bool set; /* * Since list_lru_{add,del} may be called under an IRQ-safe lock,Review static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, · line 551
Quoted 9 lines
(quoted diff) +1
-2@@ -546,9 +545,8 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, dst = list_lru_from_memcg_idx(nlru, dst_idx); list_splice_init(&src->list, &dst->list); - set = (!dst->nr_items && src->nr_items); dst->nr_items += src->nr_items; - if (set) + if (src->nr_items) memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0;This looks like a good fix.
Quoted 2 lines
To make a code more clear, we may also want to group neighbouring lines under the same "if" branch in Yang's v2 resend.
You mean something like the below (diff based on Roman's proposal)?
mm/list_lru.c +0 -0
@@ -545,10 +545,12 @@ static void memcg_drain_list_lru_node(struct
list_lru *lru, int nid,
dst = list_lru_from_memcg_idx(nlru, dst_idx);
list_splice_init(&src->list, &dst->list);- dst->nr_items += src->nr_items;
- if (src->nr_items)
+
+ if (src->nr_items) {
+ dst->nr_items += src->nr_items;
memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru));- src->nr_items = 0;
+ src->nr_items = 0;
+ }
spin_unlock_irq(&nlru->lock);
Quoted 3 lines
Thanks,
Kirill
Quoted 134 lines · Yang Shi
Quoted 118 lines · Kirill Tkhai
Quoted 104 lines · Roman Gushchin
Quoted 77 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidentlykeep the bit cleared
dst->nr_items!= 0
skipset_bit
addscr->nr_itemto dstAfter this point
dst->nr_itemmay never go zero, so reparenting will not setshrinker_mapbit anymore. And since there is no task under user slice directly, so no new object will be added to its lru to set the shrinker map bit either. That bit is kept cleared forever.How does
list_lru_del()race with reparenting? It is because reparenting replaces childen'skmemcg_idto parent's without protecting fromnlru->lock, solist_lru_del()may see parent'skmemcg_idbut actually deleting items from child's lru, but dec'ing parent'snr_items, so the parent'snr_itemsmay go negative as commit2788cf0c401c268b4819c5407493a8769b7007aa("memcg: reparentlist_lrusand freekmemcg_idon css offline") says.Can we move
kmemcg_idreplacement after reparenting? No, because the race withlist_lru_del()may result in negativesrc->nr_items, but it will never be fixed. So the shrinker may never returnSHRINK_EMPTYthen keep the shrinker map bit set always. The shrinker will be always called for nonsense.Can we synchronize
list_lru_del()and reparenting? Yes, it could be done. But it seems we need introduce a new lock or usenlru->lock. But it sounds complicated to movekmemcg_idreplacement code undernlru->lock. Andlist_lru_del()may be called quite often to exacerbate some hot path, i.e. dentry kill.Quoted 6 lines
So, it sounds acceptable to synchronize reading
nr_itemsto avoid seeing intermediate negativenr_itemsgiven the simplicity and it is typically just called by shrinkers when counting the freeable objects.The patch is tested with some shrinker intensive workloads, no noticeable regression is soptted.
Hi Yang!
It's really tricky, thank you for digging in! It's a perfect analysis!
I wonder though, if it's better to just always set the shrinker bit on reparenting if we do reparent some items? Then we'll avoid adding new synchronization to the hot path. What do you think?
--
(quoted diff) +0
-1@@ -534,7 +534,6 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, struct list_lru_node *nlru = &lru->node[nid]; int dst_idx = dst_memcg->kmemcg_id; struct list_lru_one *src, *dst; - bool set; /* * Since list_lru_{add,del} may be called under an IRQ-safe lock,Review static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, · line 551
Quoted 9 lines
(quoted diff) +1
-2@@ -546,9 +545,8 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, dst = list_lru_from_memcg_idx(nlru, dst_idx); list_splice_init(&src->list, &dst->list); - set = (!dst->nr_items && src->nr_items); dst->nr_items += src->nr_items; - if (set) + if (src->nr_items) memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0;This looks like a good fix.
Quoted 2 lines
To make a code more clear, we may also want to group neighbouring lines under the same "if" branch in Yang's v2 resend.
You mean something like the below (diff based on Roman's proposal)?
mm/list_lru.c +0
-1@@ -545,10 +545,12 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, dst =list_lru_from_memcg_idx (nlru, dst_idx);list_splice_init (&src->list, &dst->list); - dst->nr_items += src->nr_items;
mm/list_lru.c static void memcg_drain_list_lru_node(struct · line 556
Quoted 10 lines
mm/list_lru.c +5
-2-if (src->nr_items) + +if (src->nr_items) { + dst->nr_items += src->nr_items;memcg_set_shrinker_bit (dst_memcg, nid,lru_shrinker_id (lru)); - src->nr_items =0 ; + src->nr_items =0 ; + }spin_unlock_irq (&nlru->lock);
Yes.
Quoted 145 lines · Kirill Tkhai
Quoted 143 lines · Yang Shi
Quoted 118 lines · Kirill Tkhai
Quoted 104 lines · Roman Gushchin
Quoted 77 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidentlykeep the bit cleared
dst->nr_items!= 0
skipset_bit
addscr->nr_itemto dstAfter this point
dst->nr_itemmay never go zero, so reparenting will not setshrinker_mapbit anymore. And since there is no task under user slice directly, so no new object will be added to its lru to set the shrinker map bit either. That bit is kept cleared forever.How does
list_lru_del()race with reparenting? It is because reparenting replaces childen'skmemcg_idto parent's without protecting fromnlru->lock, solist_lru_del()may see parent'skmemcg_idbut actually deleting items from child's lru, but dec'ing parent'snr_items, so the parent'snr_itemsmay go negative as commit2788cf0c401c268b4819c5407493a8769b7007aa("memcg: reparentlist_lrusand freekmemcg_idon css offline") says.Can we move
kmemcg_idreplacement after reparenting? No, because the race withlist_lru_del()may result in negativesrc->nr_items, but it will never be fixed. So the shrinker may never returnSHRINK_EMPTYthen keep the shrinker map bit set always. The shrinker will be always called for nonsense.Can we synchronize
list_lru_del()and reparenting? Yes, it could be done. But it seems we need introduce a new lock or usenlru->lock. But it sounds complicated to movekmemcg_idreplacement code undernlru->lock. Andlist_lru_del()may be called quite often to exacerbate some hot path, i.e. dentry kill.Quoted 6 lines
So, it sounds acceptable to synchronize reading
nr_itemsto avoid seeing intermediate negativenr_itemsgiven the simplicity and it is typically just called by shrinkers when counting the freeable objects.The patch is tested with some shrinker intensive workloads, no noticeable regression is soptted.
Hi Yang!
It's really tricky, thank you for digging in! It's a perfect analysis!
I wonder though, if it's better to just always set the shrinker bit on reparenting if we do reparent some items? Then we'll avoid adding new synchronization to the hot path. What do you think?
--
(quoted diff) +0
-1@@ -534,7 +534,6 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, struct list_lru_node *nlru = &lru->node[nid]; int dst_idx = dst_memcg->kmemcg_id; struct list_lru_one *src, *dst; - bool set; /* * Since list_lru_{add,del} may be called under an IRQ-safe lock,Review static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, · line 551
Quoted 9 lines
(quoted diff) +1
-2@@ -546,9 +545,8 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, dst = list_lru_from_memcg_idx(nlru, dst_idx); list_splice_init(&src->list, &dst->list); - set = (!dst->nr_items && src->nr_items); dst->nr_items += src->nr_items; - if (set) + if (src->nr_items) memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0;This looks like a good fix.
Quoted 2 lines
To make a code more clear, we may also want to group neighbouring lines under the same "if" branch in Yang's v2 resend.
You mean something like the below (diff based on Roman's proposal)?
mm/list_lru.c +5
-3@@ -545,10 +545,12 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, dst =list_lru_from_memcg_idx (nlru, dst_idx);list_splice_init (&src->list, &dst->list); - dst->nr_items += src->nr_items; -if (src->nr_items) + +if (src->nr_items) { + dst->nr_items += src->nr_items;memcg_set_shrinker_bit (dst_memcg, nid,lru_shrinker_id (lru)); - src->nr_items =0 ; + src->nr_items =0 ; + }
Quoted 3 lines
mm/list_lru.c static void memcg_drain_list_lru_node(struct · line 556
Quoted 1 line
mm/list_lru.c +0
-0spin_unlock_irq (&nlru->lock);Yes.
Thanks for confirming. Will solve all the comments in v2.
Quoted 47 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidently
Quoted 4 lines
keep the bit cleared
dst->nr_items!= 0
skipset_bit
addscr->nr_itemto dst
Good catch, Yang. Thanks for investigating this.
But I agree with Roman it's better to fix that in rare-called place (memcg_drain_list_lru_node()), than in hot place (list_lru_count_one()).
Also, I'd added to description of new patch a reference to memcg_offline_kmem(), because this is the place, where child->kmemcg_id is rewritten, and this is the reason of lru's nr_items may become negative.
Quoted 66 lines
After this point
dst->nr_itemmay never go zero, so reparenting will not setshrinker_mapbit anymore. And since there is no task under user slice directly, so no new object will be added to its lru to set the shrinker map bit either. That bit is kept cleared forever.How does
list_lru_del()race with reparenting? It is because reparenting replaces childen'skmemcg_idto parent's without protecting fromnlru->lock, solist_lru_del()may see parent'skmemcg_idbut actually deleting items from child's lru, but dec'ing parent'snr_items, so the parent'snr_itemsmay go negative as commit2788cf0c401c268b4819c5407493a8769b7007aa("memcg: reparentlist_lrusand freekmemcg_idon css offline") says.Can we move
kmemcg_idreplacement after reparenting? No, because the race withlist_lru_del()may result in negativesrc->nr_items, but it will never be fixed. So the shrinker may never returnSHRINK_EMPTYthen keep the shrinker map bit set always. The shrinker will be always called for nonsense.Can we synchronize
list_lru_del()and reparenting? Yes, it could be done. But it seems we need introduce a new lock or usenlru->lock. But it sounds complicated to movekmemcg_idreplacement code undernlru->lock. Andlist_lru_del()may be called quite often to exacerbate some hot path, i.e. dentry kill.So, it sounds acceptable to synchronize reading
nr_itemsto avoid seeing intermediate negativenr_itemsgiven the simplicity and it is typically just called by shrinkers when counting the freeable objects.The patch is tested with some shrinker intensive workloads, no noticeable regression is soptted.
Cc Vladimir Davydov <[email protected]>Cc Kirill Tkhai <[email protected]>Cc Roman Gushchin <[email protected]>Cc Shakeel Butt <[email protected]>Signed-off-by Yang Shi <[email protected]>--- mm/list_lru.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)mm/list_lru.c +9
-2@@ -178,10 +178,17 @@ unsigned long list_lru_count_one(struct list_lru *lru,struct list_lru_one *l;unsigned long count; -rcu_read_lock (); +/* +* Since list_lru_{add,del} may be called under an IRQ-safe lock, +* we have to use IRQ-safe primitives here to avoid deadlock. +* +* Hold the lock to prevent from seeing transient negative +* nr_items value. +*/ +spin_lock_irq (&nlru->lock); l =list_lru_from_memcg_idx (nlru,memcg_cache_id (memcg)); count =READ_ONCE (l->nr_items); -rcu_read_unlock (); +spin_unlock_irq (&nlru->lock);return count; }
Quoted 54 lines · Kirill Tkhai
Quoted 47 lines · Yang Shi
When investigating a slab cache bloat problem, significant amount of negative dentry cache was seen, but confusingly they neither got shrunk by reclaimer (the host has very tight memory) nor be shrunk by dropping cache. The vmcore shows there are over 14M negative dentry objects on lru, but tracing result shows they were even not scanned at all. The further investigation shows the memcg's vfs
shrinker_mapbit is not set. So the reclaimer or dropping cache just skip calling vfs shrinker. So we have to reboot the hosts to get the memory back.I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting. But it seems there is race between shrinker map bit clear and reparenting by code inspection. The hypothesis is elaborated as below.
The memcg hierarchy on our production environment looks like:
root / \ system userThe main workloads are running under user slice's children, and it creates and removes memcg frequently. So reparenting happens very often under user slice, but no task is under user slice directly.
So with the frequent reparenting and tight memory pressure, the below hypothetical race condition may happen:
CPU A CPU B CPU C
reparentdst->nr_items == 0 shrinker: total_objects == 0 add src->nr_items to dst set_bit retrun SHRINK_EMPTY clear_bit list_lru_del()reparent again
dst->nr_itemsmay go negative
due to currentlist_lru_del()
on CPU C
The second run of shrinker:
readnr_itemswithout any synchronization, so it may see intermediate negativenr_itemsthentotal_objectsmay return 0 conincidentlyQuoted 4 lines
keep the bit cleared
dst->nr_items!= 0
skipset_bit
addscr->nr_itemto dst
Quoted 4 lines
Good catch, Yang. Thanks for investigating this.
But I agree with Roman it's better to fix that in rare-called place (
memcg_drain_list_lru_node()), than in hot place (list_lru_count_one()).
Yes, agreed. Will incarnate Roman's proposal in v2.
Quoted 4 lines
Also, I'd added to description of new patch a reference to
memcg_offline_kmem(), because this is the place, wherechild->kmemcg_idis rewritten, and this is the reason of lru'snr_itemsmay become negative.
Sure.
Quoted 69 lines
Quoted 67 lines
After this point
dst->nr_itemmay never go zero, so reparenting will not setshrinker_mapbit anymore. And since there is no task under user slice directly, so no new object will be added to its lru to set the shrinker map bit either. That bit is kept cleared forever.How does
list_lru_del()race with reparenting? It is because reparenting replaces childen'skmemcg_idto parent's without protecting fromnlru->lock, solist_lru_del()may see parent'skmemcg_idbut actually deleting items from child's lru, but dec'ing parent'snr_items, so the parent'snr_itemsmay go negative as commit2788cf0c401c268b4819c5407493a8769b7007aa("memcg: reparentlist_lrusand freekmemcg_idon css offline") says.Can we move
kmemcg_idreplacement after reparenting? No, because the race withlist_lru_del()may result in negativesrc->nr_items, but it will never be fixed. So the shrinker may never returnSHRINK_EMPTYthen keep the shrinker map bit set always. The shrinker will be always called for nonsense.Can we synchronize
list_lru_del()and reparenting? Yes, it could be done. But it seems we need introduce a new lock or usenlru->lock. But it sounds complicated to movekmemcg_idreplacement code undernlru->lock. Andlist_lru_del()may be called quite often to exacerbate some hot path, i.e. dentry kill.So, it sounds acceptable to synchronize reading
nr_itemsto avoid seeing intermediate negativenr_itemsgiven the simplicity and it is typically just called by shrinkers when counting the freeable objects.The patch is tested with some shrinker intensive workloads, no noticeable regression is soptted.
Cc Vladimir Davydov <[email protected]>Cc Kirill Tkhai <[email protected]>Cc Roman Gushchin <[email protected]>Cc Shakeel Butt <[email protected]>Signed-off-by Yang Shi <[email protected]>--- mm/list_lru.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)mm/list_lru.c +9
-2@@ -178,10 +178,17 @@ unsigned long list_lru_count_one(struct list_lru *lru,struct list_lru_one *l;unsigned long count; -rcu_read_lock (); +/* +* Since list_lru_{add,del} may be called under an IRQ-safe lock, +* we have to use IRQ-safe primitives here to avoid deadlock. +* +* Hold the lock to prevent from seeing transient negative +* nr_items value. +*/ +spin_lock_irq (&nlru->lock); l =list_lru_from_memcg_idx (nlru,memcg_cache_id (memcg)); count =READ_ONCE (l->nr_items); -rcu_read_unlock (); +spin_unlock_irq (&nlru->lock);return count; }