mm: thp: grab the lock before manipulation defer list
Subsystems (3)
PATCH v2 1/1 mm: thp: grab the lock before manipulation defer list
As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.
For example, the potential race would be:
CPU1 CPU2
mem_cgroup_move_account split_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock
When this sequence happens, the list_del_init() in mem_cgroup_move_account() would crash if CONFIG_DEBUG_LIST since the page is already been removed by list_del in split_huge_page_to_list().
---
v2:
* move check on compound outside suggested by Alexander
* an example of the race condition, suggested by Michal
--- mm/memcontrol.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
mm/memcontrol.c +11 -7
@@ -5368,10 +5368,12 @@ static int mem_cgroup_move_account(struct page *page, }#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (compound && !list_empty (page_deferred_list (page))) { +if (compound) {spin_lock (&from->deferred_split_queue.split_queue_lock); -list_del_init (page_deferred_list (page)); - from->deferred_split_queue.split_queue_len--; +if (!list_empty (page_deferred_list (page))) { +list_del_init (page_deferred_list (page)); + from->deferred_split_queue.split_queue_len--; + }spin_unlock (&from->deferred_split_queue.split_queue_lock); }#endif
@@ -5385,11 +5387,13 @@ static int mem_cgroup_move_account(struct page *page, page->mem_cgroup = to;#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (compound &&list_empty (page_deferred_list (page))) { +if (compound) {spin_lock (&to->deferred_split_queue.split_queue_lock); -list_add_tail (page_deferred_list (page), - &to->deferred_split_queue.split_queue); - to->deferred_split_queue.split_queue_len++; +if (list_empty (page_deferred_list (page))) { +list_add_tail (page_deferred_list (page), + &to->deferred_split_queue.split_queue); + to->deferred_split_queue.split_queue_len++; + }spin_unlock (&to->deferred_split_queue.split_queue_lock); }#endif
Signature · 1 line
2.17.1
Quoted 17 lines · Wei Yang
As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.
For example, the potential race would be:
CPU1 CPU2
mem_cgroup_move_accountsplit_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock
Quoted 8 lines
When this sequence happens, the
list_del_init()inmem_cgroup_move_account()would crash ifCONFIG_DEBUG_LISTsince the page is already been removed bylist_delinsplit_huge_page_to_list().Fixes 87eaceb3faa5 ("mm: thp: make deferred split shrinker memcg aware")Signed-off-by Wei Yang <[email protected]>Acked-by David Rientjes <[email protected]>
Thanks Wei!
Andrew, I'd also suggest:
Quoted 17 lines · Wei Yang
As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.
For example, the potential race would be:
CPU1 CPU2
mem_cgroup_move_accountsplit_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock
Quoted 8 lines
When this sequence happens, the
list_del_init()inmem_cgroup_move_account()would crash ifCONFIG_DEBUG_LISTsince the page is already been removed bylist_delinsplit_huge_page_to_list().Fixes 87eaceb3faa5 ("mm: thp: make deferred split shrinker memcg aware")Signed-off-by Wei Yang <[email protected]>Acked-by David Rientjes <[email protected]>
Thanks a lot for the changelog approvements!
Quoted 49 lines
---
v2:
* move check on compound outside suggested by Alexander
* an example of the race condition, suggested by Michal--- mm/memcontrol.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)mm/memcontrol.c +11
-7@@ -5368,10 +5368,12 @@ static int mem_cgroup_move_account(struct page *page, }#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (compound && !list_empty (page_deferred_list (page))) { +if (compound) {spin_lock (&from->deferred_split_queue.split_queue_lock); -list_del_init (page_deferred_list (page)); - from->deferred_split_queue.split_queue_len--; +if (!list_empty (page_deferred_list (page))) { +list_del_init (page_deferred_list (page)); + from->deferred_split_queue.split_queue_len--; + }spin_unlock (&from->deferred_split_queue.split_queue_lock); }#endif @@ -5385,11 +5387,13 @@ static int mem_cgroup_move_account(struct page *page, page->mem_cgroup = to;#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (compound &&list_empty (page_deferred_list (page))) { +if (compound) {spin_lock (&to->deferred_split_queue.split_queue_lock); -list_add_tail (page_deferred_list (page), - &to->deferred_split_queue.split_queue); - to->deferred_split_queue.split_queue_len++; +if (list_empty (page_deferred_list (page))) { +list_add_tail (page_deferred_list (page), + &to->deferred_split_queue.split_queue); + to->deferred_split_queue.split_queue_len++; + }spin_unlock (&to->deferred_split_queue.split_queue_lock); }#endif Signature · 1 line
2.17.1
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 5 lines · Wei Yang
As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.
For example, the potential race would be:
Quoted 11 lines
CPU1 CPU2
mem_cgroup_move_accountsplit_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock
I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:
Quoted 60 lines
When this sequence happens, the
list_del_init()inmem_cgroup_move_account()would crash ifCONFIG_DEBUG_LISTsince the page is already been removed bylist_delinsplit_huge_page_to_list().Fixes 87eaceb3faa5 ("mm: thp: make deferred split shrinker memcg aware")Signed-off-by Wei Yang <[email protected]>Acked-by David Rientjes <[email protected]>---
v2:
* move check on compound outside suggested by Alexander
* an example of the race condition, suggested by Michal--- mm/memcontrol.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)mm/memcontrol.c +11
-7@@ -5368,10 +5368,12 @@ static int mem_cgroup_move_account(struct page *page, }#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (compound && !list_empty (page_deferred_list (page))) { +if (compound) {spin_lock (&from->deferred_split_queue.split_queue_lock); -list_del_init (page_deferred_list (page)); - from->deferred_split_queue.split_queue_len--; +if (!list_empty (page_deferred_list (page))) { +list_del_init (page_deferred_list (page)); + from->deferred_split_queue.split_queue_len--; + }spin_unlock (&from->deferred_split_queue.split_queue_lock); }#endif @@ -5385,11 +5387,13 @@ static int mem_cgroup_move_account(struct page *page, page->mem_cgroup = to;#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (compound &&list_empty (page_deferred_list (page))) { +if (compound) {spin_lock (&to->deferred_split_queue.split_queue_lock); -list_add_tail (page_deferred_list (page), - &to->deferred_split_queue.split_queue); - to->deferred_split_queue.split_queue_len++; +if (list_empty (page_deferred_list (page))) { +list_add_tail (page_deferred_list (page), + &to->deferred_split_queue.split_queue); + to->deferred_split_queue.split_queue_len++; + }spin_unlock (&to->deferred_split_queue.split_queue_lock); }#endif Signature · 1 line
2.17.1
Signature · 1 line
Kirill A. Shutemov
Quoted 18 lines · Kirill A. Shutemov
Quoted 5 lines · Wei Yang
As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.
For example, the potential race would be:
Quoted 11 lines
CPU1 CPU2
mem_cgroup_move_accountsplit_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock
Quoted 2 lines
I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:
I am afraid not. Page lock is per page, while defer queue is per pgdate or memcg.
It is possible two page in the same pgdate or memcg grab page lock respectively and then access the same defer queue concurrently.
Quoted 66 lines
Quoted 60 lines
When this sequence happens, the
list_del_init()inmem_cgroup_move_account()would crash ifCONFIG_DEBUG_LISTsince the page is already been removed bylist_delinsplit_huge_page_to_list().Fixes 87eaceb3faa5 ("mm: thp: make deferred split shrinker memcg aware")Signed-off-by Wei Yang <[email protected]>Acked-by David Rientjes <[email protected]>---
v2:
* move check on compound outside suggested by Alexander
* an example of the race condition, suggested by Michal--- mm/memcontrol.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)mm/memcontrol.c +11
-7@@ -5368,10 +5368,12 @@ static int mem_cgroup_move_account(struct page *page, }#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (compound && !list_empty (page_deferred_list (page))) { +if (compound) {spin_lock (&from->deferred_split_queue.split_queue_lock); -list_del_init (page_deferred_list (page)); - from->deferred_split_queue.split_queue_len--; +if (!list_empty (page_deferred_list (page))) { +list_del_init (page_deferred_list (page)); + from->deferred_split_queue.split_queue_len--; + }spin_unlock (&from->deferred_split_queue.split_queue_lock); }#endif @@ -5385,11 +5387,13 @@ static int mem_cgroup_move_account(struct page *page, page->mem_cgroup = to;#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (compound &&list_empty (page_deferred_list (page))) { +if (compound) {spin_lock (&to->deferred_split_queue.split_queue_lock); -list_add_tail (page_deferred_list (page), - &to->deferred_split_queue.split_queue); - to->deferred_split_queue.split_queue_len++; +if (list_empty (page_deferred_list (page))) { +list_add_tail (page_deferred_list (page), + &to->deferred_split_queue.split_queue); + to->deferred_split_queue.split_queue_len++; + }spin_unlock (&to->deferred_split_queue.split_queue_lock); }#endif Signature · 1 line
2.17.1Signature · 1 line
Kirill A. Shutemov
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 19 lines · Wei Yang
Quoted 18 lines · Kirill A. Shutemov
Quoted 16 lines · Wei Yang
As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.
For example, the potential race would be:
CPU1 CPU2
mem_cgroup_move_accountsplit_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock
Quoted 8 lines
Quoted 2 lines
I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:
I am afraid not. Page lock is per page, while defer queue is per pgdate or memcg.
It is possible two page in the same pgdate or memcg grab page lock respectively and then access the same defer queue concurrently.
Look closer on the list_empty() argument. It's list_head local to the page. Too different pages can be handled in parallel without any problem in this particular scenario. As long as we as we modify it under the lock.
Said that, page lock here was somewhat accidential and I still belive we need to move the check under the lock anyway.
Signature · 1 line
Kirill A. Shutemov
Quoted 29 lines · Kirill A. Shutemov
Quoted 19 lines · Wei Yang
Quoted 18 lines · Kirill A. Shutemov
Quoted 16 lines · Wei Yang
As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.
For example, the potential race would be:
CPU1 CPU2
mem_cgroup_move_accountsplit_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlockQuoted 8 lines
Quoted 2 lines
I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:
I am afraid not. Page lock is per page, while defer queue is per pgdate or memcg.
It is possible two page in the same pgdate or memcg grab page lock respectively and then access the same defer queue concurrently.
Quoted 7 lines
Look closer on the
list_empty()argument. It'slist_headlocal to the page. Too different pages can be handled in parallel without any problem in this particular scenario. As long as we as we modify it under the lock.Said that, page lock here was somewhat accidential and I still belive we need to move the check under the lock anyway.
If my understanding is correct, you agree with my statement?
Quoted 2 lines
Signature · 1 line
Kirill A. Shutemov
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 4 lines · Wei Yang
Quoted 2 lines
Quoted 2 lines
It is possible two page in the same pgdate or memcg grab page lock respectively and then access the same defer queue concurrently.
If my understanding is correct, you agree with my statement?
Which one? If the one above then no. list_empty only accesses list_head for the struct page, not the queue.
Signature · 1 line
Kirill A. Shutemov
Quoted 9 lines · Kirill A. Shutemov
Quoted 4 lines · Wei Yang
Quoted 2 lines
Quoted 2 lines
It is possible two page in the same pgdate or memcg grab page lock respectively and then access the same defer queue concurrently.
If my understanding is correct, you agree with my statement?
Which one? If the one above then no.
list_emptyonly accesseslist_headfor the struct page, not the queue.
Ah, I get your point.
Quoted 2 lines
Signature · 1 line
Kirill A. Shutemov
Signature · 2 lines
Wei Yang Help you, Help me
Michal Hocko
maintainerQuoted 18 lines · Kirill A. Shutemov
Quoted 5 lines · Wei Yang
As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.
For example, the potential race would be:
Quoted 11 lines
CPU1 CPU2
mem_cgroup_move_accountsplit_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock
Quoted 4 lines
I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:
I am confused, if the above race is not possible then what would be a real race? We really do not want to have a patch with a misleading changelog, do we?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 19 lines · Michal Hocko
Quoted 18 lines · Kirill A. Shutemov
Quoted 16 lines · Wei Yang
As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.
For example, the potential race would be:
CPU1 CPU2
mem_cgroup_move_accountsplit_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock
Quoted 8 lines
Quoted 4 lines
I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:
I am confused, if the above race is not possible then what would be a real race? We really do not want to have a patch with a misleading changelog, do we?
The alternative is to make sure that all page_deferred_list() called with page lock taken.
I'll look into it.
Signature · 1 line
Kirill A. Shutemov
Quoted 25 lines · Kirill A. Shutemov
Quoted 24 lines · Michal Hocko
Quoted 22 lines · Kirill A. Shutemov
Quoted 5 lines · Wei Yang
As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.
For example, the potential race would be:
Quoted 11 lines
CPU1 CPU2
mem_cgroup_move_accountsplit_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlockI don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:
Quoted 8 lines
Quoted 3 lines
I am confused, if the above race is not possible then what would be a real race? We really do not want to have a patch with a misleading changelog, do we?
The alternative is to make sure that all
page_deferred_list()called with page lock taken.I'll look into it.
split_huge_page_to_list() has page lock taken.
free_transhuge_page() is in the free path and doesn't susceptible to the race.
deferred_split_scan() is trickier. list_move() should be safe against list_empty() as it will not produce false-positive list_empty(). list_del_init() *should* (correct me if I'm wrong) be safe because the page is freeing and memcg will not touch the page anymore.
deferred_split_huge_page() is a problematic one. It called from page_remove_rmap() path witch does require page lock. I don't see any obvious way to exclude race with mem_cgroup_move_account() here. Anybody else?
Wei, could you rewrite the commit message with deferred_split_huge_page() as a race source instead of split_huge_page_to_list()?
Signature · 1 line
Kirill A. Shutemov
Quoted 10 lines · Kirill A. Shutemov
split_huge_page_to_list()has page lock taken.
free_transhuge_page()is in the free path and doesn't susceptible to the race.
deferred_split_scan()is trickier.list_move()should be safe againstlist_empty()as it will not produce false-positivelist_empty().list_del_init()*should* (correct me if I'm wrong) be safe because the page is freeing and memcg will not touch the page anymore.
Quoted 8 lines
deferred_split_huge_page()is a problematic one. It called frompage_remove_rmap()path witch does require page lock. I don't see any obvious way to exclude race withmem_cgroup_move_account()here. Anybody else?Wei, could you rewrite the commit message with
deferred_split_huge_page()as a race source instead ofsplit_huge_page_to_list()?
I think describing the race in terms of deferred_split_huge_page() makes the most sense and I'd prefer a cc to stable for 5.4+. Even getting the split_queue_len, which is unsigned long, to underflow because of a list_empty(page_deferred_list()) check that is no longer accurate after the lock is taken would be a significant issue for shrinkers.
Quoted 21 lines · David Rientjes
Quoted 10 lines · Kirill A. Shutemov
split_huge_page_to_list()has page lock taken.
free_transhuge_page()is in the free path and doesn't susceptible to the race.
deferred_split_scan()is trickier.list_move()should be safe againstlist_empty()as it will not produce false-positivelist_empty().list_del_init()*should* (correct me if I'm wrong) be safe because the page is freeing and memcg will not touch the page anymore.Quoted 8 lines
deferred_split_huge_page()is a problematic one. It called frompage_remove_rmap()path witch does require page lock. I don't see any obvious way to exclude race withmem_cgroup_move_account()here. Anybody else?Wei, could you rewrite the commit message with
deferred_split_huge_page()as a race source instead ofsplit_huge_page_to_list()?
Quoted 5 lines
I think describing the race in terms of
deferred_split_huge_page()makes the most sense and I'd prefer a cc to stable for 5.4+. Even getting thesplit_queue_len, which is unsigned long, to underflow because of a list_empty(page_deferred_list()) check that is no longer accurate after the lock is taken would be a significant issue for shrinkers.
Oh, you are right. Even the list is not corrupted between deferred_split_scan() and mem_cgroup_move_account(), split_queue_len would be in a wrong state.
Hmm... to some extend, the page lock complicates the picture a little here, even it helps in some cases.
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 45 lines · Kirill A. Shutemov
Quoted 25 lines · Kirill A. Shutemov
Quoted 24 lines · Michal Hocko
Quoted 22 lines · Kirill A. Shutemov
Quoted 5 lines · Wei Yang
As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.
For example, the potential race would be:
Quoted 11 lines
CPU1 CPU2
mem_cgroup_move_accountsplit_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlockI don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:
Quoted 8 lines
Quoted 3 lines
I am confused, if the above race is not possible then what would be a real race? We really do not want to have a patch with a misleading changelog, do we?
The alternative is to make sure that all
page_deferred_list()called with page lock taken.I'll look into it.
split_huge_page_to_list()has page lock taken.
free_transhuge_page()is in the free path and doesn't susceptible to the race.
deferred_split_scan()is trickier.list_move()should be safe againstlist_empty()as it will not produce false-positivelist_empty().list_del_init()*should* (correct me if I'm wrong) be safe because the page is freeing and memcg will not touch the page anymore.
Quoted 4 lines
deferred_split_huge_page()is a problematic one. It called frompage_remove_rmap()path witch does require page lock. I don't see any obvious way to exclude race withmem_cgroup_move_account()here. Anybody else?
If my understanding is correct, the reason is deferred_split_huge_page() doesn't has page lock taken, right?
Quoted 6 lines
Wei, could you rewrite the commit message with
deferred_split_huge_page()as a race source instead ofsplit_huge_page_to_list()?Signature · 1 line
Kirill A. Shutemov
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 10 lines · Wei Yang
Quoted 10 lines
split_huge_page_to_list()has page lock taken.
free_transhuge_page()is in the free path and doesn't susceptible to the race.
deferred_split_scan()is trickier.list_move()should be safe againstlist_empty()as it will not produce false-positivelist_empty().list_del_init()*should* (correct me if I'm wrong) be safe because the page is freeing and memcg will not touch the page anymore.
Quoted 8 lines
Quoted 4 lines
deferred_split_huge_page()is a problematic one. It called frompage_remove_rmap()path witch does require page lock. I don't see any obvious way to exclude race withmem_cgroup_move_account()here. Anybody else?If my understanding is correct, the reason is
deferred_split_huge_page()doesn't has page lock taken, right?
I think the fix that you have proposed has inspired some deeper looks at the locking around the deferred split queue and the hope was that perhaps this could be protected by the page lock but it was found that at least in one path that isn't taken. So I believe your fix is still needed and any possible optimizations in this area can be proposed on top.