Skip to contentSkip to conversation

mm: thp: grab the lock before manipulation defer list

PATCH Superseded Wei Yang 4 participants 16 messages Jan 9, 2020 – Jan 15, 2020

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Wei Yang

author
#0 lore raw

PATCH v2 1/1 mm: thp: grab the lock before manipulation defer list

As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.

For example, the potential race would be:

CPU1 CPU2
mem_cgroup_move_account split_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock

When this sequence happens, the list_del_init() in mem_cgroup_move_account() would crash if CONFIG_DEBUG_LIST since the page is already been removed by list_del in split_huge_page_to_list().

Fixes 87eaceb3faa5 ("mm: thp: make deferred split shrinker memcg aware")

---
v2:
* move check on compound outside suggested by Alexander
* an example of the race condition, suggested by Michal

---
 mm/memcontrol.c | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)
mm/memcontrol.c +11 -7
index bc01423277c5..1492eefe4f3c 100644
@@ -5368,10 +5368,12 @@ static int mem_cgroup_move_account(struct page *page,
 	}
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	if (compound && !list_empty(page_deferred_list(page))) {
+	if (compound) {
 		spin_lock(&from->deferred_split_queue.split_queue_lock);
-		list_del_init(page_deferred_list(page));
-		from->deferred_split_queue.split_queue_len--;
+		if (!list_empty(page_deferred_list(page))) {
+			list_del_init(page_deferred_list(page));
+			from->deferred_split_queue.split_queue_len--;
+		}
 		spin_unlock(&from->deferred_split_queue.split_queue_lock);
 	}
 #endif
@@ -5385,11 +5387,13 @@ static int mem_cgroup_move_account(struct page *page,
 	page->mem_cgroup = to;
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	if (compound && list_empty(page_deferred_list(page))) {
+	if (compound) {
 		spin_lock(&to->deferred_split_queue.split_queue_lock);
-		list_add_tail(page_deferred_list(page),
-			      &to->deferred_split_queue.split_queue);
-		to->deferred_split_queue.split_queue_len++;
+		if (list_empty(page_deferred_list(page))) {
+			list_add_tail(page_deferred_list(page),
+				      &to->deferred_split_queue.split_queue);
+			to->deferred_split_queue.split_queue_len++;
+		}
 		spin_unlock(&to->deferred_split_queue.split_queue_lock);
 	}
 #endif
Signature · 1 line
2.17.1
#1 lore raw
Quoted 17 lines · Wei Yang

As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.

For example, the potential race would be:

CPU1 CPU2
mem_cgroup_move_account split_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock

Quoted 8 lines

When this sequence happens, the list_del_init() in mem_cgroup_move_account() would crash if CONFIG_DEBUG_LIST since the page is already been removed by list_del in split_huge_page_to_list().

Fixes 87eaceb3faa5 ("mm: thp: make deferred split shrinker memcg aware")

Thanks Wei!

Andrew, I'd also suggest:

maintainer Acked-by on v2 1/1
#2 lore raw
In reply to Wei Yang
Quoted 17 lines · Wei Yang

As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.

For example, the potential race would be:

CPU1 CPU2
mem_cgroup_move_account split_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock

Quoted 8 lines

When this sequence happens, the list_del_init() in mem_cgroup_move_account() would crash if CONFIG_DEBUG_LIST since the page is already been removed by list_del in split_huge_page_to_list().

Fixes 87eaceb3faa5 ("mm: thp: make deferred split shrinker memcg aware")

Thanks a lot for the changelog approvements!

Quoted 49 lines

---
v2:
* move check on compound outside suggested by Alexander
* an example of the race condition, suggested by Michal

---
 mm/memcontrol.c | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)
mm/memcontrol.c +11 -7
index bc01423277c5..1492eefe4f3c 100644
@@ -5368,10 +5368,12 @@ static int mem_cgroup_move_account(struct page *page,
 	}
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	if (compound && !list_empty(page_deferred_list(page))) {
+	if (compound) {
 		spin_lock(&from->deferred_split_queue.split_queue_lock);
-		list_del_init(page_deferred_list(page));
-		from->deferred_split_queue.split_queue_len--;
+		if (!list_empty(page_deferred_list(page))) {
+			list_del_init(page_deferred_list(page));
+			from->deferred_split_queue.split_queue_len--;
+		}
 		spin_unlock(&from->deferred_split_queue.split_queue_lock);
 	}
 #endif
@@ -5385,11 +5387,13 @@ static int mem_cgroup_move_account(struct page *page,
 	page->mem_cgroup = to;
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	if (compound && list_empty(page_deferred_list(page))) {
+	if (compound) {
 		spin_lock(&to->deferred_split_queue.split_queue_lock);
-		list_add_tail(page_deferred_list(page),
-			      &to->deferred_split_queue.split_queue);
-		to->deferred_split_queue.split_queue_len++;
+		if (list_empty(page_deferred_list(page))) {
+			list_add_tail(page_deferred_list(page),
+				      &to->deferred_split_queue.split_queue);
+			to->deferred_split_queue.split_queue_len++;
+		}
 		spin_unlock(&to->deferred_split_queue.split_queue_lock);
 	}
 #endif
Signature · 1 line
2.17.1
Signature · 2 lines
Michal Hocko
SUSE Labs
Acked-by on v2 1/1
#3 lore raw
In reply to Wei Yang
Quoted 5 lines · Wei Yang

As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.

For example, the potential race would be:

Quoted 11 lines

CPU1 CPU2
mem_cgroup_move_account split_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock

I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:

Quoted 60 lines

When this sequence happens, the list_del_init() in mem_cgroup_move_account() would crash if CONFIG_DEBUG_LIST since the page is already been removed by list_del in split_huge_page_to_list().

Fixes 87eaceb3faa5 ("mm: thp: make deferred split shrinker memcg aware")

---
v2:
* move check on compound outside suggested by Alexander
* an example of the race condition, suggested by Michal

---
 mm/memcontrol.c | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)
mm/memcontrol.c +11 -7
index bc01423277c5..1492eefe4f3c 100644
@@ -5368,10 +5368,12 @@ static int mem_cgroup_move_account(struct page *page,
 	}
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	if (compound && !list_empty(page_deferred_list(page))) {
+	if (compound) {
 		spin_lock(&from->deferred_split_queue.split_queue_lock);
-		list_del_init(page_deferred_list(page));
-		from->deferred_split_queue.split_queue_len--;
+		if (!list_empty(page_deferred_list(page))) {
+			list_del_init(page_deferred_list(page));
+			from->deferred_split_queue.split_queue_len--;
+		}
 		spin_unlock(&from->deferred_split_queue.split_queue_lock);
 	}
 #endif
@@ -5385,11 +5387,13 @@ static int mem_cgroup_move_account(struct page *page,
 	page->mem_cgroup = to;
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	if (compound && list_empty(page_deferred_list(page))) {
+	if (compound) {
 		spin_lock(&to->deferred_split_queue.split_queue_lock);
-		list_add_tail(page_deferred_list(page),
-			      &to->deferred_split_queue.split_queue);
-		to->deferred_split_queue.split_queue_len++;
+		if (list_empty(page_deferred_list(page))) {
+			list_add_tail(page_deferred_list(page),
+				      &to->deferred_split_queue.split_queue);
+			to->deferred_split_queue.split_queue_len++;
+		}
 		spin_unlock(&to->deferred_split_queue.split_queue_lock);
 	}
 #endif
Signature · 1 line
2.17.1
Signature · 1 line
 Kirill A. Shutemov
author
#4 lore raw
Quoted 18 lines · Kirill A. Shutemov
Quoted 5 lines · Wei Yang

As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.

For example, the potential race would be:

Quoted 11 lines

CPU1 CPU2
mem_cgroup_move_account split_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock

Quoted 2 lines

I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:

I am afraid not. Page lock is per page, while defer queue is per pgdate or memcg.

It is possible two page in the same pgdate or memcg grab page lock respectively and then access the same defer queue concurrently.

Quoted 66 lines
Quoted 60 lines

When this sequence happens, the list_del_init() in mem_cgroup_move_account() would crash if CONFIG_DEBUG_LIST since the page is already been removed by list_del in split_huge_page_to_list().

Fixes 87eaceb3faa5 ("mm: thp: make deferred split shrinker memcg aware")

---
v2:
* move check on compound outside suggested by Alexander
* an example of the race condition, suggested by Michal

---
 mm/memcontrol.c | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)
mm/memcontrol.c +11 -7
index bc01423277c5..1492eefe4f3c 100644
@@ -5368,10 +5368,12 @@ static int mem_cgroup_move_account(struct page *page,
 	}
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	if (compound && !list_empty(page_deferred_list(page))) {
+	if (compound) {
 		spin_lock(&from->deferred_split_queue.split_queue_lock);
-		list_del_init(page_deferred_list(page));
-		from->deferred_split_queue.split_queue_len--;
+		if (!list_empty(page_deferred_list(page))) {
+			list_del_init(page_deferred_list(page));
+			from->deferred_split_queue.split_queue_len--;
+		}
 		spin_unlock(&from->deferred_split_queue.split_queue_lock);
 	}
 #endif
@@ -5385,11 +5387,13 @@ static int mem_cgroup_move_account(struct page *page,
 	page->mem_cgroup = to;
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	if (compound && list_empty(page_deferred_list(page))) {
+	if (compound) {
 		spin_lock(&to->deferred_split_queue.split_queue_lock);
-		list_add_tail(page_deferred_list(page),
-			      &to->deferred_split_queue.split_queue);
-		to->deferred_split_queue.split_queue_len++;
+		if (list_empty(page_deferred_list(page))) {
+			list_add_tail(page_deferred_list(page),
+				      &to->deferred_split_queue.split_queue);
+			to->deferred_split_queue.split_queue_len++;
+		}
 		spin_unlock(&to->deferred_split_queue.split_queue_lock);
 	}
 #endif
Signature · 1 line
2.17.1
Signature · 1 line
Kirill A. Shutemov
Signature · 2 lines
Wei Yang
Help you, Help me
#5 lore raw
Quoted 19 lines · Wei Yang
Quoted 18 lines · Kirill A. Shutemov
Quoted 16 lines · Wei Yang

As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.

For example, the potential race would be:

CPU1 CPU2
mem_cgroup_move_account split_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock

Quoted 8 lines
Quoted 2 lines

I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:

I am afraid not. Page lock is per page, while defer queue is per pgdate or memcg.

It is possible two page in the same pgdate or memcg grab page lock respectively and then access the same defer queue concurrently.

Look closer on the list_empty() argument. It's list_head local to the page. Too different pages can be handled in parallel without any problem in this particular scenario. As long as we as we modify it under the lock.

Said that, page lock here was somewhat accidential and I still belive we need to move the check under the lock anyway.

Signature · 1 line
 Kirill A. Shutemov
author
#6 lore raw
Quoted 29 lines · Kirill A. Shutemov
Quoted 19 lines · Wei Yang
Quoted 18 lines · Kirill A. Shutemov
Quoted 16 lines · Wei Yang

As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.

For example, the potential race would be:

CPU1 CPU2
mem_cgroup_move_account split_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock

Quoted 8 lines
Quoted 2 lines

I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:

I am afraid not. Page lock is per page, while defer queue is per pgdate or memcg.

It is possible two page in the same pgdate or memcg grab page lock respectively and then access the same defer queue concurrently.

Quoted 7 lines

Look closer on the list_empty() argument. It's list_head local to the page. Too different pages can be handled in parallel without any problem in this particular scenario. As long as we as we modify it under the lock.

Said that, page lock here was somewhat accidential and I still belive we need to move the check under the lock anyway.

If my understanding is correct, you agree with my statement?

Quoted 2 lines
Signature · 1 line
Kirill A. Shutemov
Signature · 2 lines
Wei Yang
Help you, Help me
#7 lore raw
Quoted 4 lines · Wei Yang
Quoted 2 lines
Quoted 2 lines

It is possible two page in the same pgdate or memcg grab page lock respectively and then access the same defer queue concurrently.

If my understanding is correct, you agree with my statement?

Which one? If the one above then no. list_empty only accesses list_head for the struct page, not the queue.

Signature · 1 line
 Kirill A. Shutemov
author
#8 lore raw
Quoted 9 lines · Kirill A. Shutemov
Quoted 4 lines · Wei Yang
Quoted 2 lines
Quoted 2 lines

It is possible two page in the same pgdate or memcg grab page lock respectively and then access the same defer queue concurrently.

If my understanding is correct, you agree with my statement?

Which one? If the one above then no. list_empty only accesses list_head for the struct page, not the queue.

Ah, I get your point.

Quoted 2 lines
Signature · 1 line
Kirill A. Shutemov
Signature · 2 lines
Wei Yang
Help you, Help me
maintainer
#9 lore raw
In reply to Kirill A. Shutemov
Quoted 18 lines · Kirill A. Shutemov
Quoted 5 lines · Wei Yang

As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.

For example, the potential race would be:

Quoted 11 lines

CPU1 CPU2
mem_cgroup_move_account split_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock

Quoted 4 lines

I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:

I am confused, if the above race is not possible then what would be a real race? We really do not want to have a patch with a misleading changelog, do we?

Signature · 2 lines
Michal Hocko
SUSE Labs
#10 lore raw
Quoted 19 lines · Michal Hocko
Quoted 18 lines · Kirill A. Shutemov
Quoted 16 lines · Wei Yang

As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.

For example, the potential race would be:

CPU1 CPU2
mem_cgroup_move_account split_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock

Quoted 8 lines
Quoted 4 lines

I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:

I am confused, if the above race is not possible then what would be a real race? We really do not want to have a patch with a misleading changelog, do we?

The alternative is to make sure that all page_deferred_list() called with page lock taken.

I'll look into it.

Signature · 1 line
 Kirill A. Shutemov
#11 lore raw
Quoted 25 lines · Kirill A. Shutemov
Quoted 24 lines · Michal Hocko
Quoted 22 lines · Kirill A. Shutemov
Quoted 5 lines · Wei Yang

As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.

For example, the potential race would be:

Quoted 11 lines

CPU1 CPU2
mem_cgroup_move_account split_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock

I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:

Quoted 8 lines
Quoted 3 lines

I am confused, if the above race is not possible then what would be a real race? We really do not want to have a patch with a misleading changelog, do we?

The alternative is to make sure that all page_deferred_list() called with page lock taken.

I'll look into it.

split_huge_page_to_list() has page lock taken.

free_transhuge_page() is in the free path and doesn't susceptible to the race.

deferred_split_scan() is trickier. list_move() should be safe against list_empty() as it will not produce false-positive list_empty(). list_del_init() *should* (correct me if I'm wrong) be safe because the page is freeing and memcg will not touch the page anymore.

deferred_split_huge_page() is a problematic one. It called from page_remove_rmap() path witch does require page lock. I don't see any obvious way to exclude race with mem_cgroup_move_account() here. Anybody else?

Wei, could you rewrite the commit message with deferred_split_huge_page() as a race source instead of split_huge_page_to_list()?

Signature · 1 line
 Kirill A. Shutemov
#12 lore raw
Quoted 10 lines · Kirill A. Shutemov

split_huge_page_to_list() has page lock taken.

free_transhuge_page() is in the free path and doesn't susceptible to the race.

deferred_split_scan() is trickier. list_move() should be safe against list_empty() as it will not produce false-positive list_empty(). list_del_init() *should* (correct me if I'm wrong) be safe because the page is freeing and memcg will not touch the page anymore.

Quoted 8 lines

deferred_split_huge_page() is a problematic one. It called from page_remove_rmap() path witch does require page lock. I don't see any obvious way to exclude race with mem_cgroup_move_account() here. Anybody else?

Wei, could you rewrite the commit message with deferred_split_huge_page() as a race source instead of split_huge_page_to_list()?

I think describing the race in terms of deferred_split_huge_page() makes the most sense and I'd prefer a cc to stable for 5.4+. Even getting the split_queue_len, which is unsigned long, to underflow because of a list_empty(page_deferred_list()) check that is no longer accurate after the lock is taken would be a significant issue for shrinkers.

author
#13 lore raw
Quoted 21 lines · David Rientjes
Quoted 10 lines · Kirill A. Shutemov

split_huge_page_to_list() has page lock taken.

free_transhuge_page() is in the free path and doesn't susceptible to the race.

deferred_split_scan() is trickier. list_move() should be safe against list_empty() as it will not produce false-positive list_empty(). list_del_init() *should* (correct me if I'm wrong) be safe because the page is freeing and memcg will not touch the page anymore.

Quoted 8 lines

deferred_split_huge_page() is a problematic one. It called from page_remove_rmap() path witch does require page lock. I don't see any obvious way to exclude race with mem_cgroup_move_account() here. Anybody else?

Wei, could you rewrite the commit message with deferred_split_huge_page() as a race source instead of split_huge_page_to_list()?

Quoted 5 lines

I think describing the race in terms of deferred_split_huge_page() makes the most sense and I'd prefer a cc to stable for 5.4+. Even getting the split_queue_len, which is unsigned long, to underflow because of a list_empty(page_deferred_list()) check that is no longer accurate after the lock is taken would be a significant issue for shrinkers.

Oh, you are right. Even the list is not corrupted between deferred_split_scan() and mem_cgroup_move_account(), split_queue_len would be in a wrong state.

Hmm... to some extend, the page lock complicates the picture a little here, even it helps in some cases.

Signature · 2 lines
Wei Yang
Help you, Help me
author
#14 lore raw
In reply to Kirill A. Shutemov
Quoted 45 lines · Kirill A. Shutemov
Quoted 25 lines · Kirill A. Shutemov
Quoted 24 lines · Michal Hocko
Quoted 22 lines · Kirill A. Shutemov
Quoted 5 lines · Wei Yang

As all the other places, we grab the lock before manipulate the defer list. Current implementation may face a race condition.

For example, the potential race would be:

Quoted 11 lines

CPU1 CPU2
mem_cgroup_move_account split_huge_page_to_list
!list_empty
lock
!list_empty
list_del
unlock
lock
# !list_empty might not hold anymore
list_del_init
unlock

I don't think this particular race is possible. Both parties take page lock before messing with deferred queue, but anytway:

Quoted 8 lines
Quoted 3 lines

I am confused, if the above race is not possible then what would be a real race? We really do not want to have a patch with a misleading changelog, do we?

The alternative is to make sure that all page_deferred_list() called with page lock taken.

I'll look into it.

split_huge_page_to_list() has page lock taken.

free_transhuge_page() is in the free path and doesn't susceptible to the race.

deferred_split_scan() is trickier. list_move() should be safe against list_empty() as it will not produce false-positive list_empty(). list_del_init() *should* (correct me if I'm wrong) be safe because the page is freeing and memcg will not touch the page anymore.

Quoted 4 lines

deferred_split_huge_page() is a problematic one. It called from page_remove_rmap() path witch does require page lock. I don't see any obvious way to exclude race with mem_cgroup_move_account() here. Anybody else?

If my understanding is correct, the reason is deferred_split_huge_page() doesn't has page lock taken, right?

Quoted 6 lines

Wei, could you rewrite the commit message with deferred_split_huge_page() as a race source instead of split_huge_page_to_list()?

Signature · 1 line
Kirill A. Shutemov
Signature · 2 lines
Wei Yang
Help you, Help me
#15 lore raw
Quoted 10 lines · Wei Yang
Quoted 10 lines

split_huge_page_to_list() has page lock taken.

free_transhuge_page() is in the free path and doesn't susceptible to the race.

deferred_split_scan() is trickier. list_move() should be safe against list_empty() as it will not produce false-positive list_empty(). list_del_init() *should* (correct me if I'm wrong) be safe because the page is freeing and memcg will not touch the page anymore.

Quoted 8 lines
Quoted 4 lines

deferred_split_huge_page() is a problematic one. It called from page_remove_rmap() path witch does require page lock. I don't see any obvious way to exclude race with mem_cgroup_move_account() here. Anybody else?

If my understanding is correct, the reason is deferred_split_huge_page() doesn't has page lock taken, right?

I think the fix that you have proposed has inspired some deeper looks at the locking around the deferred split queue and the hope was that perhaps this could be protected by the page lock but it was found that at least in one path that isn't taken. So I believe your fix is still needed and any possible optimizations in this area can be proposed on top.