A few cleanup and fixup patches for migration
Subsystems (8)
Miaohe Lin
maintainerv4 cover A few cleanup and fixup patches for migration
Hi everyone,
This series contains a few patches to remove unneeded lock page and PageMovable check, reduce the rcu lock duration. Also we fix potential pte_unmap on an not mapped pte. More details can be found in the respective changelogs. Thanks!
---
v4:
collect Reviewed-by tag of Oscar.
fix build error reported by kernel test robot.
v3:
[PATCH 1/4] mm/migration: reduce the rcu lock duration
add the analysis that the original race condition isn't possible now alsoreduce the rcu lock duration in kernel_migrate_pages
[PATCH 2/4] mm/migration: remove unneeded lock page and PageMovable check
let free_pages_prepare() clear PG_isolated for us
[PATCH 3/4] mm/migration: return errno when isolate_huge_page failed
rename isolate_huge_page to isolate_hugetlb
[PATCH 4/4] mm/migration: fix potential pte_unmap on an not mapped pte
collect Reviewed-by tag
v2:
collect Reviewed-by tag
make isolate_huge_page consistent with isolate_lru_page
add hugetlbfs variant of hugetlb_migration_entry_wait
v1:
rebase [1] on mainline.
[1] https://lore.kernel.org/lkml/[email protected]/T/
---
Miaohe Lin (4):
mm: reduce the rcu lock duration
mm/migration: remove unneeded lock page and PageMovable check mm/migration: return errno when isolate_huge_page failed mm/migration: fix potential pte_unmap on an not mapped pte
include/linux/hugetlb.h | 6 +++--- include/linux/swapops.h | 12 ++++++++---- mm/gup.c | 2 +- mm/hugetlb.c | 15 +++++++-------- mm/memory-failure.c | 2 +- mm/memory_hotplug.c | 2 +- mm/mempolicy.c | 5 ++--- mm/migrate.c | 42 +++++++++++++++++++++++++---------------- 8 files changed, 49 insertions(+), 37 deletions(-)
Signature · 1 line
2.23.0
Miaohe Lin
maintainerPATCH v4 1/4 mm: reduce the rcu lock duration
Commit 3268c63eded4 ("mm: fix move/migrate_pages() race on task struct") extends the period of the rcu_read_lock until after the permissions checks are done to prevent the task pointed to from changing from under us. But the task_struct refcount is also taken at that time, the reference to task is guaranteed to be stable. So it's unnecessary to extend the period of the rcu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.
--- mm/mempolicy.c | 3 +-- mm/migrate.c | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-)
mm/mempolicy.c +1 -2
@@ -1609,6 +1609,7 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode, goto out; }get_task_struct (task); +rcu_read_unlock (); err = -EINVAL ;
@@ -1617,11 +1618,9 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,* Use the regular "ptrace_may_access()" checks. */ if (!ptrace_may_access (task,PTRACE_MODE_READ_REALCREDS )) { -rcu_read_unlock (); err = -EPERM ; goto out_put; } -rcu_read_unlock (); task_nodes =cpuset_mems_allowed (task);/* Is the user allowed to access the target nodes? */
mm/migrate.c +1 -2
@@ -1902,17 +1902,16 @@ static struct mm_struct *find_mm_struct(pid_t pid, nodemask_t *mem_nodes)return ERR_PTR (-ESRCH ); }get_task_struct (task); +rcu_read_unlock ();/* * Check if this process has the right to modify the specified * process. Use the regular "ptrace_may_access()" checks. */ if (!ptrace_may_access (task,PTRACE_MODE_READ_REALCREDS )) { -rcu_read_unlock (); mm =ERR_PTR (-EPERM ); goto out; } -rcu_read_unlock (); mm =ERR_PTR (security_task_movememory (task));if (IS_ERR (mm))
Signature · 1 line
2.23.0
Re: [PATCH v4 1/4] mm: reduce the rcu lock duration
Quoted 7 lines · Miaohe Lin
Commit
3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done to prevent the task pointed to from changing from under us. But thetask_structrefcount is also taken at that time, the reference to task is guaranteed to be stable. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.
Sorry for late reply, I am busy on something else recently.
I have just read the whole thread of the original patch discussion. During discussion, in
https://lore.kernel.org/lkml/[email protected]/
a patch that is same as your one is proposed. Then in the following message, Eric think that the rcu read lock should be released until permission is checked,
https://lore.kernel.org/lkml/[email protected]/
"
At the moment I suspect the permissions checks are not safe unless performed under both rcu_read_lock and task_lock to ensure that the task<->mm association does not change on us while we are working. Even with that the cred can change under us but at least we know the cred will be valid until rcu_read_unlock happens. "
So the rcu lock duration is enlarged in the following message.
https://lore.kernel.org/lkml/[email protected]/
But, after some thought, I don't think extended rcu read lock adds much value. Because after permission checking the permission may still be changed. There's no much difference.
So, I have no objection to the patch itself. But you should add more information in patch description about why the RCU proected region is extended and why we can reduce it.
Best Regards,
Huang, Ying
Quoted 74 lines
Reviewed-by Muchun Song <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Oscar Salvador <[email protected]>Signed-off-by Miaohe Lin <[email protected]>Cc Huang Ying <[email protected]>Cc David Howells <[email protected]>Cc Christoph Lameter <[email protected]>---
mm/mempolicy.c | 3 +--
mm/migrate.c | 3 +--2 files changed, 2 insertions(+), 4 deletions(-)mm/mempolicy.c +1
-0@@ -1609,6 +1609,7 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode, goto out; } get_task_struct(task); +rcu_read_unlock ();err =
-EINVAL;mm/mempolicy.c +0
-2@@ -1617,11 +1618,9 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,* Use the regular "ptrace_may_access()" checks. */ if (!ptrace_may_access(task, PTRACE_MODE_READ_REALCREDS)) { -rcu_read_unlock (); err = -EPERM; goto out_put; } -rcu_read_unlock ();
task_nodes=cpuset_mems_allowed(task);
/* Is the user allowed to access the target nodes? */mm/migrate.c +1
-0@@ -1902,17 +1902,16 @@ static struct mm_struct *find_mm_struct(pid_t pid, nodemask_t *mem_nodes) return ERR_PTR(-ESRCH); } get_task_struct(task); +rcu_read_unlock ();/*
* Check if this process has the right to modify the specified
* process. Use the regular "ptrace_may_access()" checks. */
if (!ptrace_may_access(task,PTRACE_MODE_READ_REALCREDS)) {mm/migrate.c +0
-1-rcu_read_unlock ();mm =
ERR_PTR(-EPERM);
goto out;
}mm/migrate.c +0
-1-rcu_read_unlock ();mm = ERR_PTR(security_task_movememory(task)); if (
IS_ERR(mm))
Miaohe Lin
maintainerRe: [PATCH v4 1/4] mm: reduce the rcu lock duration
Quoted 10 lines · Ying Huang
Quoted 7 lines · Miaohe Lin
Commit
3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done to prevent the task pointed to from changing from under us. But thetask_structrefcount is also taken at that time, the reference to task is guaranteed to be stable. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.Sorry for late reply, I am busy on something else recently.
That's all right. Many thanks for your hard work. :)
Quoted 24 lines
I have just read the whole thread of the original patch discussion. During discussion, in
https://lore.kernel.org/lkml/[email protected]/
a patch that is same as your one is proposed. Then in the following message, Eric think that the rcu read lock should be released until permission is checked,
https://lore.kernel.org/lkml/[email protected]/
"
At the moment I suspect the permissions checks are not safe unless performed under bothrcu_read_lockandtask_lockto ensure that the task<->mm association does not change on us while we are working. Even with that the cred can change under us but at least we know the cred will be valid untilrcu_read_unlockhappens. "So the rcu lock duration is enlarged in the following message.
Quoted 7 lines
But, after some thought, I don't think extended rcu read lock adds much value. Because after permission checking the permission may still be changed. There's no much difference.
So, I have no objection to the patch itself. But you should add more information in patch description about why the RCU proected region is extended and why we can reduce it.
Does below patch description makes sense for you?
"
Commit 3268c63eded4 ("mm: fix move/migrate_pages() race on task struct") extends the period of the rcu_read_lock until after the permissions checks are done because it suspects the permissions checks are not safe unless performed under both rcu_read_lock and task_lock to ensure the task<->mm association does not change on us while we are working [1]. But extended rcu read lock does not add much value. Because after permission checking the permission may still be changed. There's no much difference. So it's unnecessary to extend the period of the rcu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.
[1] https://lore.kernel.org/lkml/[email protected]/
"
Quoted 3 lines
Best Regards,
Huang, Ying
Thanks again!
Quoted 80 lines
Quoted 73 lines
Reviewed-by Muchun Song <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Oscar Salvador <[email protected]>Signed-off-by Miaohe Lin <[email protected]>Cc Huang Ying <[email protected]>Cc David Howells <[email protected]>Cc Christoph Lameter <[email protected]>---
mm/mempolicy.c | 3 +--
mm/migrate.c | 3 +--2 files changed, 2 insertions(+), 4 deletions(-)mm/mempolicy.c +1
-0@@ -1609,6 +1609,7 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode, goto out; } get_task_struct(task); +rcu_read_unlock ();err =
-EINVAL;mm/mempolicy.c +0
-2@@ -1617,11 +1618,9 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,* Use the regular "ptrace_may_access()" checks. */ if (!ptrace_may_access(task, PTRACE_MODE_READ_REALCREDS)) { -rcu_read_unlock (); err = -EPERM; goto out_put; } -rcu_read_unlock ();
task_nodes=cpuset_mems_allowed(task);
/* Is the user allowed to access the target nodes? */mm/migrate.c +1
-0@@ -1902,17 +1902,16 @@ static struct mm_struct *find_mm_struct(pid_t pid, nodemask_t *mem_nodes) return ERR_PTR(-ESRCH); } get_task_struct(task); +rcu_read_unlock ();/*
* Check if this process has the right to modify the specified
* process. Use the regular "ptrace_may_access()" checks. */
if (!ptrace_may_access(task,PTRACE_MODE_READ_REALCREDS)) {mm/migrate.c +0
-1-rcu_read_unlock ();mm =
ERR_PTR(-EPERM);
goto out;
}mm/migrate.c +0
-1-rcu_read_unlock ();Quoted 2 lines
mm = ERR_PTR(security_task_movememory(task)); if (
IS_ERR(mm)).
Re: [PATCH v4 1/4] mm: reduce the rcu lock duration
Quoted 60 lines · Miaohe Lin
Quoted 10 lines · Ying Huang
Quoted 7 lines · Miaohe Lin
Commit
3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done to prevent the task pointed to from changing from under us. But thetask_structrefcount is also taken at that time, the reference to task is guaranteed to be stable. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.Sorry for late reply, I am busy on something else recently.
That's all right. Many thanks for your hard work. :)
Quoted 24 lines
I have just read the whole thread of the original patch discussion. During discussion, in
https://lore.kernel.org/lkml/[email protected]/
a patch that is same as your one is proposed. Then in the following message, Eric think that the rcu read lock should be released until permission is checked,
https://lore.kernel.org/lkml/[email protected]/
"
At the moment I suspect the permissions checks are not safe unless performed under bothrcu_read_lockandtask_lockto ensure that the task<->mm association does not change on us while we are working. Even with that the cred can change under us but at least we know the cred will be valid untilrcu_read_unlockhappens. "So the rcu lock duration is enlarged in the following message.
Quoted 7 lines
But, after some thought, I don't think extended rcu read lock adds much value. Because after permission checking the permission may still be changed. There's no much difference.
So, I have no objection to the patch itself. But you should add more information in patch description about why the RCU proected region is extended and why we can reduce it.
Does below patch description makes sense for you?
"
Commit3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done because it suspects the permissions checks are not safe unless performed under bothrcu_read_lockandtask_lockto ensure the task<->mm association does not change on us while we are working [1]. But extended rcu read lock does not add much value. Because after permission checking the permission may still be changed. There's no much difference. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.
Quoted 2 lines
It doesn't make sense to me.
I don't see any sleeping functions called from find_mm_struct or kernel_migrate_pages in the area kernel_migrate_pages in the area of the code protected by get_task_struct. So at a very basic level I see a justification for dirtying a cache line twice with get_task_struct and put_task_struct to reduce rcu_read_lock hold times.
I would contend that a reasonable cleanup based up on the current state of the code would be to extend the rcu_read_lock over get_task_mm so that a reference to task_struct does not need to be taken. That has the potential to reduce contention and reduce lock hold times.
The code is missing a big fat comment with the assertion that it is ok if the permission checks are racy because the race is small, and the worst case thing that happens is the page is migrated to another numa node.
Given that the get_mm_task takes task_lock the cost of dirtying the cache line is already being paid. Perhaps not extending task_lock hold times a little bit is justified, but I haven't seen that case made.
This seems like code that is called little enough it would be better for it to be correct, and not need big fat comments explaining why it doesn't matter that they code is deliberately buggy.
In short it does not make sense to me to justify a patch for performance reasons when it appears that extending the rcu_read_lock hold time and not touch the task reference count would stop dirtying a cache line and likely have more impact.
Eric
Miaohe Lin
maintainerRe: [PATCH v4 1/4] mm: reduce the rcu lock duration
Quoted 1 line · Eric W. Biederman
Miaohe Lin <[email protected]> writes:
snip
Quoted 18 lines
Quoted 13 lines
"
Commit3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done because it suspects the permissions checks are not safe unless performed under bothrcu_read_lockandtask_lockto ensure the task<->mm association does not change on us while we are working [1]. But extended rcu read lock does not add much value. Because after permission checking the permission may still be changed. There's no much difference. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.Quoted 2 lines
It doesn't make sense to me.
Quoted 8 lines
I don't see any sleeping functions called from
find_mm_structorkernel_migrate_pagesin the areakernel_migrate_pagesin the area of the code protected byget_task_struct. So at a very basic level I see a justification for dirtying a cache line twice withget_task_structandput_task_structto reducercu_read_lockhold times.I would contend that a reasonable cleanup based up on the current state of the code would be to extend the
rcu_read_lockoverget_task_mmso
If so, security_task_movememory will be called inside rcu lock. It might call sleeping functions, e.g. smack_log(). I think it's not a good idea.
Quoted 10 lines
that a reference to
task_structdoes not need to be taken. That has the potential to reduce contention and reduce lock hold times.The code is missing a big fat comment with the assertion that it is ok if the permission checks are racy because the race is small, and the worst case thing that happens is the page is migrated to another numa node.
Quoted 8 lines
Given that the
get_mm_tasktakestask_lockthe cost of dirtying the cache line is already being paid. Perhaps not extendingtask_lockhold times a little bit is justified, but I haven't seen that case made.This seems like code that is called little enough it would be better for it to be correct, and not need big fat comments explaining why it doesn't matter that they code is deliberately buggy.
Agree. A big fat comments will make code hard to follow.
Quoted 5 lines
In short it does not make sense to me to justify a patch for performance reasons when it appears that extending the
rcu_read_lockhold time and not touch the task reference count would stop dirtying a cache line and likely have more impact.
IMHO, incremented task refcount should make code works correctly. And extending the rcu_read_lock over get_task_mm will break the things because sleeping functions might be called while holding rcu lock.
Does the patch itself makes sense for you? Should I rephase the commit log further? I'm afraid I didn't get your point correctly.
Quoted 2 lines
Eric
Thanks!
Quoted 2 lines
.
Re: [PATCH v4 1/4] mm: reduce the rcu lock duration
Quoted 27 lines · Miaohe Lin
Quoted 1 line · Eric W. Biederman
Miaohe Lin <[email protected]> writes:
snip
Quoted 24 lines
Quoted 13 lines
"
Commit3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done because it suspects the permissions checks are not safe unless performed under bothrcu_read_lockandtask_lockto ensure the task<->mm association does not change on us while we are working [1]. But extended rcu read lock does not add much value. Because after permission checking the permission may still be changed. There's no much difference. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.Quoted 2 lines
It doesn't make sense to me.
I don't see any sleeping functions called from
find_mm_structorkernel_migrate_pagesin the areakernel_migrate_pagesin the area of the code protected byget_task_struct. So at a very basic level I see a justification for dirtying a cache line twice withget_task_structandput_task_structto reducercu_read_lockhold times.
Quoted 6 lines
Quoted 2 lines
I would contend that a reasonable cleanup based up on the current state of the code would be to extend the
rcu_read_lockoverget_task_mmsoIf so,
security_task_movememorywill be called inside rcu lock. It might call sleeping functions, e.g.smack_log(). I think it's not a good idea.
In general the security functions are not allowed to sleep. The audit mechanism typically preallocates memory so it does not need to sleep. From a quick skim through the code I don't see smack_log sleeping.
Certainly the security hooks are not supposed to be inserted into the code that they prevent reasonable implementation decisions.
Which is to say if there is a good (non-security hook reason) for supporting sleeping deal with it. Otherwise the security hooks has a bug and needs to get fixed/removed.
Quoted 14 lines
Quoted 14 lines
that a reference to
task_structdoes not need to be taken. That has the potential to reduce contention and reduce lock hold times.The code is missing a big fat comment with the assertion that it is ok if the permission checks are racy because the race is small, and the worst case thing that happens is the page is migrated to another numa node.
Given that the
get_mm_tasktakestask_lockthe cost of dirtying the cache line is already being paid. Perhaps not extendingtask_lockhold times a little bit is justified, but I haven't seen that case made.
Quoted 6 lines
Quoted 4 lines
This seems like code that is called little enough it would be better for it to be correct, and not need big fat comments explaining why it doesn't matter that they code is deliberately buggy.
Agree. A big fat comments will make code hard to follow.
No.
The code is impossible to follow currently.
The code either requires a comment pointing out that it is deliberately racy, or the code needs to be fixed.
Clever and subtle code always requires a comment if for no other reason then to alert the reader that something a typical is going on.
Quoted 8 lines
Quoted 4 lines
In short it does not make sense to me to justify a patch for performance reasons when it appears that extending the
rcu_read_lockhold time and not touch the task reference count would stop dirtying a cache line and likely have more impact.IMHO, incremented task refcount should make code works correctly. And extending the
rcu_read_lockoverget_task_mmwill break the things because sleeping functions might be called while holding rcu lock.
Which sleeping functions?
I can't find any. In particular smack_task_movememory calls smk_curacc_on_task which is the same function called by security_task_getpgid. security_task_getpgid is called under rcu_read_lock. So smack won't sleep.
Quoted 2 lines
Does the patch itself makes sense for you? Should I rephase the commit log further? I'm afraid I didn't get your point correctly.
The patch makes no sense to me because I don't see it doing anything worth doing.
get/put_task_struct both dirty a cache line and are expensive especially when contended. Dirtying a cache line that is contended is the pretty much the most expensive native cpu operation. In pathological scenarios I have seen it take up to 1s. Realistically in a cache cold scenario (which is not as bad as a contended scenario) you are looking at 100ns or more just to execute get_task_struct/put_task_struct. That is the kind of cost it would be nice to avoid all together (even if the pathological case never comes up).
So I see two pieces of code where we could use the cheap operation rcu_read_lock/rcu_read_unlock to remove the expensive operation get_task_struct/put_task_struct. Instead I see people removing rcu_read_lock/rcu_read_unlock.
That makes no sense. Especially as your implied reason for making this change is to make the code have better performance. Improving performance is the reason for making critical sections smaller isn't it?
Eric
Miaohe Lin
maintainerRe: [PATCH v4 1/4] mm: reduce the rcu lock duration
Quoted 41 lines · Eric W. Biederman
Quoted 27 lines · Miaohe Lin
Quoted 1 line · Eric W. Biederman
Miaohe Lin <[email protected]> writes:
snip
Quoted 24 lines
Quoted 13 lines
"
Commit3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done because it suspects the permissions checks are not safe unless performed under bothrcu_read_lockandtask_lockto ensure the task<->mm association does not change on us while we are working [1]. But extended rcu read lock does not add much value. Because after permission checking the permission may still be changed. There's no much difference. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.Quoted 2 lines
It doesn't make sense to me.
I don't see any sleeping functions called from
find_mm_structorkernel_migrate_pagesin the areakernel_migrate_pagesin the area of the code protected byget_task_struct. So at a very basic level I see a justification for dirtying a cache line twice withget_task_structandput_task_structto reducercu_read_lockhold times.Quoted 6 lines
Quoted 2 lines
I would contend that a reasonable cleanup based up on the current state of the code would be to extend the
rcu_read_lockoverget_task_mmsoIf so,
security_task_movememorywill be called inside rcu lock. It might call sleeping functions, e.g.smack_log(). I think it's not a good idea.In general the security functions are not allowed to sleep. The audit mechanism typically preallocates memory so it does not need to sleep. From a quick skim through the code I don't see
smack_logsleeping.
Quoted 6 lines
Certainly the security hooks are not supposed to be inserted into the code that they prevent reasonable implementation decisions.
Which is to say if there is a good (non-security hook reason) for supporting sleeping deal with it. Otherwise the security hooks has a bug and needs to get fixed/removed.
I see. Many thanks for explanation.
Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
that a reference to
task_structdoes not need to be taken. That has the potential to reduce contention and reduce lock hold times.The code is missing a big fat comment with the assertion that it is ok if the permission checks are racy because the race is small, and the worst case thing that happens is the page is migrated to another numa node.
Given that the
get_mm_tasktakestask_lockthe cost of dirtying the cache line is already being paid. Perhaps not extendingtask_lockhold times a little bit is justified, but I haven't seen that case made.Quoted 6 lines
Quoted 4 lines
This seems like code that is called little enough it would be better for it to be correct, and not need big fat comments explaining why it doesn't matter that they code is deliberately buggy.
Agree. A big fat comments will make code hard to follow.
No.
Quoted 7 lines
The code is impossible to follow currently.
The code either requires a comment pointing out that it is deliberately racy, or the code needs to be fixed.
Clever and subtle code always requires a comment if for no other reason then to alert the reader that something a typical is going on.
Yes, clever and subtle code requires a comment but others might not.
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
In short it does not make sense to me to justify a patch for performance reasons when it appears that extending the
rcu_read_lockhold time and not touch the task reference count would stop dirtying a cache line and likely have more impact.IMHO, incremented task refcount should make code works correctly. And extending the
rcu_read_lockoverget_task_mmwill break the things because sleeping functions might be called while holding rcu lock.
Quoted 6 lines
Which sleeping functions?
I can't find any. In particular
smack_task_movememorycallssmk_curacc_on_taskwhich is the same function called bysecurity_task_getpgid.security_task_getpgidis called underrcu_read_lock. So smack won't sleep.
Sorry, I didn't take a close look at smack_log code. So I thought it could sleep.
Quoted 16 lines
Quoted 3 lines
Does the patch itself makes sense for you? Should I rephase the commit log further? I'm afraid I didn't get your point correctly.
The patch makes no sense to me because I don't see it doing anything worth doing.
get/put_task_struct both dirty a cache line and are expensive especially when contended. Dirtying a cache line that is contended is the pretty much the most expensive native cpu operation. In pathological scenarios I have seen it take up to 1s. Realistically in a cache cold scenario (which is not as bad as a contended scenario) you are looking at 100ns or more just to execute get_task_struct/put_task_struct. That is the kind of cost it would be nice to avoid all together (even if the pathological case never comes up).
Quoted 9 lines
So I see two pieces of code where we could use the cheap operation rcu_read_lock/rcu_read_unlock to remove the expensive operation get_task_struct/put_task_struct. Instead I see people removing rcu_read_lock/rcu_read_unlock.
That makes no sense. Especially as your implied reason for making this change is to make the code have better performance. Improving performance is the reason for making critical sections smaller isn't it?
I think you're right. We should extend the rcu_read_lock over get_task_mm so we can remove the expensive operation get_task_struct/put_task_struct and thus avoid possible cacheline penalty. But is the extended rcu lock enough to ensure the task reference is stable when calling security check functions and performing cpuset node validation? Or this race is just OK and can be left alone with a comment?
Quoted 1 line
Eric
Many thanks for your comment and suggestion!
Quoted 2 lines
.
Re: [PATCH v4 1/4] mm: reduce the rcu lock duration
Quoted 130 lines · Miaohe Lin
Quoted 41 lines · Eric W. Biederman
Quoted 27 lines · Miaohe Lin
Quoted 1 line · Eric W. Biederman
Miaohe Lin <[email protected]> writes:
snip
Quoted 24 lines
Quoted 13 lines
"
Commit3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done because it suspects the permissions checks are not safe unless performed under bothrcu_read_lockandtask_lockto ensure the task<->mm association does not change on us while we are working [1]. But extended rcu read lock does not add much value. Because after permission checking the permission may still be changed. There's no much difference. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.Quoted 2 lines
It doesn't make sense to me.
I don't see any sleeping functions called from
find_mm_structorkernel_migrate_pagesin the areakernel_migrate_pagesin the area of the code protected byget_task_struct. So at a very basic level I see a justification for dirtying a cache line twice withget_task_structandput_task_structto reducercu_read_lockhold times.Quoted 6 lines
Quoted 2 lines
I would contend that a reasonable cleanup based up on the current state of the code would be to extend the
rcu_read_lockoverget_task_mmsoIf so,
security_task_movememorywill be called inside rcu lock. It might call sleeping functions, e.g.smack_log(). I think it's not a good idea.In general the security functions are not allowed to sleep. The audit mechanism typically preallocates memory so it does not need to sleep. From a quick skim through the code I don't see
smack_logsleeping.Quoted 6 lines
Certainly the security hooks are not supposed to be inserted into the code that they prevent reasonable implementation decisions.
Which is to say if there is a good (non-security hook reason) for supporting sleeping deal with it. Otherwise the security hooks has a bug and needs to get fixed/removed.
I see. Many thanks for explanation.
Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
that a reference to
task_structdoes not need to be taken. That has the potential to reduce contention and reduce lock hold times.The code is missing a big fat comment with the assertion that it is ok if the permission checks are racy because the race is small, and the worst case thing that happens is the page is migrated to another numa node.
Given that the
get_mm_tasktakestask_lockthe cost of dirtying the cache line is already being paid. Perhaps not extendingtask_lockhold times a little bit is justified, but I haven't seen that case made.Quoted 6 lines
Quoted 4 lines
This seems like code that is called little enough it would be better for it to be correct, and not need big fat comments explaining why it doesn't matter that they code is deliberately buggy.
Agree. A big fat comments will make code hard to follow.
No.
Quoted 7 lines
The code is impossible to follow currently.
The code either requires a comment pointing out that it is deliberately racy, or the code needs to be fixed.
Clever and subtle code always requires a comment if for no other reason then to alert the reader that something a typical is going on.
Yes, clever and subtle code requires a comment but others might not.
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
In short it does not make sense to me to justify a patch for performance reasons when it appears that extending the
rcu_read_lockhold time and not touch the task reference count would stop dirtying a cache line and likely have more impact.IMHO, incremented task refcount should make code works correctly. And extending the
rcu_read_lockoverget_task_mmwill break the things because sleeping functions might be called while holding rcu lock.Quoted 6 lines
Which sleeping functions?
I can't find any. In particular
smack_task_movememorycallssmk_curacc_on_taskwhich is the same function called bysecurity_task_getpgid.security_task_getpgidis called underrcu_read_lock. So smack won't sleep.Sorry, I didn't take a close look at
smack_logcode. So I thought it could sleep.Quoted 16 lines
Quoted 3 lines
Does the patch itself makes sense for you? Should I rephase the commit log further? I'm afraid I didn't get your point correctly.
The patch makes no sense to me because I don't see it doing anything worth doing.
get/put_task_struct both dirty a cache line and are expensive especially when contended. Dirtying a cache line that is contended is the pretty much the most expensive native cpu operation. In pathological scenarios I have seen it take up to 1s. Realistically in a cache cold scenario (which is not as bad as a contended scenario) you are looking at 100ns or more just to execute get_task_struct/put_task_struct. That is the kind of cost it would be nice to avoid all together (even if the pathological case never comes up).
Quoted 9 lines
So I see two pieces of code where we could use the cheap operation rcu_read_lock/rcu_read_unlock to remove the expensive operation get_task_struct/put_task_struct. Instead I see people removing rcu_read_lock/rcu_read_unlock.
That makes no sense. Especially as your implied reason for making this change is to make the code have better performance. Improving performance is the reason for making critical sections smaller isn't it?
Quoted 5 lines
I think you're right. We should extend the
rcu_read_lockoverget_task_mmso we can remove the expensive operation get_task_struct/put_task_struct and thus avoid possible cacheline penalty. But is the extended rcu lock enough to ensure the task reference is stable when calling security check functions and performing cpuset node validation? Or this race is just OK and can be left alone with a comment?
The extending the rcu_read_lock is just about removing the expense of get_task_struct/put_task_struct. It can be handled separately from other issues at play in this code.
The rcu_read_lock guarantees that task does not go away. The rcu_read_lock does not guarantee that task->mm does not change.
A separate but related issue is should the task_lock in get_task_mm be extended to cover the security checks so that everything checks against the same mm.
Possibly the code could be refactored or reordered so that everything is guaranteed to check against the same mm.
If the checks are not made to guarantee they are all checking against the same mm, the code needs a comment to say that there is a tiny race. The comment should say we don't care about the tiny race because the worst that can happen is that a page is migrated to a different numa node. And so we don't care.
Eric
Miaohe Lin
maintainerRe: [PATCH v4 1/4] mm: reduce the rcu lock duration
Quoted 138 lines · Eric W. Biederman
Quoted 130 lines · Miaohe Lin
Quoted 41 lines · Eric W. Biederman
Quoted 27 lines · Miaohe Lin
Quoted 1 line · Eric W. Biederman
Miaohe Lin <[email protected]> writes:
snip
Quoted 24 lines
Quoted 13 lines
"
Commit3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done because it suspects the permissions checks are not safe unless performed under bothrcu_read_lockandtask_lockto ensure the task<->mm association does not change on us while we are working [1]. But extended rcu read lock does not add much value. Because after permission checking the permission may still be changed. There's no much difference. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.Quoted 2 lines
It doesn't make sense to me.
I don't see any sleeping functions called from
find_mm_structorkernel_migrate_pagesin the areakernel_migrate_pagesin the area of the code protected byget_task_struct. So at a very basic level I see a justification for dirtying a cache line twice withget_task_structandput_task_structto reducercu_read_lockhold times.Quoted 6 lines
Quoted 2 lines
I would contend that a reasonable cleanup based up on the current state of the code would be to extend the
rcu_read_lockoverget_task_mmsoIf so,
security_task_movememorywill be called inside rcu lock. It might call sleeping functions, e.g.smack_log(). I think it's not a good idea.In general the security functions are not allowed to sleep. The audit mechanism typically preallocates memory so it does not need to sleep. From a quick skim through the code I don't see
smack_logsleeping.Quoted 6 lines
Certainly the security hooks are not supposed to be inserted into the code that they prevent reasonable implementation decisions.
Which is to say if there is a good (non-security hook reason) for supporting sleeping deal with it. Otherwise the security hooks has a bug and needs to get fixed/removed.
I see. Many thanks for explanation.
Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
that a reference to
task_structdoes not need to be taken. That has the potential to reduce contention and reduce lock hold times.The code is missing a big fat comment with the assertion that it is ok if the permission checks are racy because the race is small, and the worst case thing that happens is the page is migrated to another numa node.
Given that the
get_mm_tasktakestask_lockthe cost of dirtying the cache line is already being paid. Perhaps not extendingtask_lockhold times a little bit is justified, but I haven't seen that case made.Quoted 6 lines
Quoted 4 lines
This seems like code that is called little enough it would be better for it to be correct, and not need big fat comments explaining why it doesn't matter that they code is deliberately buggy.
Agree. A big fat comments will make code hard to follow.
No.
Quoted 7 lines
The code is impossible to follow currently.
The code either requires a comment pointing out that it is deliberately racy, or the code needs to be fixed.
Clever and subtle code always requires a comment if for no other reason then to alert the reader that something a typical is going on.
Yes, clever and subtle code requires a comment but others might not.
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
In short it does not make sense to me to justify a patch for performance reasons when it appears that extending the
rcu_read_lockhold time and not touch the task reference count would stop dirtying a cache line and likely have more impact.IMHO, incremented task refcount should make code works correctly. And extending the
rcu_read_lockoverget_task_mmwill break the things because sleeping functions might be called while holding rcu lock.Quoted 6 lines
Which sleeping functions?
I can't find any. In particular
smack_task_movememorycallssmk_curacc_on_taskwhich is the same function called bysecurity_task_getpgid.security_task_getpgidis called underrcu_read_lock. So smack won't sleep.Sorry, I didn't take a close look at
smack_logcode. So I thought it could sleep.Quoted 16 lines
Quoted 3 lines
Does the patch itself makes sense for you? Should I rephase the commit log further? I'm afraid I didn't get your point correctly.
The patch makes no sense to me because I don't see it doing anything worth doing.
get/put_task_struct both dirty a cache line and are expensive especially when contended. Dirtying a cache line that is contended is the pretty much the most expensive native cpu operation. In pathological scenarios I have seen it take up to 1s. Realistically in a cache cold scenario (which is not as bad as a contended scenario) you are looking at 100ns or more just to execute get_task_struct/put_task_struct. That is the kind of cost it would be nice to avoid all together (even if the pathological case never comes up).
Quoted 9 lines
So I see two pieces of code where we could use the cheap operation rcu_read_lock/rcu_read_unlock to remove the expensive operation get_task_struct/put_task_struct. Instead I see people removing rcu_read_lock/rcu_read_unlock.
That makes no sense. Especially as your implied reason for making this change is to make the code have better performance. Improving performance is the reason for making critical sections smaller isn't it?
Quoted 5 lines
I think you're right. We should extend the
rcu_read_lockoverget_task_mmso we can remove the expensive operation get_task_struct/put_task_struct and thus avoid possible cacheline penalty. But is the extended rcu lock enough to ensure the task reference is stable when calling security check functions and performing cpuset node validation? Or this race is just OK and can be left alone with a comment?
Quoted 8 lines
The extending the
rcu_read_lockis just about removing the expense of get_task_struct/put_task_struct. It can be handled separately from other issues at play in this code.The
rcu_read_lockguarantees that task does not go away. Thercu_read_lockdoes not guarantee thattask->mmdoes not change.
It seems task_lock is needed in this case.
Quoted 8 lines
A separate but related issue is should the
task_lockinget_task_mmbe extended to cover the security checks so that everything checks against the same mm.Possibly the code could be refactored or reordered so that everything is guaranteed to check against the same mm.
IMHO, it might be overkill to do this.
Quoted 9 lines
If the checks are not made to guarantee they are all checking against the same mm, the code needs a comment to say that there is a tiny race. The comment should say we don't care about the tiny race because the worst that can happen is that a page is migrated to a different numa node. And so we don't care.
I tend to do this one. To make sure, is the below code change what you suggest?
mm/mempolicy.c +12 -11
@@ -1612,42 +1612,43 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,rcu_read_lock (); task = pid ?find_task_by_vpid (pid) : current;if (!task) { -rcu_read_unlock (); err = -ESRCH ; - goto out; + goto out_unlock; } -get_task_struct (task); -rcu_read_unlock (); err = -EINVAL ;/* * Check if this process has the right to modify the specified process. * Use the regular "ptrace_may_access()" checks. +* The below checks are not made to guarantee they are all checking +* against the same mm. But we don't care about such tiny race because +* the worst that can happen is that a page is migrated to a different +* numa node. And so we don't care. */ if (!ptrace_may_access (task,PTRACE_MODE_READ_REALCREDS )) { err = -EPERM ; - goto out_put; + goto out_unlock; } task_nodes =cpuset_mems_allowed (task);/* Is the user allowed to access the target nodes? */ if (!nodes_subset (*new, task_nodes) && !capable (CAP_SYS_NICE )) { err = -EPERM ; - goto out_put; + goto out_unlock; } task_nodes =cpuset_mems_allowed (current);nodes_and (*new, *new, task_nodes);if (nodes_empty (*new)) - goto out_put; + goto out_unlock; err =security_task_movememory (task);if (err) - goto out_put; + goto out_unlock; mm =get_task_mm (task); -put_task_struct (task); +rcu_read_unlock ();if (!mm) { err = -EINVAL ;
@@ -1663,8 +1664,8 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,return err; -out_put: -put_task_struct (task); +out_unlock: +rcu_read_unlock (); goto out; }
Quoted 1 line
Eric
Many thanks for your valuable suggestion!
Quoted 2 lines
.
Andrew Morton
maintainerRe: [PATCH v4 1/4] mm: reduce the rcu lock duration
Quoted 11 lines · On Tue, 7 Jun 2022 17:19:53 +0800 Miaohe Lin <[email protected]> wrote:
Quoted 9 lines
If the checks are not made to guarantee they are all checking against the same mm, the code needs a comment to say that there is a tiny race. The comment should say we don't care about the tiny race because the worst that can happen is that a page is migrated to a different numa node. And so we don't care.
I tend to do this one. To make sure, is the below code change what you suggest?
Eric went quiet.
As I understand it, any changes arising from this discussion can be done as a followup patch. So I'm planning on moving this 4-patch series into the non-rebasing mm-stable branch late next week.
Miaohe Lin
maintainerRe: [PATCH v4 1/4] mm: reduce the rcu lock duration
Quoted 12 lines · Andrew Morton
Quoted 10 lines · On Tue, 7 Jun 2022 17:19:53 +0800 Miaohe Lin <[email protected]> wrote:
Quoted 9 lines
If the checks are not made to guarantee they are all checking against the same mm, the code needs a comment to say that there is a tiny race. The comment should say we don't care about the tiny race because the worst that can happen is that a page is migrated to a different numa node. And so we don't care.
Quoted 7 lines
Quoted 1 line
I tend to do this one. To make sure, is the below code change what you suggest?
Eric went quiet.
As I understand it, any changes arising from this discussion can be done as a followup patch. So I'm planning on moving this 4-patch series into the non-rebasing mm-stable branch late next week.
I will send a followup patch when discussion is done. Thanks.
Quoted 4 lines
.
Re: [PATCH v4 1/4] mm: reduce the rcu lock duration
In reply to Miaohe LinQuoted 8 lines · Miaohe Lin
Commit
3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done to prevent the task pointed to from changing from under us. But thetask_structrefcount is also taken at that time, the reference to task is guaranteed to be stable. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.
Quoted 7 lines
Reviewed-by Muchun Song <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Oscar Salvador <[email protected]>Signed-off-by Miaohe Lin <[email protected]>Cc Huang Ying <[email protected]>Cc David Howells <[email protected]>Cc Christoph Lameter <[email protected]>
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v4 1/4] mm: reduce the rcu lock duration
In reply to Miaohe LinQuoted 7 lines · Miaohe Lin
Commit
3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done to prevent the task pointed to from changing from under us. But thetask_structrefcount is also taken at that time, the reference to task is guaranteed to be stable. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.
But why bother? You know the RCU read lock isn't a "real" lock, right?
Re: [PATCH v4 1/4] mm: reduce the rcu lock duration
Quoted 10 lines · Matthew Wilcox
Quoted 7 lines · Miaohe Lin
Commit
3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done to prevent the task pointed to from changing from under us. But thetask_structrefcount is also taken at that time, the reference to task is guaranteed to be stable. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.But why bother? You know the RCU read lock isn't a "real" lock, right?
Looking over this code some more, I think this may harm performance. ptrace_may_access() itself takes the rcu_read_lock(). So we currently have:
rcu_read_lock()rcu_read_lock();rcu_read_unlock();rcu_read_unlock();
In at least one RCU configuration, rcu_read_lock() maps to preempt_disable(). Nested preempt_disable() just bump a counter, while that counter reaching zero incurs some actual work. So nested rcu_read_lock() can be better than sequential lock/unlock/lock/unlock.
This needs far better justification.
Miaohe Lin
maintainerRe: [PATCH v4 1/4] mm: reduce the rcu lock duration
Quoted 12 lines · Matthew Wilcox
Quoted 10 lines · Matthew Wilcox
Quoted 7 lines · Miaohe Lin
Commit
3268c63eded4("mm: fix move/migrate_pages() race on task struct") extends the period of thercu_read_lockuntil after the permissions checks are done to prevent the task pointed to from changing from under us. But thetask_structrefcount is also taken at that time, the reference to task is guaranteed to be stable. So it's unnecessary to extend the period of thercu_read_lock. Release the rcu lock after task refcount is successfully grabbed to reduce the rcu holding time.But why bother? You know the RCU read lock isn't a "real" lock, right?
Quoted 8 lines
Looking over this code some more, I think this may harm performance.
ptrace_may_access()itself takes thercu_read_lock(). So we currently have:
rcu_read_lock()rcu_read_lock();rcu_read_unlock();rcu_read_unlock();
More precisely, we currently have:
rcu_read_lock()task_lock()rcu_read_lock();rcu_read_unlock();task_unlock()rcu_read_unlock();
Quoted 5 lines
In at least one RCU configuration,
rcu_read_lock()maps topreempt_disable(). Nestedpreempt_disable()just bump a counter, while that counter reaching zero incurs some actual work. So nestedrcu_read_lock()can be better than sequential lock/unlock/lock/unlock.
In this case, I agree with you.
But when task_lock is heavily contented, it might take a long time. So in this case, I think it's better to do the sequential rcu_lock+unlock to avoid long rcu lock duration. Or am I miss something?
Quoted 2 lines
This needs far better justification.
Thanks!
Quoted 3 lines
.
Miaohe Lin
maintainerPATCH v4 2/4 mm/migration: remove unneeded lock page and PageMovable check
In reply to Miaohe LinWhen non-lru movable page was freed from under us, __ClearPageMovable must have been done. So we can remove unneeded lock page and PageMovable check here. Also free_pages_prepare() will clear PG_isolated for us, so we can further remove ClearPageIsolated as suggested by David.
--- mm/migrate.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
mm/migrate.c +2 -7
@@ -1090,15 +1090,10 @@ static int unmap_and_move(new_page_t get_new_page,return -ENOSYS ;if (page_count (page) ==1 ) { -/* page was freed from under us. So we are done. */ +/* Page was freed from under us. So we are done. */ ClearPageActive (page);ClearPageUnevictable (page); -if (unlikely (__PageMovable (page))) { -lock_page (page); -if (!PageMovable (page)) -ClearPageIsolated (page); -unlock_page (page); - } +/* free_pages_prepare() will clear PG_isolated. */ goto out; }
Signature · 1 line
2.23.0
Re: [PATCH v4 2/4] mm/migration: remove unneeded lock page and PageMovable check
Quoted 24 lines · Miaohe Lin
When non-lru movable page was freed from under us,
__ClearPageMovablemust have been done. So we can remove unneeded lock page and PageMovable check here. Alsofree_pages_prepare()will clearPG_isolatedfor us, so we can further remove ClearPageIsolated as suggested by David.Signed-off-by Miaohe Lin <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Oscar Salvador <[email protected]>--- mm/migrate.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)mm/migrate.c +1
-1@@ -1090,15 +1090,10 @@ static int unmap_and_move(new_page_t get_new_page,return -ENOSYS ;if (page_count (page) ==1 ) { -/* page was freed from under us. So we are done. */ +/* Page was freed from under us. So we are done. */ ClearPageActive (page);ClearPageUnevictable (page);
mm/migrate.c static int unmap_and_move(new_page_t get_new_page, · line 1099
Quoted 10 lines
mm/migrate.c +1
-6-if (unlikely (__PageMovable (page))) { -lock_page (page); -if (!PageMovable (page)) -ClearPageIsolated (page); -unlock_page (page); - } +/* free_pages_prepare() will clear PG_isolated. */ goto out; }
Signature · 3 lines
Thanks, David / dhildenb
Miaohe Lin
maintainerPATCH v4 3/4 mm/migration: return errno when isolate_huge_page failed
In reply to Miaohe LinWe might fail to isolate huge page due to e.g. the page is under migration which cleared HPageMigratable. We should return errno in this case rather than always return 1 which could confuse the user, i.e. the caller might think all of the memory is migrated while the hugetlb page is left behind. We make the prototype of isolate_huge_page consistent with isolate_lru_page as suggested by Huang Ying and rename isolate_huge_page to isolate_hugetlb as suggested by Muchun to improve the readability.
--- include/linux/hugetlb.h | 6 +++--- mm/gup.c | 2 +- mm/hugetlb.c | 11 +++++------ mm/memory-failure.c | 2 +- mm/memory_hotplug.c | 2 +- mm/mempolicy.c | 2 +- mm/migrate.c | 7 ++++--- 7 files changed, 16 insertions(+), 16 deletions(-)
include/linux/hugetlb.h +3 -3
@@ -170,7 +170,7 @@ bool hugetlb_reserve_pages(struct inode *inode, long from, long to, vm_flags_t vm_flags);long hugetlb_unreserve_pages (struct inode *inode,long start,long end,long freed); -bool isolate_huge_page (struct page *page,struct list_head *list); +int isolate_hugetlb (struct page *page,struct list_head *list);int get_hwpoison_huge_page (struct page *page,bool *hugetlb);int get_huge_page_for_hwpoison (unsigned long pfn,int flags);void putback_active_hugepage (struct page *page);
@@ -376,9 +376,9 @@ static inline pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr,return NULL ; } -static inline bool isolate_huge_page (struct page *page,struct list_head *list) +static inline int isolate_hugetlb (struct page *page,struct list_head *list) { -return false; +return -EBUSY ; }static inline int get_hwpoison_huge_page (struct page *page,bool *hugetlb)
mm/gup.c +1 -1
@@ -1898,7 +1898,7 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages,* Try to move out any movable page before pinning the range. */ if (folio_test_hugetlb (folio)) { -if (!isolate_huge_page (&folio->page, +if (isolate_hugetlb (&folio->page, &movable_page_list)) isolation_error_count++;continue ;
mm/hugetlb.c +5 -6
@@ -2765,8 +2765,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page,* Fail with -EBUSY if not possible. */ spin_unlock_irq (&hugetlb_lock); -if (!isolate_huge_page (old_page, list)) - ret = -EBUSY ; + ret =isolate_hugetlb (old_page, list);spin_lock_irq (&hugetlb_lock); goto free_new; }else if (!HPageFreed (old_page)) {
@@ -2842,7 +2841,7 @@ int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list)if (hstate_is_gigantic (h))return -ENOMEM ; -if (page_count (head) &&isolate_huge_page (head, list)) +if (page_count (head) && !isolate_hugetlb (head, list)) ret =0 ;else if (!page_count (head)) ret =alloc_and_dissolve_huge_page (h, head, list);
@@ -6952,15 +6951,15 @@ follow_huge_pgd(struct mm_struct *mm, unsigned long address, pgd_t *pgd, int flareturn pte_page (*(pte_t *)pgd) + ((address & ~PGDIR_MASK ) >>PAGE_SHIFT ); } -bool isolate_huge_page (struct page *page,struct list_head *list) +int isolate_hugetlb (struct page *page,struct list_head *list) { -bool ret = true; +int ret =0 ;spin_lock_irq (&hugetlb_lock);if (!PageHeadHuge (page) || !HPageMigratable (page) || !get_page_unless_zero (page)) { - ret = false; + ret = -EBUSY ; goto unlock; }ClearHPageMigratable (page);
mm/memory-failure.c +1 -1
@@ -2166,7 +2166,7 @@ static bool isolate_page(struct page *page, struct list_head *pagelist)bool lru =PageLRU (page);if (PageHuge (page)) { - isolated =isolate_huge_page (page, pagelist); + isolated = !isolate_hugetlb (page, pagelist); }else {if (lru) isolated = !isolate_lru_page (page);
mm/memory_hotplug.c +1 -1
@@ -1643,7 +1643,7 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)if (PageHuge (page)) { pfn =page_to_pfn (head) +compound_nr (head) -1 ; -isolate_huge_page (head, &source); +isolate_hugetlb (head, &source);continue ; }else if (PageTransHuge (page)) pfn =page_to_pfn (head) +thp_nr_pages (page) -1 ;
mm/mempolicy.c +1 -1
@@ -602,7 +602,7 @@ static int queue_pages_hugetlb(pte_t *pte, unsigned long hmask,/* With MPOL_MF_MOVE, we migrate only unshared hugepage. */ if (flags & (MPOL_MF_MOVE_ALL ) || (flags &MPOL_MF_MOVE &&page_mapcount (page) ==1 )) { -if (!isolate_huge_page (page, qp->pagelist) && +if (isolate_hugetlb (page, qp->pagelist) && (flags &MPOL_MF_STRICT ))/* * Failed to isolate page but allow migrating pages
mm/migrate.c +4 -3
@@ -133,7 +133,7 @@ static void putback_movable_page(struct page *page)* * This function shall be used whenever the isolated pageset has been * built from lru, balloon, hugetlbfs page. See isolate_migratepages_range() -* and isolate_huge_page(). +* and isolate_hugetlb(). */ void putback_movable_pages (struct list_head *l) {
@@ -1627,8 +1627,9 @@ static int add_page_for_migration(struct mm_struct *mm, unsigned long addr,if (PageHuge (page)) {if (PageHead (page)) { -isolate_huge_page (page, pagelist); - err =1 ; + err =isolate_hugetlb (page, pagelist); +if (!err) + err =1 ; } }else {struct page *head;
Signature · 1 line
2.23.0
Miaohe Lin
maintainerPATCH v4 4/4 mm/migration: fix potential pte_unmap on an not mapped pte
In reply to Miaohe Lin__migration_entry_wait and migration_entry_wait_on_locked assume pte is always mapped from caller. But this is not the case when it's called from migration_entry_wait_huge and follow_huge_pmd. Add a hugetlbfs variant that calls hugetlb_migration_entry_wait(ptep == NULL) to fix this issue.
--- include/linux/swapops.h | 12 ++++++++---- mm/hugetlb.c | 4 ++-- mm/migrate.c | 23 +++++++++++++++++++---- 3 files changed, 29 insertions(+), 10 deletions(-)
include/linux/swapops.h +8 -4
@@ -244,8 +244,10 @@ extern void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, spinlock_t *ptl);extern void migration_entry_wait (struct mm_struct *mm,pmd_t *pmd,unsigned long address); -extern void migration_entry_wait_huge (struct vm_area_struct *vma, -struct mm_struct *mm,pte_t *pte); +#ifdef CONFIG_HUGETLB_PAGE +extern void __migration_entry_wait_huge (pte_t *ptep,spinlock_t *ptl); +extern void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte); +#endif #else static inline swp_entry_t make_readable_migration_entry (pgoff_t offset) {
@@ -271,8 +273,10 @@ static inline void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, spinlock_t *ptl) { }static inline void migration_entry_wait (struct mm_struct *mm,pmd_t *pmd,unsigned long address) { } -static inline void migration_entry_wait_huge (struct vm_area_struct *vma, -struct mm_struct *mm,pte_t *pte) { } +#ifdef CONFIG_HUGETLB_PAGE +static inline void __migration_entry_wait_huge (pte_t *ptep,spinlock_t *ptl) { } +static inline void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte) { } +#endif static inline int is_writable_migration_entry (swp_entry_t entry) {return 0 ;
mm/hugetlb.c +2 -2
@@ -5694,7 +5694,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,*/ entry =huge_ptep_get (ptep);if (unlikely (is_hugetlb_entry_migration (entry))) { -migration_entry_wait_huge (vma, mm, ptep); +migration_entry_wait_huge (vma, ptep);return 0 ; }else if (unlikely (is_hugetlb_entry_hwpoisoned (entry))) return VM_FAULT_HWPOISON_LARGE |
@@ -6919,7 +6919,7 @@ follow_huge_pmd(struct mm_struct *mm, unsigned long address, }else {if (is_hugetlb_entry_migration (pte)) {spin_unlock (ptl); -__migration_entry_wait (mm, (pte_t *)pmd, ptl); +__migration_entry_wait_huge ((pte_t *)pmd, ptl); goto retry; }/*
mm/migrate.c +19 -4
@@ -315,13 +315,28 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd,__migration_entry_wait (mm, ptep, ptl); } -void migration_entry_wait_huge (struct vm_area_struct *vma, -struct mm_struct *mm,pte_t *pte) +#ifdef CONFIG_HUGETLB_PAGE +void __migration_entry_wait_huge (pte_t *ptep,spinlock_t *ptl) { -spinlock_t *ptl =huge_pte_lockptr (hstate_vma (vma), mm, pte); -__migration_entry_wait (mm, pte, ptl); +pte_t pte; + +spin_lock (ptl); + pte =huge_ptep_get (ptep); + +if (unlikely (!is_hugetlb_entry_migration (pte))) +spin_unlock (ptl); +else +migration_entry_wait_on_locked (pte_to_swp_entry (pte),NULL , ptl); } +void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte) +{ +spinlock_t *ptl =huge_pte_lockptr (hstate_vma (vma), vma->vm_mm, pte); + +__migration_entry_wait_huge (pte, ptl); +} +#endif +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION void pmd_migration_entry_wait (struct mm_struct *mm,pmd_t *pmd) {
Signature · 1 line
2.23.0