mm: cachestat: avoid bogus workingset test during swapping & invalidation races
Subsystems (2)
PATCH 1/1 mm: cachestat: avoid bogus workingset test during swapping & invalidation races
When cachestat against shmem races with swapping and invalidation, the shadow entry might not exist: swapout IO is still in progress and we're before __remove_mapping; or swapin/invalidation/swapoff has removed the shadow from swapcache after we saw a shmem swap entry.
This will send a NULL to workingset_test_recent(). The latter purely operates on pointer bits, so it won't crash - node 0, memcg ID 0, eviction timestamp 0, etc. are all valid inputs - but it's a bogus test. In theory that could result in a false "recently evicted" count.
Such a false positive wouldn't be the end of the world. But for code clarity and (future) robustness, be explicit about this case.
--- mm/filemap.c | 3 +++ 1 file changed, 3 insertions(+)
mm/filemap.c +3 -0
@@ -4199,6 +4199,9 @@ static void filemap_cachestat(struct address_space *mapping,swp_entry_t swp =radix_to_swp_entry (folio); shadow =get_shadow_from_swap_cache (swp); +/* can race with swapping & invalidation */ +if (!shadow) + goto resched; }#endif if (workingset_test_recent (shadow, true, &workingset))
Signature · 1 line
2.44.0
Quoted 9 lines · Johannes Weiner
When cachestat against shmem races with swapping and invalidation, the shadow entry might not exist: swapout IO is still in progress and we're before
__remove_mapping; or swapin/invalidation/swapoff has removed the shadow from swapcache after we saw a shmem swap entry.This will send a NULL to
workingset_test_recent(). The latter purely operates on pointer bits, so it won't crash - node 0, memcg ID 0, eviction timestamp 0, etc. are all valid inputs - but it's a bogus test. In theory that could result in a false "recently evicted" count.
Good catch!
Quoted 16 lines
Such a false positive wouldn't be the end of the world. But for code clarity and (future) robustness, be explicit about this case.
Fixes cf264e1329fb ("cachestat: implement cachestat syscall")Reported-by Jann Horn <[email protected]>Signed-off-by Johannes Weiner <[email protected]>--- mm/filemap.c | 3 +++ 1 file changed, 3 insertions(+)mm/filemap.c +0
-0
mm/filemap.c static void filemap_cachestat(struct address_space *mapping, · line 4200
Quoted 2 lines
mm/filemap.c +0
-0@@ -4199,6 +4199,9 @@ static void filemap_cachestat(struct address_space *mapping,swp_entry_t swp =radix_to_swp_entry (folio);
IIUC, we should first check if it's a real swap entry using non_swap_entry(), right? Since there maybe other types of entries in shmem. And need to get_swap_device() to prevent concurrent swapoff here, get_shadow_from_swap_cache() won't do it for us.
Thanks.
Quoted 7 lines
mm/filemap.c +3
-0shadow =get_shadow_from_swap_cache (swp); +/* can race with swapping & invalidation */ +if (!shadow) + goto resched; }#endif if (workingset_test_recent (shadow, true, &workingset))
Quoted 24 lines · Chengming Zhou
Quoted 9 lines · Johannes Weiner
When cachestat against shmem races with swapping and invalidation, the shadow entry might not exist: swapout IO is still in progress and we're before
__remove_mapping; or swapin/invalidation/swapoff has removed the shadow from swapcache after we saw a shmem swap entry.This will send a NULL to
workingset_test_recent(). The latter purely operates on pointer bits, so it won't crash - node 0, memcg ID 0, eviction timestamp 0, etc. are all valid inputs - but it's a bogus test. In theory that could result in a false "recently evicted" count.Good catch!
Quoted 11 lines
Such a false positive wouldn't be the end of the world. But for code clarity and (future) robustness, be explicit about this case.
Fixes cf264e1329fb ("cachestat: implement cachestat syscall")Reported-by Jann Horn <[email protected]>Signed-off-by Johannes Weiner <[email protected]>--- mm/filemap.c | 3 +++ 1 file changed, 3 insertions(+)
Quoted 10 lines
mm/filemap.c static void filemap_cachestat(struct address_space *mapping, · line 4200
Quoted 7 lines
mm/filemap.c +0
-0@@ -4199,6 +4199,9 @@ static void filemap_cachestat(struct address_space *mapping,swp_entry_t swp =radix_to_swp_entry (folio);IIUC, we should first check if it's a real swap entry using
non_swap_entry(), right? Since there maybe other types of entries in shmem.
Good point, it could be a poisoned entry. I'll add the non_swap_entry() check on swp.
Quoted 2 lines
And need to
get_swap_device()to prevent concurrent swapoff here,get_shadow_from_swap_cache()won't do it for us.
We're holding rcu_read_lock() for the xarray iteration, so if we see the swap entry in the shmem mapping, it means we beat shmem_unuse() and swapoff hasn't run synchronize_rcu() yet.
So it's safe. But I think it could use a comment. Maybe the documentation of get_swap_device() should mention this option too?
Quoted 39 lines · Johannes Weiner
Quoted 24 lines · Chengming Zhou
Quoted 9 lines · Johannes Weiner
When cachestat against shmem races with swapping and invalidation, the shadow entry might not exist: swapout IO is still in progress and we're before
__remove_mapping; or swapin/invalidation/swapoff has removed the shadow from swapcache after we saw a shmem swap entry.This will send a NULL to
workingset_test_recent(). The latter purely operates on pointer bits, so it won't crash - node 0, memcg ID 0, eviction timestamp 0, etc. are all valid inputs - but it's a bogus test. In theory that could result in a false "recently evicted" count.Good catch!
Quoted 11 lines
Such a false positive wouldn't be the end of the world. But for code clarity and (future) robustness, be explicit about this case.
Fixes cf264e1329fb ("cachestat: implement cachestat syscall")Reported-by Jann Horn <[email protected]>Signed-off-by Johannes Weiner <[email protected]>--- mm/filemap.c | 3 +++ 1 file changed, 3 insertions(+)Quoted 10 lines
mm/filemap.c static void filemap_cachestat(struct address_space *mapping, · line 4200
Quoted 7 lines
mm/filemap.c +0
-0@@ -4199,6 +4199,9 @@ static void filemap_cachestat(struct address_space *mapping,swp_entry_t swp =radix_to_swp_entry (folio);IIUC, we should first check if it's a real swap entry using
non_swap_entry(), right? Since there maybe other types of entries in shmem.Good point, it could be a poisoned entry. I'll add the
non_swap_entry()check on swp.
Quoted 6 lines
Quoted 2 lines
And need to
get_swap_device()to prevent concurrent swapoff here,get_shadow_from_swap_cache()won't do it for us.We're holding
rcu_read_lock()for the xarray iteration, so if we see the swap entry in the shmem mapping, it means we beatshmem_unuse()and swapoff hasn't runsynchronize_rcu()yet.
Ah, you are right, so it's safe.
Quoted 3 lines
So it's safe. But I think it could use a comment. Maybe the documentation of
get_swap_device()should mention this option too?
PATCH 1/1 mm: cachestat: fix two shmem bugs
When cachestat on shmem races with swapping and invalidation, there are two possible bugs:
1) A swapin error can have resulted in a poisoned swap entry in the shmem inode's xarray. Calling get_shadow_from_swap_cache() on it will result in an out-of-bounds access to swapper_spaces[].
Validate the entry with non_swap_entry() before going further.
2) When we find a valid swap entry in the shmem's inode, the shadow entry in the swapcache might not exist yet: swap IO is still in progress and we're before __remove_mapping; swapin, invalidation, or swapoff have removed the shadow from swapcache after we saw the shmem swap entry.
This will send a NULL to workingset_test_recent(). The latter purely operates on pointer bits, so it won't crash - node 0, memcg ID 0, eviction timestamp 0, etc. are all valid inputs - but it's a bogus test. In theory that could result in a false "recently evicted" count.
Such a false positive wouldn't be the end of the world. But for code clarity and (future) robustness, be explicit about this case.
Bail on get_shadow_from_swap_cache() returning NULL.
--- mm/filemap.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
mm/filemap.c +16 -0
@@ -4198,7 +4198,23 @@ static void filemap_cachestat(struct address_space *mapping,/* shmem file - in swap cache */ swp_entry_t swp =radix_to_swp_entry (folio); +/* swapin error results in poisoned entry */ +if (non_swap_entry (swp)) + goto resched; + +/* +* Getting a swap entry from the shmem +* inode means we beat +* shmem_unuse(). rcu_read_lock() +* ensures swapoff waits for us before +* freeing the swapper space. However, +* we can race with swapping and +* invalidation, so there might not be +* a shadow in the swapcache (yet). +*/ shadow =get_shadow_from_swap_cache (swp); +if (!shadow) + goto resched; }#endif if (workingset_test_recent (shadow, true, &workingset))
Signature · 1 line
2.44.0
Re: [PATCH] mm: cachestat: fix two shmem bugs
Quoted 24 lines · Johannes Weiner
When cachestat on shmem races with swapping and invalidation, there are two possible bugs:
1) A swapin error can have resulted in a poisoned swap entry in the shmem inode's xarray. Calling
get_shadow_from_swap_cache()on it will result in an out-of-bounds access to swapper_spaces[].Validate the entry with
non_swap_entry()before going further.2) When we find a valid swap entry in the shmem's inode, the shadow entry in the swapcache might not exist yet: swap IO is still in progress and we're before
__remove_mapping; swapin, invalidation, or swapoff have removed the shadow from swapcache after we saw the shmem swap entry.This will send a NULL to
workingset_test_recent(). The latter purely operates on pointer bits, so it won't crash - node 0, memcg ID 0, eviction timestamp 0, etc. are all valid inputs - but it's a bogus test. In theory that could result in a false "recently evicted" count.Such a false positive wouldn't be the end of the world. But for code clarity and (future) robustness, be explicit about this case.
Quoted 7 lines
Bail on
get_shadow_from_swap_cache()returning NULL.Fixes cf264e1329fb ("cachestat: implement cachestat syscall")Cc [email protected] [v6.5+]Reported-by Chengming Zhou <[email protected]> [Bug #1]Reported-by Jann Horn <[email protected]> [Bug #2]Signed-off-by Johannes Weiner <[email protected]>
Looks good to me.
Thanks.
Quoted 32 lines
--- mm/filemap.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)mm/filemap.c +16
-0@@ -4198,7 +4198,23 @@ static void filemap_cachestat(struct address_space *mapping,/* shmem file - in swap cache */ swp_entry_t swp =radix_to_swp_entry (folio); +/* swapin error results in poisoned entry */ +if (non_swap_entry (swp)) + goto resched; + +/* +* Getting a swap entry from the shmem +* inode means we beat +* shmem_unuse(). rcu_read_lock() +* ensures swapoff waits for us before +* freeing the swapper space. However, +* we can race with swapping and +* invalidation, so there might not be +* a shadow in the swapcache (yet). +*/ shadow =get_shadow_from_swap_cache (swp); +if (!shadow) + goto resched; }#endif if (workingset_test_recent (shadow, true, &workingset))
Re: [PATCH] mm: cachestat: fix two shmem bugs
In reply to Johannes WeinerQuoted 25 lines · Johannes Weiner
When cachestat on shmem races with swapping and invalidation, there are two possible bugs:
1) A swapin error can have resulted in a poisoned swap entry in the shmem inode's xarray. Calling
get_shadow_from_swap_cache()on it will result in an out-of-bounds access to swapper_spaces[].Validate the entry with
non_swap_entry()before going further.2) When we find a valid swap entry in the shmem's inode, the shadow entry in the swapcache might not exist yet: swap IO is still in progress and we're before
__remove_mapping; swapin, invalidation, or swapoff have removed the shadow from swapcache after we saw the shmem swap entry.This will send a NULL to
workingset_test_recent(). The latter purely operates on pointer bits, so it won't crash - node 0, memcg ID 0, eviction timestamp 0, etc. are all valid inputs - but it's a bogus test. In theory that could result in a false "recently evicted" count.Such a false positive wouldn't be the end of the world. But for code clarity and (future) robustness, be explicit about this case.
Quoted 7 lines
Bail on
get_shadow_from_swap_cache()returning NULL.Fixes cf264e1329fb ("cachestat: implement cachestat syscall")Cc: [email protected] [v6.5+] Reported-by: Chengming Zhou <[email protected]> [Bug #1] Reported-by: Jann Horn <[email protected]> [Bug #2]Signed-off-by Johannes Weiner <[email protected]>
Nice catch! Thanks for the report, Chengming and Jann, and thanks for the fix, Johannes!
Quoted 35 lines
--- mm/filemap.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)mm/filemap.c +16
-1@@ -4198,7 +4198,23 @@ static void filemap_cachestat(struct address_space *mapping,/* shmem file - in swap cache */ swp_entry_t swp =radix_to_swp_entry (folio); +/* swapin error results in poisoned entry */ +if (non_swap_entry (swp)) + goto resched; + +/* +* Getting a swap entry from the shmem +* inode means we beat +* shmem_unuse(). rcu_read_lock() +* ensures swapoff waits for us before +* freeing the swapper space. However, +* we can race with swapping and +* invalidation, so there might not be +* a shadow in the swapcache (yet). +*/ shadow =get_shadow_from_swap_cache (swp); +if (!shadow) + goto resched; }#endif if (workingset_test_recent (shadow, true, &workingset)) --2.44.0
Re: [PATCH] mm: cachestat: fix two shmem bugs
In reply to Johannes Weiner💖
Nhat Pham reacted via Gmail
<https://www.google.com/gmail/about/?utm_source=gmail-in-product&utm_medium=et&utm_campaign=emojireactionemail#app>
{
"emoji": "💖",
"version": 1
}