mm: fix zswap writeback race condition
Subsystems (2)
PATCH v2 1/1 mm: fix zswap writeback race condition
The zswap writeback mechanism can cause a race condition resulting in memory corruption, where a swapped out page gets swapped in with data that was written to a different page.
The race unfolds like this:
1. a page with data A and swap offset X is stored in zswap
2. page A is removed off the LRU by zpool driver for writeback in zswap-shrink work, data for A is mapped by zpool driver
3. user space program faults and invalidates page entry A, offset X is considered free
4. kswapd stores page B at offset X in zswap (zswap could also be full, if so, page B would then be IOed to X, then skip step 5.)
5. entry A is replaced by B in tree->rbroot, this doesn't affect the local reference held by zswap-shrink work
6. zswap-shrink work writes back A at X, and frees zswap entry A
7. swapin of slot X brings A in memory instead of B
The fix:
Once the swap page cache has been allocated (case ZSWAP_SWAPCACHE_NEW), zswap-shrink work just checks that the local zswap_entry reference is still the same as the one in the tree. If it's not the same it means that it's either been invalidated or replaced, in both cases the writeback is aborted because the local entry contains stale data.
Reproducer:
I originally found this by running stress overnight to validate my work on the zswap writeback mechanism, it manifested after hours on my test machine. The key to make it happen is having zswap writebacks, so whatever setup pumps /sys/kernel/debug/zswap/written_back_pages should do the trick.
In order to reproduce this faster on a vm, I setup a system with ~100M of available memory and a 500M swap file, then running stress --vm 1 --vm-bytes 300000000 --vm-stride 4000 makes it happen in matter of tens of minutes. One can speed things up even more by swinging /sys/module/zswap/parameters/max_pool_percent up and down between, say, 20 and 1; this makes it reproduce in tens of seconds. It's crucial to set --vm-stride to something other than 4096 otherwise stress won't realize that memory has been corrupted because all pages would have the same data.
V2:
- updated comment with better explaination of the situation being addressed in the check
--- mm/zswap.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
mm/zswap.c +16 -0
@@ -995,6 +995,22 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) goto fail;case ZSWAP_SWAPCACHE_NEW :/* page is locked */ +/* +* Having a local reference to the zswap entry doesn't exclude +* swapping from invalidating and recycling the swap slot. Once +* the swapcache is secured against concurrent swapping to and +* from the slot, recheck that the entry is still current before +* writing. +*/ +spin_lock (&tree->lock); +if (zswap_rb_search (&tree->rbroot, entry->offset) != entry) { +spin_unlock (&tree->lock); +delete_from_swap_cache (page_folio (page)); + ret = -ENOMEM ; + goto fail; + } +spin_unlock (&tree->lock); +/* decompress */ acomp_ctx =raw_cpu_ptr (entry->pool->acomp_ctx); dlen =PAGE_SIZE ;
Signature · 1 line
2.34.1
Hi Domenico,
Quoted 11 lines · Domenico Cerasuolo
1. a page with data A and swap offset X is stored in zswap
2. page A is removed off the LRU by zpool driver for writeback in zswap-shrink work, data for A is mapped by zpool driver
3. user space program faults and invalidates page entry A, offset X is considered free
4. kswapd stores page B at offset X in zswap (zswap could also be full, if so, page B would then be IOed to X, then skip step 5.)
5. entry A is replaced by B intree->rbroot, this doesn't affect the local reference held by zswap-shrink work
6. zswap-shrink work writes back A at X, and frees zswap entry A
7. swapin of slot X brings A in memory instead of B
Thanks for the interesting discovery.
Quoted 24 lines
V2:
- updated comment with better explaination of the situation being addressed in the checkSigned-off-by Domenico Cerasuolo <[email protected]>Acked-by Johannes Weiner <[email protected]>--- mm/zswap.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)mm/zswap.c +6
-0@@ -995,6 +995,22 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) goto fail;case ZSWAP_SWAPCACHE_NEW :/* page is locked */ +/* +* Having a local reference to the zswap entry doesn't exclude +* swapping from invalidating and recycling the swap slot. Once +* the swapcache is secured against concurrent swapping to and +* from the slot, recheck that the entry is still current before +* writing.
mm/zswap.c static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) · line 1013
Quoted 10 lines
mm/zswap.c +10
-0+*/ +spin_lock (&tree->lock); +if (zswap_rb_search (&tree->rbroot, entry->offset) != entry) { +spin_unlock (&tree->lock); +delete_from_swap_cache (page_folio (page)); + ret = -ENOMEM ; + goto fail; + } +spin_unlock (&tree->lock); +
The race condition is still there, just making it much harder to hit. What happens after you perform the rb tree search, release tree lock. Then the entry gets invalid and recycled right here before the decompress step?
mm/zswap.c static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) · line 1016
Quoted 4 lines
mm/zswap.c +0
-0/* decompress */ acomp_ctx =raw_cpu_ptr (entry->pool->acomp_ctx); dlen =PAGE_SIZE ;
Chris
Johannes Weiner
maintainerQuoted 27 lines · Chris Li
Hi Domenico,
Quoted 11 lines · Domenico Cerasuolo
1. a page with data A and swap offset X is stored in zswap
2. page A is removed off the LRU by zpool driver for writeback in zswap-shrink work, data for A is mapped by zpool driver
3. user space program faults and invalidates page entry A, offset X is considered free
4. kswapd stores page B at offset X in zswap (zswap could also be full, if so, page B would then be IOed to X, then skip step 5.)
5. entry A is replaced by B intree->rbroot, this doesn't affect the local reference held by zswap-shrink work
6. zswap-shrink work writes back A at X, and frees zswap entry A
7. swapin of slot X brings A in memory instead of BThanks for the interesting discovery.
Quoted 10 lines
V2:
- updated comment with better explaination of the situation being addressed in the checkSigned-off-by Domenico Cerasuolo <[email protected]>Acked-by Johannes Weiner <[email protected]>--- mm/zswap.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
Quoted 8 lines
mm/zswap.c static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) · line 997
Quoted 8 lines
mm/zswap.c +0
-0@@ -995,6 +995,22 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) goto fail;case ZSWAP_SWAPCACHE_NEW :/* page is locked */
^^^^^^^^^^^^^^^^^^^^
Quoted 13 lines
Quoted 13 lines
mm/zswap.c +13
-0+/* +* Having a local reference to the zswap entry doesn't exclude +* swapping from invalidating and recycling the swap slot. Once +* the swapcache is secured against concurrent swapping to and +* from the slot, recheck that the entry is still current before +* writing. +*/ +spin_lock (&tree->lock); +if (zswap_rb_search (&tree->rbroot, entry->offset) != entry) { +spin_unlock (&tree->lock); +delete_from_swap_cache (page_folio (page)); + ret = -ENOMEM ; + goto fail;
Quoted 8 lines
mm/zswap.c static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) · line 1013
Quoted 3 lines
mm/zswap.c +3
-0+ } +spin_unlock (&tree->lock); +The race condition is still there, just making it much harder to hit. What happens after you perform the rb tree search, release tree lock. Then the entry gets invalid and recycled right here before the decompress step?
Recyling can only happen up until we see ZSWAP_SWAPCACHE_NEW.
Once we see it, we're holding the page lock* on a new swapcache page for a valid, in-use** swp_entry_t.
The lock of the swapcache page prevents swapin, which would be required for the count to drop and the entry to be recycled.
__read_swap_cache_async() checked that the entry is valid, so the slot cannot be allocated to someone else.
Now we just have to check if that entry is the right one, iow the slot wasn't recycled.
If the slot wasn't recycled, we know we have the right data and we can start the IO and unlock the page. (After that swapins can continue and the data can change, but regular writeback vs redirtying rules apply.)
If the slot was indeed recycled before we get ZSWAP_SWAPCACHE_NEW, we see the mismatch, delete the page from the swapcache and unlock it. A racing do_swap_page() may have found and reffed the page in swapcache, and acquire the page lock after us; but it'll see it's no longer in the swapcache, drop the reference (free the page) and retry the fault.
Chris Li
Reviewed-byQuoted 27 lines · Johannes Weiner
Quoted 2 lines
Quoted 2 lines
case
ZSWAP_SWAPCACHE_NEW: /* page is locked */^^^^^^^^^^^^^^^^^^^^Quoted 13 lines
Quoted 13 lines
mm/zswap.c +13
-0+/* +* Having a local reference to the zswap entry doesn't exclude +* swapping from invalidating and recycling the swap slot. Once +* the swapcache is secured against concurrent swapping to and +* from the slot, recheck that the entry is still current before +* writing. +*/ +spin_lock (&tree->lock); +if (zswap_rb_search (&tree->rbroot, entry->offset) != entry) { +spin_unlock (&tree->lock); +delete_from_swap_cache (page_folio (page)); + ret = -ENOMEM ; + goto fail;Quoted 8 lines
mm/zswap.c static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) · line 1013
Quoted 3 lines
mm/zswap.c +3
-0+ } +spin_unlock (&tree->lock); +The race condition is still there, just making it much harder to hit. What happens after you perform the rb tree search, release tree lock. Then the entry gets invalid and recycled right here before the decompress step?
Quoted 7 lines
Recyling can only happen up until we see
ZSWAP_SWAPCACHE_NEW.Once we see it, we're holding the page lock* on a new swapcache page for a valid, in-use**
swp_entry_t.The lock of the swapcache page prevents swapin, which would be required for the count to drop and the entry to be recycled.
Thanks for the explain. I miss the locked page will prevent swapin part.
Quoted 10 lines
__read_swap_cache_async()checked that the entry is valid, so the slot cannot be allocated to someone else.Now we just have to check if that entry is the right one, iow the slot wasn't recycled.
If the slot wasn't recycled, we know we have the right data and we can start the IO and unlock the page. (After that swapins can continue and the data can change, but regular writeback vs redirtying rules apply.)
Quoted 5 lines
If the slot was indeed recycled before we get
ZSWAP_SWAPCACHE_NEW, we see the mismatch, delete the page from the swapcache and unlock it. A racingdo_swap_page()may have found and reffed the page in swapcache, and acquire the page lock after us; but it'll see it's no longer in the swapcache, drop the reference (free the page) and retry the fault.
LGTM then. Please feel free to add:
Chris