mm, swap: fix race between swapoff and some swap operations
Subsystems (3)
PATCH 1/1 mm, swap: fix race between swapoff and some swap operations
From: Huang Ying <[email protected]>
When swapin is performed, after getting the swap entry information from the page table, system will swap in the swap entry, without any lock held to prevent the swap device from being swapoff. This may cause the race like below,
CPU 1 CPU 2
----- -----
do_swap_page
swapin_readahead
__read_swap_cache_async
swapoff swapcache_prepare
p->swap_map = NULL __swap_duplicate
p->swap_map[?] /* !!! NULL pointer access */
Because swapoff is usually done when system shutdown only, the race may not hit many people in practice. But it is still a race need to be fixed.
To fix the race, get_swap_device() is added to check whether the specified swap entry is valid in its swap device. If so, it will keep the swap entry valid via preventing the swap device from being swapoff, until put_swap_device() is called.
Because swapoff() is very rare code path, to make the normal path runs as fast as possible, rcu_read_lock/unlock() and synchronize_rcu() instead of reference count is used to implement get/put_swap_device(). From get_swap_device() to put_swap_device(), RCU reader side is locked, so synchronize_rcu() in swapoff() will wait until put_swap_device() is called.
In addition to swap_map, cluster_info, etc. data structure in the struct swap_info_struct, the swap cache radix tree will be freed after swapoff, so this patch fixes the race between swap cache looking up and swapoff too.
Races between some other swap cache usages and swapoff are fixed too via calling synchronize_rcu() between clearing PageSwapCache() and freeing swap cache data structure.
Another possible method to fix this is to use preempt_off() + stop_machine() to prevent the swap device from being swapoff when its data structure is being accessed. The overhead in hot-path of both methods is similar. The advantages of RCU based method are,
1. stop_machine() may disturb the normal execution code path on other CPUs.
2. File cache uses RCU to protect its radix tree. If the similar mechanism is used for swap cache too, it is easier to share code between them.
3. RCU is used to protect swap cache in total_swapcache_pages() and exit_swap_address_space() already. The two mechanisms can be merged to simplify the logic.
Changelog:
v9:
- Rebased on latest mmotm
- Revise the comments and patch description
v8:
- Use swp_swap_info() to cleanup the code per Daniel's comments
- Use rcu_read_lock/unlock and synchronize_rcu() per Andrea Arcangeli's comments
- Added Fixes tag per Michal Hocko's comments
v7:
- Rebased on patch: "mm, swap: bounds check swap_info accesses to avoid NULL derefs"
v6:
- Add more comments to get_swap_device() to make it more clear about possible swapoff or swapoff+swapon.
v5:
- Replace RCU with stop_machine()
v4:
- Use synchronize_rcu() in enable_swap_info() to reduce overhead of normal paths further.
v3:
- Re-implemented with RCU to reduce the overhead of normal paths
v2:
- Re-implemented with SRCU to reduce the overhead of normal paths.
- Avoid to check whether the swap device has been swapoff in get_swap_device(). Because we can check the origin of the swap entry to make sure the swap device hasn't bee swapoff.
--- include/linux/swap.h | 13 +++- mm/memory.c | 2 +- mm/swap_state.c | 16 ++++- mm/swapfile.c | 154 ++++++++++++++++++++++++++++++++++--------- 4 files changed, 146 insertions(+), 39 deletions(-)
include/linux/swap.h +10 -3
@@ -175,8 +175,9 @@ enum {SWP_PAGE_DISCARD = (1 <<10 ),/* freed swap page-cluster discards */ SWP_STABLE_WRITES = (1 <<11 ),/* no overwrite PG_writeback pages */ SWP_SYNCHRONOUS_IO = (1 <<12 ),/* synchronous IO is efficient */ +SWP_VALID = (1 <<13 ),/* swap is valid to be operated on? */ /* add others here before... */ - SWP_SCANNING = (1 << 13), /* refcount in scan_swap_map */ + SWP_SCANNING = (1 << 14), /* refcount in scan_swap_map */ };#define SWAP_CLUSTER_MAX 32UL
@@ -460,7 +461,7 @@ extern unsigned int count_swap_pages(int, int);extern sector_t map_swap_page (struct page *,struct block_device **);extern sector_t swapdev_block (int ,pgoff_t );extern int page_swapcount (struct page *); -extern int __swap_count (struct swap_info_struct *si,swp_entry_t entry); +extern int __swap_count (swp_entry_t entry);extern int __swp_swapcount (swp_entry_t entry);extern int swp_swapcount (swp_entry_t entry);extern struct swap_info_struct *page_swap_info (struct page *);
@@ -470,6 +471,12 @@ extern int try_to_free_swap(struct page *);struct backing_dev_info ;extern int init_swap_address_space (unsigned int type,unsigned long nr_pages);extern void exit_swap_address_space (unsigned int type); +extern struct swap_info_struct *get_swap_device (swp_entry_t entry); + +static inline void put_swap_device (struct swap_info_struct *si) +{ +rcu_read_unlock (); +}#else /* CONFIG_SWAP */
@@ -576,7 +583,7 @@ static inline int page_swapcount(struct page *page)return 0 ; } -static inline int __swap_count (struct swap_info_struct *si,swp_entry_t entry) +static inline int __swap_count (swp_entry_t entry) {return 0 ; }
mm/memory.c +1 -1
@@ -2806,7 +2806,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)struct swap_info_struct *si =swp_swap_info (entry);if (si->flags &SWP_SYNCHRONOUS_IO && -__swap_count (si, entry) ==1 ) { +__swap_count (entry) ==1 ) {/* skip swapcache */ page =alloc_page_vma (GFP_HIGHUSER_MOVABLE , vma, vmf->address);
mm/swap_state.c +13 -3
@@ -310,8 +310,13 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, unsigned long addr) {struct page *page; +struct swap_info_struct *si; + si =get_swap_device (entry); +if (!si) +return NULL ; page =find_get_page (swap_address_space (entry),swp_offset (entry)); +put_swap_device (si);INC_CACHE_INFO (find_total);if (page) {
@@ -354,8 +359,8 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr, bool *new_page_allocated) { - struct page *found_page, *new_page = NULL; -struct address_space *swapper_space =swap_address_space (entry); + struct page *found_page = NULL, *new_page = NULL; +struct swap_info_struct *si;int err; *new_page_allocated = false;
@@ -365,7 +370,12 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,* called after lookup_swap_cache() failed, re-calling * that would confuse statistics. */ - found_page =find_get_page (swapper_space,swp_offset (entry)); + si =get_swap_device (entry); +if (!si) +break ; + found_page =find_get_page (swap_address_space (entry), +swp_offset (entry)); +put_swap_device (si);if (found_page)break ;
mm/swapfile.c +122 -32
264 lines not on this page · raw message
Signature · 1 line
2.20.1