mm/swap: fix SWP_PFN_BITS with CONFIG_PHYS_ADDR_T_64BIT on 32bit
Subsystems (2)
PATCH 1/1 mm/swap: fix SWP_PFN_BITS with CONFIG_PHYS_ADDR_T_64BIT on 32bit
We use "unsigned long" to store a PFN in the kernel and phys_addr_t to store a physical address.
On a 64bit system, both are 64bit wide. However, on a 32bit system, the latter might be 64bit wide. This is, for example, the case on x86 with
PAE: phys_addr_t and PTEs are 64bit wide, while "unsigned long" only spans 32bit.
The current definition of SWP_PFN_BITS without MAX_PHYSMEM_BITS misses that case, and assumes that the maximum PFN is limited by a 32bit phys_addr_t. This implies, that SWP_PFN_BITS will currently only be able to cover 4 GiB - 1 on any 32bit system, which is wrong. We can end up masking off valid PFN bits from the swap offset.
Ideally, we'd use something like "sizeof(phys_addr_t) * 8 - PAGE_SHIFT", however, we might exceed SWP_TYPE_SHIFT and make the BUILD_BUG_ON() in is_pfn_swap_entry() unhappy. Note that swp_entry_t is effectively an unsigned long.
Consequently, simply rely on SWP_TYPE_SHIFT in case we're on 32bit and CONFIG_PHYS_ADDR_T_64BIT is defined.
For example, on an 8 GiB x86 PAE system, we currently fail removing migration entries (remove_migration_ptes()) that target PFNs above 4 GiB, because mm/page_vma_mapped.c:check_pte() will fail to identify a PFN match as swp_offset_pfn() wrongly masks off valid PFN bits.
With THP, split_huge_page_to_list()->...->remap_page() will leave migration entries in place and continue to unlock the page, which is wrong. Later, when we stumble over these migration entries (e.g., via /proc/self/pagemap), pfn_swap_entry_to_page() will BUG_ON() because these migration entries shouldn't exist anymore and the page was unlocked.
[ 33.067591] kernel BUG at include/linux/swapops.h:497! [ 33.067597] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 33.067602] CPU: 3 PID: 742 Comm: cow Tainted: G E 6.1.0-rc8+ #16 [ 33.067605] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-1.fc36 04/01/2014 [ 33.067606] EIP: pagemap_pmd_range+0x644/0x650 [ 33.067612] Code: 00 00 00 00 66 90 89 ce b9 00 f0 ff ff e9 ff fb ... [ 33.067615] EAX: ee394000 EBX: 00000002 ECX: ee394000 EDX: 00000000 [ 33.067617] ESI: c1b0ded4 EDI: 00024a00 EBP: c1b0ddb4 ESP: c1b0dd68 [ 33.067619] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 EFLAGS: 00010246 [ 33.067624] CR0: 80050033 CR2: b7a00000 CR3: 01bbbd20 CR4: 00350ef0 [ 33.067625] Call Trace: [ 33.067628] ? madvise_free_pte_range+0x720/0x720 [ 33.067632] ? smaps_pte_range+0x4b0/0x4b0 [ 33.067634] walk_pgd_range+0x325/0x720 [ 33.067637] ? mt_find+0x1d6/0x3a0 [ 33.067641] ? mt_find+0x1d6/0x3a0 [ 33.067643] __walk_page_range+0x164/0x170 [ 33.067646] walk_page_range+0xf9/0x170 [ 33.067648] ? __kmem_cache_alloc_node+0x2a8/0x340 [ 33.067653] pagemap_read+0x124/0x280 [ 33.067658] ? default_llseek+0x101/0x160 [ 33.067662] ? smaps_account+0x1d0/0x1d0 [ 33.067664] vfs_read+0x90/0x290 [ 33.067667] ? do_madvise.part.0+0x24b/0x390 [ 33.067669] ? debug_smp_processor_id+0x12/0x20 [ 33.067673] ksys_pread64+0x58/0x90 [ 33.067675] __ia32_sys_ia32_pread64+0x1b/0x20 [ 33.067680] __do_fast_syscall_32+0x4c/0xc0 [ 33.067683] do_fast_syscall_32+0x29/0x60 [ 33.067686] do_SYSENTER_32+0x15/0x20 [ 33.067689] entry_SYSENTER_32+0x98/0xf1
---
This makes my x86 PAE case work as expected again. Only cross compiled on other architectures.
--- include/linux/swapops.h | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
include/linux/swapops.h +5 -3
@@ -32,11 +32,13 @@* store PFN, we only need SWP_PFN_BITS bits. Each of the pfn swap entries * can use the extra bits to store other information besides PFN. */ -#ifdef MAX_PHYSMEM_BITS +#if defined(MAX_PHYSMEM_BITS )#define SWP_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT) -#else /* MAX_PHYSMEM_BITS */ +#elif !defined(CONFIG_64BIT ) && defined(CONFIG_PHYS_ADDR_T_64BIT ) +#define SWP_PFN_BITS SWP_TYPE_SHIFT +#else #define SWP_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT) -#endif /* MAX_PHYSMEM_BITS */ +#endif /* defined(MAX_PHYSMEM_BITS) */ #define SWP_PFN_MASK (BIT(SWP_PFN_BITS) - 1)/**
base-commit: 76dcd734eca23168cb008912c0f69ff408905235
Signature · 1 line
2.38.1
Quoted 65 lines · David Hildenbrand
We use "unsigned long" to store a PFN in the kernel and
phys_addr_tto store a physical address.On a 64bit system, both are 64bit wide. However, on a 32bit system, the latter might be 64bit wide. This is, for example, the case on x86 with
PAE:phys_addr_tand PTEs are 64bit wide, while "unsigned long" only spans 32bit.The current definition of
SWP_PFN_BITSwithoutMAX_PHYSMEM_BITSmisses that case, and assumes that the maximum PFN is limited by a 32bitphys_addr_t. This implies, thatSWP_PFN_BITSwill currently only be able to cover 4 GiB - 1 on any 32bit system, which is wrong. We can end up masking off valid PFN bits from the swap offset.Ideally, we'd use something like "
sizeof(phys_addr_t)* 8 -PAGE_SHIFT", however, we might exceedSWP_TYPE_SHIFTand make theBUILD_BUG_ON()inis_pfn_swap_entry()unhappy. Note thatswp_entry_tis effectively an unsigned long.Consequently, simply rely on
SWP_TYPE_SHIFTin case we're on 32bit andCONFIG_PHYS_ADDR_T_64BITis defined.For example, on an 8 GiB x86 PAE system, we currently fail removing migration entries (
remove_migration_ptes()) that target PFNs above 4 GiB, because mm/page_vma_mapped.c:check_pte() will fail to identify a PFN match asswp_offset_pfn()wrongly masks off valid PFN bits.With THP, split_huge_page_to_list()->...->remap_page() will leave migration entries in place and continue to unlock the page, which is wrong. Later, when we stumble over these migration entries (e.g., via
/proc/self/pagemap),pfn_swap_entry_to_page()willBUG_ON()because these migration entries shouldn't exist anymore and the page was unlocked.[ 33.067591] kernel BUG at include/linux/swapops.h:497! [ 33.067597] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 33.067602] CPU: 3 PID: 742 Comm: cow Tainted: G E 6.1.0-rc8+ #16 [ 33.067605] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-1.fc36 04/01/2014 [ 33.067606] EIP: pagemap_pmd_range+0x644/0x650 [ 33.067612] Code: 00 00 00 00 66 90 89 ce b9 00 f0 ff ff e9 ff fb ... [ 33.067615] EAX: ee394000 EBX: 00000002 ECX: ee394000 EDX: 00000000 [ 33.067617] ESI: c1b0ded4 EDI: 00024a00 EBP: c1b0ddb4 ESP: c1b0dd68 [ 33.067619] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 EFLAGS: 00010246 [ 33.067624] CR0: 80050033 CR2: b7a00000 CR3: 01bbbd20 CR4: 00350ef0 [ 33.067625] Call Trace: [ 33.067628] ? madvise_free_pte_range+0x720/0x720 [ 33.067632] ? smaps_pte_range+0x4b0/0x4b0 [ 33.067634] walk_pgd_range+0x325/0x720 [ 33.067637] ? mt_find+0x1d6/0x3a0 [ 33.067641] ? mt_find+0x1d6/0x3a0 [ 33.067643] __walk_page_range+0x164/0x170 [ 33.067646] walk_page_range+0xf9/0x170 [ 33.067648] ? __kmem_cache_alloc_node+0x2a8/0x340 [ 33.067653] pagemap_read+0x124/0x280 [ 33.067658] ? default_llseek+0x101/0x160 [ 33.067662] ? smaps_account+0x1d0/0x1d0 [ 33.067664] vfs_read+0x90/0x290 [ 33.067667] ? do_madvise.part.0+0x24b/0x390 [ 33.067669] ? debug_smp_processor_id+0x12/0x20 [ 33.067673] ksys_pread64+0x58/0x90 [ 33.067675] __ia32_sys_ia32_pread64+0x1b/0x20 [ 33.067680] __do_fast_syscall_32+0x4c/0xc0 [ 33.067683] do_fast_syscall_32+0x29/0x60 [ 33.067686] do_SYSENTER_32+0x15/0x20 [ 33.067689] entry_SYSENTER_32+0x98/0xf1
Quoted 7 lines
Fixes 0d206b5d2e0d ("mm/swap: add swp_offset_pfn() to fetch PFN from swap entry")Cc Andrew Morton <[email protected]>Cc Peter Xu <[email protected]>Cc Yang Shi <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrea Arcangeli <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Thanks for debugging this one.
Quoted 4 lines
---
This makes my x86 PAE case work as expected again. Only cross compiled on other architectures.
IIUC it's not about PAE but !SPARSEMEM, as PAE actually has it defined when with sparsemem:
#ifdef CONFIG_X86_32
# ifdef CONFIG_X86_PAE # define SECTION_SIZE_BITS 29 # define MAX_PHYSMEM_BITS 36 # else # define SECTION_SIZE_BITS 26 # define MAX_PHYSMEM_BITS 32 # endif #else /* CONFIG_X86_32 */ # define SECTION_SIZE_BITS 27 /* matt - 128 is convenient right now */ # define MAX_PHYSMEM_BITS (pgtable_l5_enabled() ? 52 : 46) #endif
One trivial comment below.
Quoted 10 lines
--- include/linux/swapops.h | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)include/linux/swapops.h +0
-0
include/linux/swapops.h line 38
Quoted 9 lines
include/linux/swapops.h +3
-2@@ -32,11 +32,13 @@* store PFN, we only need SWP_PFN_BITS bits. Each of the pfn swap entries * can use the extra bits to store other information besides PFN. */ -#ifdef MAX_PHYSMEM_BITS +#if defined(MAX_PHYSMEM_BITS )#define SWP_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT) -#else /* MAX_PHYSMEM_BITS */ +#elif !defined(CONFIG_64BIT ) && defined(CONFIG_PHYS_ADDR_T_64BIT ) +#define SWP_PFN_BITS SWP_TYPE_SHIFT
Can we add a comment showing where SWP_TYPE_SHIFT comes from? It should be a min value comes from either the limitation of phys address width, or from definition of swp_entry_t (which is unsigned long).
Or I'd rather make this then the code explains better on itself, and the change should be smaller too:
#ifdef MAX_PHYSMEM_BITS
#define SWP_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT)
#else /* MAX_PHYSMEM_BITS */
-#define SWP_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT)
+#define SWP_PFN_BITS MIN((sizeof(phys_addr_t) * 8) - \
+ PAGE_SHIFT, SWP_TYPE_SHIFT)
#endif /* MAX_PHYSMEM_BITS */
#define SWP_PFN_MASK (BIT(SWP_PFN_BITS) - 1)
What do you think?
Quoted 12 lines
include/linux/swapops.h +2
-1+#else #define SWP_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT) -#endif /* MAX_PHYSMEM_BITS */ +#endif /* defined(MAX_PHYSMEM_BITS) */ #define SWP_PFN_MASK (BIT(SWP_PFN_BITS) - 1)/** base-commit:
76dcd734eca23168cb008912c0f69ff408905235Signature · 1 line
2.38.1
Signature · 1 line
Peter Xu
Quoted 2 lines
Thanks for debugging this one.
Sure!
Quoted 10 lines
Quoted 5 lines
---
This makes my x86 PAE case work as expected again. Only cross compiled on other architectures.
IIUC it's not about PAE but !SPARSEMEM, as PAE actually has it defined when with sparsemem:
#ifdef
CONFIG_X86_32
Quoted 11 lines
# ifdef CONFIG_X86_PAE # define SECTION_SIZE_BITS 29 # define MAX_PHYSMEM_BITS 36 # else # define SECTION_SIZE_BITS 26 # define MAX_PHYSMEM_BITS 32 # endif #else /* CONFIG_X86_32 */ # define SECTION_SIZE_BITS 27 /* matt - 128 is convenient right now */ # define MAX_PHYSMEM_BITS (pgtable_l5_enabled() ? 52 : 46) #endif
Indeed, I'll extend the description accordingly.
Quoted 39 lines
One trivial comment below.
Quoted 10 lines
--- include/linux/swapops.h | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)include/linux/swapops.h +0
-0include/linux/swapops.h line 38
Quoted 9 lines
include/linux/swapops.h +3
-2@@ -32,11 +32,13 @@* store PFN, we only need SWP_PFN_BITS bits. Each of the pfn swap entries * can use the extra bits to store other information besides PFN. */ -#ifdef MAX_PHYSMEM_BITS +#if defined(MAX_PHYSMEM_BITS )#define SWP_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT) -#else /* MAX_PHYSMEM_BITS */ +#elif !defined(CONFIG_64BIT ) && defined(CONFIG_PHYS_ADDR_T_64BIT ) +#define SWP_PFN_BITS SWP_TYPE_SHIFTCan we add a comment showing where
SWP_TYPE_SHIFTcomes from? It should be a min value comes from either the limitation of phys address width, or from definition ofswp_entry_t(which is unsigned long).Or I'd rather make this then the code explains better on itself, and the change should be smaller too:
#ifdef MAX_PHYSMEM_BITS #define SWP_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT) #else /* MAX_PHYSMEM_BITS */(quoted diff) +2
-1-#define SWP_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT) +#define SWP_PFN_BITS MIN((sizeof(phys_addr_t) * 8) - \ + PAGE_SHIFT, SWP_TYPE_SHIFT) #endif /* MAX_PHYSMEM_BITS */ #define SWP_PFN_MASK (BIT(SWP_PFN_BITS) - 1)
Quoted 1 line
What do you think?
Sure, if we can make the compiler happy:
./include/linux/swapops.h: In function 'swp_offset_pfn':./include/linux/swapops.h:38:41: error: implicit declaration of function 'MIN' [-Werror=implicit-function-declaration]
38 | #define SWP_PFN_BITS
MIN((sizeof(phys_addr_t) * 8) - \
| ^~~./include/vdso/bits.h:7:44: note: in definition of macro 'BIT'
7 | #define BIT(nr) (UL(1) << (nr))
| ^~./include/linux/swapops.h:41:46: note: in expansion of macro 'SWP_PFN_BITS'
41 | #define SWP_PFN_MASK (BIT(SWP_PFN_BITS) - 1)
| ^~~~~~~~~~~~./include/linux/swapops.h:119:36: note: in expansion of macro 'SWP_PFN_MASK'
119 | return swp_offset(entry) & SWP_PFN_MASK;
| ^~~~~~~~~~~~using "min" instead gives plenty of warnings. min_t() seems to work:
include/linux/swapops.h +4 -2
@@ -34,8 +34,10 @@*/ #ifdef MAX_PHYSMEM_BITS #define SWP_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT) -#else /* MAX_PHYSMEM_BITS */ -#define SWP_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT) +#else /* MAX_PHYSMEM_BITS */ +#define SWP_PFN_BITS min_t(phys_addr_t, \ + (sizeof(phys_addr_t) * 8) - \ + PAGE_SHIFT, SWP_TYPE_SHIFT)#endif /* MAX_PHYSMEM_BITS */ #define SWP_PFN_MASK (BIT(SWP_PFN_BITS) - 1)
I'm currently cross-compiling that and will give it a churn.
Thanks!
Signature · 3 lines
Thanks, David / dhildenb