mm/vma: convert vm_flags_t to vma_flags_t in vma code
Subsystems (22)
Lorenzo Stoakes (Oracle)
maintainerv3 cover mm/vma: convert vm_flags_t to vma_flags_t in vma code
This series converts a lot of the existing use of the legacy vm_flags_t data type to the new vma_flags_t type which replaces it.
In order to do so it adds a number of additional helpers:
* vma_flags_empty() - Determines whether a vma_flags_t value has no bits set.
* vma_flags_and() - Performs a bitwise AND between two vma_flags_t values.
* vma_flags_diff_pair() - Determines which flags are not shared between a pair of VMA flags (typically non-constant values)
* append_vma_flags() - Similar to mk_vma_flags(), but allows a vma_flags_t value to be specified (typically a constant value) which will be copied and appended to to create a new vma_flags_t value, with additional flags specified to append to it.
* vma_flags_same() - Determines if a vma_flags_t value is exactly equal to a set of VMA flags.
* vma_flags_same_mask() - Determines if a vma_flags_t value is eactly equal to another vma_flags_t value (typically constant).
* vma_flags_same_pair() - Determines if a pair of vma_flags_t values are exactly equal to one another (typically both non-constant).
* vma_flags_to_legacy() - Converts a vma_flags_t value to a vm_flags_t value, used to enable more iterative introduction of the use of vma_flags_t.
* legacy_to_vma_flags() - Converts a vm_flags_t value to a vma_flags-t value, for the same purpose.
* vma_flags_test_single_mask() - Tests whether a vma_flags_t value contain the single flag specified in an input vma_flags_t flag mask, or if that flag mask is empty, is defined to return false. Useful for config-predicated VMA flag mask defines.
* vma_test() - Tests whether a VMA's flags contain a specific singular VMA flag.
* vma_test_any() - Tests whether a VMA's flags contain any of a set of VMA flags.
* vma_test_any_mask() - Tests whether a VMA's flags contain any of the flags specified in another, typically constant, vma_flags_t value.
* vma_test_single_mask() - Tests whether a VMA's flags contain the single flag specified in an input vma_flags_t flag mask, or if that flag mask is empty, is defined to return false. Useful for config-predicated VMA flag mask defines.
* vma_clear_flags() - Clears a specific set of VMA flags from a vma_flags_t value.
* vma_clear_flags_mask() - Clears those flag set in a vma_flags_t value (typically constant) from a (typically not constant) vma_flags_t value.
The series mostly focuses on the the VMA specific code, especially that contained in mm/vma.c and mm/vma.h.
It updates both brk() and mmap() logic to utils vma_flags_t values as much as is practiaclly possible at this point, changing surrounding logic to be able to do so.
It also updates the vma_modify_xxx() functions where they interact with VMA flags directly to use vm_flags_t values where possible.
There is extensive testing added in the VMA userland tests to assert that all of these new VMA flag functions work correctly.
v3:
* Folded in tags, thanks Paul, Vlastimil!
* Respun to apply the correct suggestions/reports from
https://sashiko.dev/#/patchset/cover.1773665966.git.ljs%40kernel.org
(note there is also quite a lot of noise, that is ignored).
* Const-ified vma_flags_t * param for vma_flags_empty() as per Sashiko.
* Obtained sticky flag masks after VMA write lock acquired on merge and update vma_expand() similarly. This is meaningful, as the maybe guard flag is set atomically which might race outside of a VMA write lock, as per Sashiko.
* Dropped comment about 'VM_NONE convenience' from 5/23 commit message as the VMA_xxx form flags can now provide that.
* Updated legacy_to_vma_flags() to ensure upper bits are cleared for NUM_VMA_FLAG_BITS > 64 as per Sashiko.
* Updated legacy_to_vma_flags() to use vma_flags_overwrite_word() for consistency.
* Refreshed vma_flags_overwrite_word(), vma_flag_overwrite_word_once(), vma_flags_set_word() and vma_flags_clear_word() in the VMA test dup.h header to keep them consistent with the kernel.
* Updated VMA_DATA_DEFAULT_FLAGS declaration on arm64 to be predicated on CONFIG_ARM64_MTE as to whether VMA_MTE_ALLOWED_BIT is set, as per Sashiko.
* Fixed bug where the VMA did not have VMA_SOFTDIRTY_BIT set if pgtable_supports_soft_dirty(), but rather the stack variable vma_flags, as per Sashiko.
* Corrected vmag -> vma typo in VMA test code as per Sashiko.
* Fixed typo in 20/23 commit message 'correctly' -> 'correct' as per Sashiko.
* Fixed VMA flag clear tests to consistently do vma_clear_flags_mask(&vma, mask) rather than vma_flags_clear_mask(&vma.flags, mask) as per Sashiko.
* Added missing vma_start_write() in mseal_apply() as per Sahiko.
v2:
* Rebased on mm-unstable.
* Added vma_flags_count() and vma[_flags]_test_single_mask() for testing whether flags have a single flag set depending on an input flag mask, returning false if the flag mask is empty.
* Added tests for vma_flags_count() and vma[_flags]_test_single_mask().
* Updated the KSM VMA_DROPPABLE test to use vma_flags_test_single_mask().
* Updated the newly-introduced-since-rebase vma_supports_mlock() to use vma_flags_t.
https://lore.kernel.org/linux-mm/[email protected]/
v1:
https://lore.kernel.org/linux-mm/[email protected]/
Lorenzo Stoakes (Oracle) (23):
mm/vma: add vma_flags_empty(), vma_flags_and(), vma_flags_diff_pair() tools/testing/vma: add unit tests flag empty, diff_pair, and[_mask] mm/vma: add further vma_flags_t unions
tools/testing/vma: convert bulk of test code to vma_flags_t mm/vma: use new VMA flags for sticky flags logic tools/testing/vma: fix VMA flag tests
mm/vma: add append_vma_flags() helper
tools/testing/vma: add simple test for append_vma_flags()
mm: unexport vm_brk_flags() and eliminate vm_flags parameter mm/vma: introduce vma_flags_same[_mask/_pair]()
mm/vma: introduce [vma_flags,legacy]_to_[legacy,vma_flags]() helpers tools/testing/vma: test that legacy flag helpers work correctly mm/vma: introduce vma_test[_any[_mask]](), and make inlining
consistent
tools/testing/vma: update VMA flag tests to test vma_test[_any_mask]()
mm: introduce vma_flags_count() and vma[_flags]_test_single_mask() tools/testing/vma: test vma_flags_count,vma[_flags]_test_single_mask
mm: convert do_brk_flags() to use vma_flags_t
mm: update vma_supports_mlock() to use new VMA flags mm/vma: introduce vma_clear_flags[_mask]() tools/testing/vma: update VMA tests to test vma_clear_flags[_mask]() mm/vma: convert as much as we can in mm/vma.c to vma_flags_t mm/vma: convert vma_modify_flags[_uffd]() to use vma_flags_t mm/vma: convert __mmap_region() to use vma_flags_t
arch/arc/include/asm/page.h | 2 +- arch/arm/include/asm/page.h | 2 +- arch/arm64/include/asm/page.h | 7 +- arch/hexagon/include/asm/page.h | 2 +- arch/loongarch/include/asm/page.h | 2 +- arch/mips/include/asm/page.h | 2 +- arch/nios2/include/asm/page.h | 2 +- arch/powerpc/include/asm/page.h | 4 +- arch/powerpc/include/asm/page_32.h | 2 +- arch/powerpc/include/asm/page_64.h | 12 +- arch/riscv/include/asm/page.h | 2 +- arch/s390/include/asm/page.h | 2 +- arch/x86/include/asm/page_types.h | 2 +- arch/x86/um/asm/vm-flags.h | 4 +- fs/binfmt_elf.c | 3 +- include/linux/ksm.h | 10 +- include/linux/mm.h | 328 ++++++++++++++++++++------ include/linux/mm_types.h | 52 ++++- include/linux/mman.h | 49 ---- include/linux/userfaultfd_k.h | 3 + mm/internal.h | 7 +- mm/ksm.c | 43 ++-- mm/madvise.c | 10 +- mm/mlock.c | 38 +-- mm/mmap.c | 19 +- mm/mprotect.c | 53 +++-- mm/mremap.c | 6 +- mm/mseal.c | 11 +- mm/userfaultfd.c | 21 +- mm/vma.c | 211 ++++++++++------- mm/vma.h | 86 ++++++- mm/vma_exec.c | 5 +- security/selinux/hooks.c | 4 +- tools/include/linux/bitmap.h | 13 ++ tools/lib/bitmap.c | 10 + tools/testing/vma/include/custom.h | 25 -- tools/testing/vma/include/dup.h | 287 +++++++++++++++++------ tools/testing/vma/include/stubs.h | 11 +- tools/testing/vma/shared.c | 8 +- tools/testing/vma/shared.h | 22 +- tools/testing/vma/tests/merge.c | 311 +++++++++++++------------ tools/testing/vma/tests/mmap.c | 18 +- tools/testing/vma/tests/vma.c | 359 +++++++++++++++++++++++++---- tools/testing/vma/vma_internal.h | 6 + 44 files changed, 1435 insertions(+), 641 deletions(-)
--
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 1/23 mm/vma: add vma_flags_empty(), vma_flags_and(), vma_flags_diff_pair()
Firstly, add the ability to determine if VMA flags are empty, that is no flags are set in a vma_flags_t value.
Next, add the ability to obtain the equivalent of the bitwise and of two vma_flags_t values, via vma_flags_and().
Next, add the ability to obtain the difference between two sets of VMA flags, that is the equivalent to the exclusive bitwise OR of the two sets of flags, via vma_flags_diff_pair().
vma_flags_xxx_mask() typically operates on a pointer to a vma_flags_t value, which is assumed to be an lvalue of some kind (such as a field in a struct or a stack variable) and an rvalue of some kind (typically a constant set of VMA flags obtained e.g. via mk_vma_flags() or equivalent).
However vma_flags_diff_pair() is intended to operate on two lvalues, so use the _pair() suffix to make this clear.
Finally, update VMA userland tests to add these helpers.
We also port bitmap_xor() and __bitmap_xor() to the tools/ headers and source to allow the tests to work with vma_flags_diff_pair().
--- include/linux/mm.h | 60 ++++++++++++++++++++++++++++----- include/linux/mm_types.h | 8 +++++ tools/include/linux/bitmap.h | 13 +++++++ tools/lib/bitmap.c | 10 ++++++ tools/testing/vma/include/dup.h | 36 +++++++++++++++++++- 5 files changed, 117 insertions(+), 10 deletions(-)
include/linux/mm.h +51 -9
@@ -1053,6 +1053,19 @@ static __always_inline vma_flags_t __mk_vma_flags(size_t count,return flags; } +/* +* Helper macro which bitwise-or combines the specified input flags into a +* vma_flags_t bitmap value. E.g.: +* +* vma_flags_t flags = mk_vma_flags(VMA_IO_BIT, VMA_PFNMAP_BIT, +* VMA_DONTEXPAND_BIT, VMA_DONTDUMP_BIT); +* +* The compiler cleverly optimises away all of the work and this ends up being +* equivalent to aggregating the values manually. +*/ +#define mk_vma_flags (...) __mk_vma_flags(COUNT_ARGS(__VA_ARGS__), \ + (const vma_flag_t []){__VA_ARGS__}) +/* * Test whether a specific VMA flag is set, e.g.: *
@@ -1067,17 +1080,30 @@ static __always_inline bool vma_flags_test(const vma_flags_t *flags, }/* -* Helper macro which bitwise-or combines the specified input flags into a -* vma_flags_t bitmap value. E.g.: -* -* vma_flags_t flags = mk_vma_flags(VMA_IO_BIT, VMA_PFNMAP_BIT, -* VMA_DONTEXPAND_BIT, VMA_DONTDUMP_BIT); +* Obtain a set of VMA flags which contain the overlapping flags contained +* within flags and to_and. +*/ +static __always_inline vma_flags_t vma_flags_and_mask(const vma_flags_t *flags, +vma_flags_t to_and) +{ +vma_flags_t dst; +unsigned long *bitmap_dst = dst.__vma_flags; +const unsigned long *bitmap = flags->__vma_flags; +const unsigned long *bitmap_to_and = to_and.__vma_flags; + +bitmap_and (bitmap_dst, bitmap, bitmap_to_and,NUM_VMA_FLAG_BITS ); +return dst; +} + +/* +* Obtain a set of VMA flags which contains the specified overlapping flags, +* e.g.: * -* The compiler cleverly optimises away all of the work and this ends up being -* equivalent to aggregating the values manually. +* vma_flags_t read_flags = vma_flags_and(&flags, VMA_READ_BIT, +* VMA_MAY_READ_BIT); */ -#define mk_vma_flags (...) __mk_vma_flags(COUNT_ARGS(__VA_ARGS__), \ - (const vma_flag_t []){__VA_ARGS__}) +#define vma_flags_and (flags, ...) \ + vma_flags_and_mask(flags, mk_vma_flags(__VA_ARGS__))/* Test each of to_test flags in flags, non-atomically. */ static __always_inline bool vma_flags_test_any_mask(const vma_flags_t *flags,
@@ -1151,6 +1177,22 @@ static __always_inline void vma_flags_clear_mask(vma_flags_t *flags,#define vma_flags_clear (flags, ...) \ vma_flags_clear_mask(flags, mk_vma_flags(__VA_ARGS__)) +/* +* Obtain a VMA flags value containing those flags that are present in flags or +* flags_other but not in both. +*/ +static __always_inline vma_flags_t vma_flags_diff_pair(const vma_flags_t *flags, +const vma_flags_t *flags_other) +{ +vma_flags_t dst; +const unsigned long *bitmap_other = flags_other->__vma_flags; +const unsigned long *bitmap = flags->__vma_flags; +unsigned long *bitmap_dst = dst.__vma_flags; + +bitmap_xor (bitmap_dst, bitmap, bitmap_other,NUM_VMA_FLAG_BITS ); +return dst; +} +/* * Helper to test that ALL specified flags are set in a VMA. *
include/linux/mm_types.h +8 -0
@@ -870,6 +870,14 @@ typedef struct {#define EMPTY_VMA_FLAGS ((vma_flags_t){ }) +/* Are no flags set in the specified VMA flags? */ +static __always_inline bool vma_flags_empty(const vma_flags_t *flags) +{ +const unsigned long *bitmap = flags->__vma_flags; + +return bitmap_empty (bitmap,NUM_VMA_FLAG_BITS ); +} +/* * Describes a VMA that is about to be mmap()'ed. Drivers may choose to * manipulate mutable fields which will cause those fields to be updated in the
tools/include/linux/bitmap.h +13 -0
@@ -28,6 +28,8 @@ bool __bitmap_subset(const unsigned long *bitmap1, const unsigned long *bitmap2, unsigned int nbits);bool __bitmap_andnot (unsigned long *dst,const unsigned long *bitmap1,const unsigned long *bitmap2,unsigned int nbits); +void __bitmap_xor (unsigned long *dst,const unsigned long *bitmap1, +const unsigned long *bitmap2,unsigned int nbits);#define BITMAP_FIRST_WORD_MASK (start) (~0UL << ((start) & (BITS_PER_LONG - 1)))#define BITMAP_LAST_WORD_MASK (nbits) (~0UL >> (-(nbits) & (BITS_PER_LONG - 1)))
@@ -209,4 +211,15 @@ static inline void bitmap_clear(unsigned long *map, unsigned int start, else__bitmap_clear (map ,start ,nbits ); } + +static __always_inline +void bitmap_xor(unsigned long *dst, const unsigned long *src1, +const unsigned long *src2,unsigned int nbits) +{ +if (small_const_nbits (nbits)) + *dst = *src1 ^ *src2; +else +__bitmap_xor (dst, src1, src2, nbits); +} +#endif /* _TOOLS_LINUX_BITMAP_H */
tools/lib/bitmap.c +10 -0
@@ -169,3 +169,13 @@ bool __bitmap_subset(const unsigned long *bitmap1,return false;return true; } + +void __bitmap_xor (unsigned long *dst,const unsigned long *bitmap1, +const unsigned long *bitmap2,unsigned int bits) +{ +unsigned int k; +unsigned int nr =BITS_TO_LONGS (bits); + +for (k =0 ; k < nr; k++) + dst[k] = bitmap1[k] ^ bitmap2[k]; +}
tools/testing/vma/include/dup.h +35 -1
@@ -422,6 +422,13 @@ struct vma_iterator {#define MAPCOUNT_ELF_CORE_MARGIN (5)#define DEFAULT_MAX_MAP_COUNT (USHRT_MAX - MAPCOUNT_ELF_CORE_MARGIN) +static __always_inline bool vma_flags_empty(const vma_flags_t *flags) +{ +const unsigned long *bitmap = flags->__vma_flags; + +return bitmap_empty (bitmap,NUM_VMA_FLAG_BITS ); +} +/* What action should be taken after an .mmap_prepare call is complete? */ enum mmap_action_type {MMAP_NOTHING ,/* Mapping is complete, no further action. */
@@ -855,6 +862,21 @@ static __always_inline bool vma_flags_test(const vma_flags_t *flags,return test_bit ((__force int )bit, bitmap); } +static __always_inline vma_flags_t vma_flags_and_mask(const vma_flags_t *flags, +vma_flags_t to_and) +{ +vma_flags_t dst; +unsigned long *bitmap_dst = dst.__vma_flags; +const unsigned long *bitmap = flags->__vma_flags; +const unsigned long *bitmap_to_and = to_and.__vma_flags; + +bitmap_and (bitmap_dst, bitmap, bitmap_to_and,NUM_VMA_FLAG_BITS ); +return dst; +} + +#define vma_flags_and (flags, ...) \ + vma_flags_and_mask(flags, mk_vma_flags(__VA_ARGS__)) + static __always_inline bool vma_flags_test_any_mask(const vma_flags_t *flags,vma_flags_t to_test) {
@@ -901,8 +923,20 @@ static __always_inline void vma_flags_clear_mask(vma_flags_t *flags, vma_flags_t#define vma_flags_clear (flags, ...) \ vma_flags_clear_mask(flags, mk_vma_flags(__VA_ARGS__)) +static __always_inline vma_flags_t vma_flags_diff_pair(const vma_flags_t *flags, +const vma_flags_t *flags_other) +{ +vma_flags_t dst; +const unsigned long *bitmap_other = flags_other->__vma_flags; +const unsigned long *bitmap = flags->__vma_flags; +unsigned long *bitmap_dst = dst.__vma_flags; + +bitmap_xor (bitmap_dst, bitmap, bitmap_other,NUM_VMA_FLAG_BITS ); +return dst; +} +static inline bool vma_test_all_mask (const struct vm_area_struct *vma, -vma_flags_t flags) +vma_flags_t flags) {return vma_flags_test_all_mask (&vma->flags, flags); }
--
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 2/23 tools/testing/vma: add unit tests flag empty, diff_pair, and[_mask]
In reply to Lorenzo Stoakes (Oracle)Add VMA unit tests to assert that:
* vma_flags_empty()
* vma_flags_diff_pair()
* vma_flags_and_mask()
* vma_flags_and()
All function as expected.
In additional to the added tests, in order to make testing easier, add vma_flags_same_mask() and vma_flags_same() for testing only. If/when these are required in kernel code, they can be moved over.
Also add ASSERT_FLAGS_[NOT_]SAME[_MASK](), ASSERT_FLAGS_[NON]EMPTY() test helpers to make asserting flag state easier and more convenient.
--- tools/testing/vma/include/custom.h | 12 +++ tools/testing/vma/shared.h | 18 ++++ tools/testing/vma/tests/vma.c | 137 +++++++++++++++++++++++++++++ 3 files changed, 167 insertions(+)
tools/testing/vma/include/custom.h +12 -0
@@ -120,3 +120,15 @@ static inline unsigned long vma_kernel_pagesize(struct vm_area_struct *vma) {return PAGE_SIZE ; } + +/* Place here until needed in the kernel code. */ +static __always_inline bool vma_flags_same_mask(vma_flags_t *flags, +vma_flags_t flags_other) +{ +const unsigned long *bitmap = flags->__vma_flags; +const unsigned long *bitmap_other = flags_other.__vma_flags; + +return bitmap_equal (bitmap, bitmap_other,NUM_VMA_FLAG_BITS ); +} +#define vma_flags_same (flags, ...) \ + vma_flags_same_mask(flags, mk_vma_flags(__VA_ARGS__))
tools/testing/vma/shared.h +18 -0
@@ -35,6 +35,24 @@#define ASSERT_EQ (_val1, _val2) ASSERT_TRUE((_val1) == (_val2))#define ASSERT_NE (_val1, _val2) ASSERT_TRUE((_val1) != (_val2)) +#define ASSERT_FLAGS_SAME_MASK (_flags, _flags_other) \ + ASSERT_TRUE(vma_flags_same_mask((_flags), (_flags_other))) + +#define ASSERT_FLAGS_NOT_SAME_MASK (_flags, _flags_other) \ + ASSERT_FALSE(vma_flags_same_mask((_flags), (_flags_other))) + +#define ASSERT_FLAGS_SAME (_flags, ...) \ + ASSERT_TRUE(vma_flags_same(_flags, __VA_ARGS__)) + +#define ASSERT_FLAGS_NOT_SAME (_flags, ...) \ + ASSERT_FALSE(vma_flags_same(_flags, __VA_ARGS__)) + +#define ASSERT_FLAGS_EMPTY (_flags) \ + ASSERT_TRUE(vma_flags_empty(_flags)) + +#define ASSERT_FLAGS_NONEMPTY (_flags) \ + ASSERT_FALSE(vma_flags_empty(_flags)) +#define IS_SET (_val, _flags) ((_val & _flags) == _flags)extern bool fail_prealloc;
tools/testing/vma/tests/vma.c +137 -0
147 lines not on this page · raw message
Signature · 1 line
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 3/23 mm/vma: add further vma_flags_t unions
In reply to Lorenzo Stoakes (Oracle)In order to utilise the new vma_flags_t type, we currently place it in union with legacy vm_flags fields of type vm_flags_t to make the transition smoother.
Add vma_flags_t union entries for mm->def_flags and vmg->vm_flags - mm->def_vma_flags and vmg->vma_flags respectively.
Once the conversion is complete, these will be replaced with vma_flags_t entries alone.
Also update the VMA tests to reflect the change.
--- include/linux/mm_types.h | 6 +++++- mm/vma.h | 6 +++++- tools/testing/vma/include/dup.h | 5 ++++- 3 files changed, 14 insertions(+), 3 deletions(-)
include/linux/mm_types.h +5 -1
@@ -1262,7 +1262,11 @@ struct mm_struct {unsigned long data_vm;/* VM_WRITE & ~VM_SHARED & ~VM_STACK */ unsigned long exec_vm;/* VM_EXEC & ~VM_WRITE & ~VM_STACK */ unsigned long stack_vm;/* VM_STACK */ -vm_flags_t def_flags; +union { +/* Temporary while VMA flags are being converted. */ +vm_flags_t def_flags; +vma_flags_t def_vma_flags; + };/** * @write_protect_seq: Locked when any thread is write
mm/vma.h +5 -1
@@ -98,7 +98,11 @@ struct vma_merge_struct {unsigned long end;pgoff_t pgoff; -vm_flags_t vm_flags; +union { +/* Temporary while VMA flags are being converted. */ +vm_flags_t vm_flags; +vma_flags_t vma_flags; + };struct file *file;struct anon_vma *anon_vma;struct mempolicy *policy;
tools/testing/vma/include/dup.h +4 -1
@@ -33,7 +33,10 @@ struct mm_struct {unsigned long exec_vm;/* VM_EXEC & ~VM_WRITE & ~VM_STACK */ unsigned long stack_vm;/* VM_STACK */ -unsigned long def_flags; +union { +vm_flags_t def_flags; +vma_flags_t def_vma_flags; + };mm_flags_t flags;/* Must use mm_flags_* helpers to access */ };
--
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 4/23 tools/testing/vma: convert bulk of test code to vma_flags_t
In reply to Lorenzo Stoakes (Oracle)Covert the test code to utilise vma_flags_t as opposed to the deprecate vm_flags_t as much as possible.
As part of this change, add VMA_STICKY_FLAGS and VMA_SPECIAL_FLAGS as early versions of what these defines will look like in the kernel logic once this logic is implemented.
--- tools/testing/vma/include/custom.h | 7 + tools/testing/vma/include/dup.h | 7 +- tools/testing/vma/shared.c | 8 +- tools/testing/vma/shared.h | 4 +- tools/testing/vma/tests/merge.c | 313 +++++++++++++++-------------- tools/testing/vma/tests/vma.c | 10 +- 6 files changed, 186 insertions(+), 163 deletions(-)
tools/testing/vma/include/custom.h +7 -0
@@ -132,3 +132,10 @@ static __always_inline bool vma_flags_same_mask(vma_flags_t *flags, }#define vma_flags_same (flags, ...) \ vma_flags_same_mask(flags, mk_vma_flags(__VA_ARGS__)) +#define VMA_SPECIAL_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_DONTEXPAND_BIT, \ + VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT) +#ifdef CONFIG_MEM_SOFT_DIRTY +#define VMA_STICKY_FLAGS mk_vma_flags(VMA_SOFTDIRTY_BIT, VMA_MAYBE_GUARD_BIT) +#else +#define VMA_STICKY_FLAGS mk_vma_flags(VMA_MAYBE_GUARD_BIT) +#endif
tools/testing/vma/include/dup.h +2 -5
@@ -507,10 +507,7 @@ struct vm_area_desc {/* Mutable fields. Populated with initial state. */ pgoff_t pgoff;struct file *vm_file; -union { -vm_flags_t vm_flags; -vma_flags_t vma_flags; - }; +vma_flags_t vma_flags;pgprot_t page_prot;/* Write-only fields. */
@@ -1146,7 +1143,7 @@ static inline int __compat_vma_mmap(const struct file_operations *f_op, .pgoff = vma->vm_pgoff, .vm_file = vma->vm_file, - .vm_flags = vma->vm_flags, + .vma_flags = vma->flags, .page_prot = vma->vm_page_prot, .action.type = MMAP_NOTHING, /* Default */
tools/testing/vma/shared.c +4 -4
@@ -14,7 +14,7 @@ struct task_struct __current;struct vm_area_struct *alloc_vma (struct mm_struct *mm,unsigned long start,unsigned long end, -pgoff_t pgoff,vm_flags_t vm_flags) +pgoff_t pgoff,vma_flags_t vma_flags) {struct vm_area_struct *vma =vm_area_alloc (mm);
@@ -24,7 +24,7 @@ struct vm_area_struct *alloc_vma(struct mm_struct *mm, vma->vm_start = start; vma->vm_end = end; vma->vm_pgoff = pgoff; -vm_flags_reset (vma, vm_flags); + vma->flags = vma_flags;vma_assert_detached (vma);return vma;
@@ -38,9 +38,9 @@ void detach_free_vma(struct vm_area_struct *vma)struct vm_area_struct *alloc_and_link_vma (struct mm_struct *mm,unsigned long start,unsigned long end, -pgoff_t pgoff,vm_flags_t vm_flags) +pgoff_t pgoff,vma_flags_t vma_flags) { -struct vm_area_struct *vma =alloc_vma (mm, start, end, pgoff, vm_flags); +struct vm_area_struct *vma =alloc_vma (mm, start, end, pgoff, vma_flags);if (vma ==NULL )return NULL ;
tools/testing/vma/shared.h +2 -2
@@ -94,7 +94,7 @@ static inline void dummy_close(struct vm_area_struct *)/* Helper function to simply allocate a VMA. */ struct vm_area_struct *alloc_vma (struct mm_struct *mm,unsigned long start,unsigned long end, -pgoff_t pgoff,vm_flags_t vm_flags); +pgoff_t pgoff,vma_flags_t vma_flags);/* Helper function to detach and free a VMA. */ void detach_free_vma (struct vm_area_struct *vma);
@@ -102,7 +102,7 @@ void detach_free_vma(struct vm_area_struct *vma);/* Helper function to allocate a VMA and link it to the tree. */ struct vm_area_struct *alloc_and_link_vma (struct mm_struct *mm,unsigned long start,unsigned long end, -pgoff_t pgoff,vm_flags_t vm_flags); +pgoff_t pgoff,vma_flags_t vma_flags);/* * Helper function to reset the dummy anon_vma to indicate it has not been
tools/testing/vma/tests/merge.c +166 -147
849 lines not on this page · raw message
tools/testing/vma/tests/vma.c +5 -5
@@ -22,7 +22,8 @@ static bool compare_legacy_flags(vm_flags_t legacy_flags, vma_flags_t flags)static bool test_copy_vma (void ) { -vm_flags_t vm_flags =VM_READ |VM_WRITE |VM_MAYREAD |VM_MAYWRITE ; +vma_flags_t vma_flags =mk_vma_flags (VMA_READ_BIT ,VMA_WRITE_BIT , +VMA_MAYREAD_BIT ,VMA_MAYWRITE_BIT );struct mm_struct mm = {};bool need_locks = false;VMA_ITERATOR (vmi, &mm,0 );
@@ -30,7 +31,7 @@ static bool test_copy_vma(void)/* Move backwards and do not merge. */ - vma =alloc_and_link_vma (&mm,0x3000 ,0x5000 ,3 , vm_flags); + vma =alloc_and_link_vma (&mm,0x3000 ,0x5000 ,3 , vma_flags); vma_new =copy_vma (&vma,0 ,0x2000 ,0 , &need_locks);ASSERT_NE (vma_new, vma);ASSERT_EQ (vma_new->vm_start,0 );
@@ -42,8 +43,8 @@ static bool test_copy_vma(void)/* Move a VMA into position next to another and merge the two. */ - vma =alloc_and_link_vma (&mm,0 ,0x2000 ,0 , vm_flags); - vma_next =alloc_and_link_vma (&mm,0x6000 ,0x8000 ,6 , vm_flags); + vma =alloc_and_link_vma (&mm,0 ,0x2000 ,0 , vma_flags); + vma_next =alloc_and_link_vma (&mm,0x6000 ,0x8000 ,6 , vma_flags); vma_new =copy_vma (&vma,0x4000 ,0x2000 ,4 , &need_locks);vma_assert_attached (vma_new);
@@ -61,7 +62,6 @@ static bool test_vma_flags_unchanged(void)struct vm_area_struct vma;struct vm_area_desc desc; - vma.flags =EMPTY_VMA_FLAGS ; desc.vma_flags =EMPTY_VMA_FLAGS ;
Signature · 1 line
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 5/23 mm/vma: use new VMA flags for sticky flags logic
In reply to Lorenzo Stoakes (Oracle)Use the new vma_flags_t flags implementation to perform the logic around sticky flags and what flags are ignored on VMA merge.
We make use of the new vma_flags_empty(), vma_flags_diff_pair(), and vma_flags_and_mask() functionality.
Also update the VMA tests accordingly.
--- include/linux/mm.h | 32 ++++++++++++-------- mm/vma.c | 48 ++++++++++++++++++++++-------- tools/testing/vma/include/custom.h | 5 ---- tools/testing/vma/include/dup.h | 9 ++++-- 4 files changed, 62 insertions(+), 32 deletions(-)
include/linux/mm.h +19 -13
@@ -540,6 +540,7 @@ enum {/* VMA basic access permission flags */ #define VM_ACCESS_FLAGS (VM_READ | VM_WRITE | VM_EXEC) +#define VMA_ACCESS_FLAGS mk_vma_flags(VMA_READ_BIT, VMA_WRITE_BIT, VMA_EXEC_BIT)/* * Special vmas that are non-mergable, non-mlock()able.
@@ -585,27 +586,32 @@ enum {* possesses it but the other does not, the merged VMA should nonetheless have * applied to it: * -* VM_SOFTDIRTY - if a VMA is marked soft-dirty, that is has not had its -* references cleared via /proc/$pid/clear_refs, any merged VMA -* should be considered soft-dirty also as it operates at a VMA -* granularity. +* VMA_SOFTDIRTY_BIT - if a VMA is marked soft-dirty, that is has not had its +* references cleared via /proc/$pid/clear_refs, any +* merged VMA should be considered soft-dirty also as it +* operates at a VMA granularity. * -* VM_MAYBE_GUARD - If a VMA may have guard regions in place it implies that -* mapped page tables may contain metadata not described by the -* VMA and thus any merged VMA may also contain this metadata, -* and thus we must make this flag sticky. +* VMA_MAYBE_GUARD_BIT - If a VMA may have guard regions in place it implies +* that mapped page tables may contain metadata not +* described by the VMA and thus any merged VMA may also +* contain this metadata, and thus we must make this flag +* sticky. */ -#define VM_STICKY (VM_SOFTDIRTY | VM_MAYBE_GUARD) +#ifdef CONFIG_MEM_SOFT_DIRTY +#define VMA_STICKY_FLAGS mk_vma_flags(VMA_SOFTDIRTY_BIT, VMA_MAYBE_GUARD_BIT) +#else +#define VMA_STICKY_FLAGS mk_vma_flags(VMA_MAYBE_GUARD_BIT) +#endif /* * VMA flags we ignore for the purposes of merge, i.e. one VMA possessing one * of these flags and the other not does not preclude a merge. * -* VM_STICKY - When merging VMAs, VMA flags must match, unless they are -* 'sticky'. If any sticky flags exist in either VMA, we simply -* set all of them on the merged VMA. +* VMA_STICKY_FLAGS - When merging VMAs, VMA flags must match, unless they +* are 'sticky'. If any sticky flags exist in either VMA, +* we simply set all of them on the merged VMA. */ -#define VM_IGNORE_MERGE VM_STICKY +#define VMA_IGNORE_MERGE_FLAGS VMA_STICKY_FLAGS/* * Flags which should result in page tables being copied on fork. These are
mm/vma.c +36 -12
120 lines not on this page · raw message
tools/testing/vma/include/custom.h +0 -5
@@ -134,8 +134,3 @@ static __always_inline bool vma_flags_same_mask(vma_flags_t *flags, vma_flags_same_mask(flags, mk_vma_flags(__VA_ARGS__))#define VMA_SPECIAL_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_DONTEXPAND_BIT, \ VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT) -#ifdef CONFIG_MEM_SOFT_DIRTY -#define VMA_STICKY_FLAGS mk_vma_flags(VMA_SOFTDIRTY_BIT, VMA_MAYBE_GUARD_BIT) -#else -#define VMA_STICKY_FLAGS mk_vma_flags(VMA_MAYBE_GUARD_BIT) -#endif
tools/testing/vma/include/dup.h +7 -2
@@ -338,6 +338,7 @@ enum {/* VMA basic access permission flags */ #define VM_ACCESS_FLAGS (VM_READ | VM_WRITE | VM_EXEC) +#define VMA_ACCESS_FLAGS mk_vma_flags(VMA_READ_BIT, VMA_WRITE_BIT, VMA_EXEC_BIT)/* * Special vmas that are non-mergable, non-mlock()able.
@@ -363,9 +364,13 @@ enum {#define CAP_IPC_LOCK 14 -#define VM_STICKY (VM_SOFTDIRTY | VM_MAYBE_GUARD) +#ifdef CONFIG_MEM_SOFT_DIRTY +#define VMA_STICKY_FLAGS mk_vma_flags(VMA_SOFTDIRTY_BIT, VMA_MAYBE_GUARD_BIT) +#else +#define VMA_STICKY_FLAGS mk_vma_flags(VMA_MAYBE_GUARD_BIT) +#endif -#define VM_IGNORE_MERGE VM_STICKY +#define VMA_IGNORE_MERGE_FLAGS VMA_STICKY_FLAGS#define VM_COPY_ON_FORK (VM_PFNMAP | VM_MIXEDMAP | VM_UFFD_WP | VM_MAYBE_GUARD)
Signature · 1 line
2.53.0
Re: [PATCH v3 05/23] mm/vma: use new VMA flags for sticky flags logic
Quoted 9 lines · Lorenzo Stoakes (Oracle)
Use the new
vma_flags_tflags implementation to perform the logic around sticky flags and what flags are ignored on VMA merge.We make use of the new
vma_flags_empty(),vma_flags_diff_pair(), andvma_flags_and_mask()functionality.Also update the VMA tests accordingly.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 6/23 tools/testing/vma: fix VMA flag tests
In reply to Lorenzo Stoakes (Oracle)The VMA tests are incorrectly referencing NUM_VMA_FLAGS, which doesn't exist, rather they should reference NUM_VMA_FLAG_BITS.
Additionally, remove the custom-written implementation of __mk_vma_flags() as this means we are not testing the code as present in the kernel, rather add the actual __mk_vma_flags() to dup.h and add #ifdef's to handle declarations differently depending on NUM_VMA_FLAG_BITS.
--- tools/testing/vma/include/custom.h | 19 ------- tools/testing/vma/include/dup.h | 21 ++++++- tools/testing/vma/tests/vma.c | 88 +++++++++++++++++++++++++----- 3 files changed, 92 insertions(+), 36 deletions(-)
tools/testing/vma/include/custom.h +0 -19
@@ -29,8 +29,6 @@ extern unsigned long dac_mmap_min_addr;*/ #define pr_warn_once pr_err -#define pgtable_supports_soft_dirty () 1 -struct anon_vma {struct anon_vma *root;struct rb_root_cached rb_root;
@@ -99,23 +97,6 @@ static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt)refcount_set (&vma->vm_refcnt,0 ); } -static __always_inline vma_flags_t __mk_vma_flags(size_t count, -const vma_flag_t *bits) -{ -vma_flags_t flags; -int i; - -/* -* For testing purposes: allow invalid bit specification so we can -* easily test. -*/ -vma_flags_clear_all (&flags); -for (i =0 ; i < count; i++) -if (bits[i] <NUM_VMA_FLAG_BITS ) -vma_flags_set_flag (&flags, bits[i]); -return flags; -} -static inline unsigned long vma_kernel_pagesize (struct vm_area_struct *vma) {return PAGE_SIZE ;
tools/testing/vma/include/dup.h +18 -3
@@ -854,10 +854,21 @@ static inline void vm_flags_clear(struct vm_area_struct *vma,vma_flags_clear_word (&vma->flags, flags); } -static inline vma_flags_t __mk_vma_flags (size_t count,const vma_flag_t *bits); +static __always_inline vma_flags_t __mk_vma_flags(size_t count, +const vma_flag_t *bits) +{ +vma_flags_t flags; +int i; + +vma_flags_clear_all (&flags); +for (i =0 ; i < count; i++) +vma_flags_set_flag (&flags, bits[i]); + +return flags; +} -#define mk_vma_flags (...) __mk_vma_flags(COUNT_ARGS(__VA_ARGS__), \ - (const vma_flag_t []){__VA_ARGS__}) +#define mk_vma_flags (...) __mk_vma_flags(COUNT_ARGS(__VA_ARGS__), \ + (const vma_flag_t []){__VA_ARGS__}) static __always_inline bool vma_flags_test(const vma_flags_t *flags,vma_flag_t bit)
@@ -1390,3 +1401,7 @@ static inline int get_sysctl_max_map_count(void) {return READ_ONCE (sysctl_max_map_count); } + +#ifndef pgtable_supports_soft_dirty +#define pgtable_supports_soft_dirty () IS_ENABLED(CONFIG_MEM_SOFT_DIRTY) +#endif
tools/testing/vma/tests/vma.c +74 -14
212 lines not on this page · raw message
Signature · 1 line
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 7/23 mm/vma: add append_vma_flags() helper
In reply to Lorenzo Stoakes (Oracle)In order to be able to efficiently combine VMA flag masks with additional VMA flag bits we need to extend the concept introduced in mk_vma_flags() and __mk_vma_flags() by allowing the specification of a VMA flag mask to append VMA flag bits to.
Update __mk_vma_flags() to allow for this and update mk_vma_flags() accordingly, and also provide append_vma_flags() to allow for the caller to specify which VMA flags mask to append to.
Finally, update the VMA flags tests to reflect the change.
--- include/linux/mm.h | 20 ++++++++++++++------ tools/testing/vma/include/dup.h | 14 +++++++------- 2 files changed, 21 insertions(+), 13 deletions(-)
include/linux/mm.h +14 -6
@@ -1047,13 +1047,11 @@ static __always_inline void vma_flags_set_flag(vma_flags_t *flags,__set_bit ((__force int )bit, bitmap); } -static __always_inline vma_flags_t __mk_vma_flags(size_t count, -const vma_flag_t *bits) +static __always_inline vma_flags_t __mk_vma_flags(vma_flags_t flags, +size_t count,const vma_flag_t *bits) { -vma_flags_t flags;int i; -vma_flags_clear_all (&flags);for (i =0 ; i < count; i++)vma_flags_set_flag (&flags, bits[i]);return flags;
@@ -1069,8 +1067,18 @@ static __always_inline vma_flags_t __mk_vma_flags(size_t count,* The compiler cleverly optimises away all of the work and this ends up being * equivalent to aggregating the values manually. */ -#define mk_vma_flags (...) __mk_vma_flags(COUNT_ARGS(__VA_ARGS__), \ - (const vma_flag_t []){__VA_ARGS__}) +#define mk_vma_flags (...) __mk_vma_flags(EMPTY_VMA_FLAGS, \ + COUNT_ARGS(__VA_ARGS__), (const vma_flag_t []){__VA_ARGS__}) + +/* +* Helper macro which acts like mk_vma_flags, only appending to a copy of the +* specified flags rather than establishing new flags. E.g.: +* +* vma_flags_t flags = append_vma_flags(VMA_STACK_DEFAULT_FLAGS, VMA_STACK_BIT, +* VMA_ACCOUNT_BIT); +*/ +#define append_vma_flags (flags, ...) __mk_vma_flags(flags, \ + COUNT_ARGS(__VA_ARGS__), (const vma_flag_t []){__VA_ARGS__})/* * Test whether a specific VMA flag is set, e.g.:
tools/testing/vma/include/dup.h +7 -7
@@ -854,21 +854,21 @@ static inline void vm_flags_clear(struct vm_area_struct *vma,vma_flags_clear_word (&vma->flags, flags); } -static __always_inline vma_flags_t __mk_vma_flags(size_t count, -const vma_flag_t *bits) +static __always_inline vma_flags_t __mk_vma_flags(vma_flags_t flags, +size_t count,const vma_flag_t *bits) { -vma_flags_t flags;int i; -vma_flags_clear_all (&flags);for (i =0 ; i < count; i++)vma_flags_set_flag (&flags, bits[i]); -return flags; } -#define mk_vma_flags (...) __mk_vma_flags(COUNT_ARGS(__VA_ARGS__), \ - (const vma_flag_t []){__VA_ARGS__}) +#define mk_vma_flags (...) __mk_vma_flags(EMPTY_VMA_FLAGS, \ + COUNT_ARGS(__VA_ARGS__), (const vma_flag_t []){__VA_ARGS__}) + +#define append_vma_flags (flags, ...) __mk_vma_flags(flags, \ + COUNT_ARGS(__VA_ARGS__), (const vma_flag_t []){__VA_ARGS__}) static __always_inline bool vma_flags_test(const vma_flags_t *flags,vma_flag_t bit)
Signature · 1 line
2.53.0
Re: [PATCH v3 07/23] mm/vma: add append_vma_flags() helper
Quoted 5 lines · Lorenzo Stoakes (Oracle)
In order to be able to efficiently combine VMA flag masks with additional VMA flag bits we need to extend the concept introduced in
mk_vma_flags()and__mk_vma_flags()by allowing the specification of a VMA flag mask to append VMA flag bits to.
Quoted 7 lines
Update
__mk_vma_flags()to allow for this and updatemk_vma_flags()accordingly, and also provideappend_vma_flags()to allow for the caller to specify which VMA flags mask to append to.Finally, update the VMA flags tests to reflect the change.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 8/23 tools/testing/vma: add simple test for append_vma_flags()
In reply to Lorenzo Stoakes (Oracle)Add a simple test for append_vma_flags() to assert that it behaves as expected.
Additionally, include the VMA_REMAP_FLAGS definition in the VMA tests to allow us to use this value in the testing.
--- tools/testing/vma/include/dup.h | 3 +++ tools/testing/vma/tests/vma.c | 25 +++++++++++++++++++++++++ 2 files changed, 28 insertions(+)
tools/testing/vma/include/dup.h +3 -0
@@ -345,6 +345,9 @@ enum {*/ #define VM_SPECIAL (VM_IO | VM_DONTEXPAND | VM_PFNMAP | VM_MIXEDMAP) +#define VMA_REMAP_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_PFNMAP_BIT, \ + VMA_DONTEXPAND_BIT, VMA_DONTDUMP_BIT) +#define DEFAULT_MAP_WINDOW ((1UL << 47) - PAGE_SIZE)#define TASK_SIZE_LOW DEFAULT_MAP_WINDOW#define TASK_SIZE_MAX DEFAULT_MAP_WINDOW
tools/testing/vma/tests/vma.c +25 -0
@@ -555,6 +555,30 @@ static bool test_vma_flags_and(void)return true; } +/* Ensure append_vma_flags() acts as expected. */ +static bool test_append_vma_flags (void ) +{ +vma_flags_t flags =append_vma_flags (VMA_REMAP_FLAGS ,VMA_READ_BIT , +VMA_WRITE_BIT +#if NUM_VMA_FLAG_BITS > 64 + ,64 ,65 +#endif + ); + +ASSERT_FLAGS_SAME (&flags,VMA_IO_BIT ,VMA_PFNMAP_BIT , +VMA_DONTEXPAND_BIT ,VMA_DONTDUMP_BIT ,VMA_READ_BIT , +VMA_WRITE_BIT +#if NUM_VMA_FLAG_BITS > 64 + ,64 ,65 +#endif + ); + + flags =append_vma_flags (EMPTY_VMA_FLAGS ,VMA_READ_BIT ,VMA_WRITE_BIT ); +ASSERT_FLAGS_SAME (&flags,VMA_READ_BIT ,VMA_WRITE_BIT ); + +return true; +} +static void run_vma_tests (int *num_tests,int *num_fail) {TEST (copy_vma);
@@ -569,4 +593,5 @@ static void run_vma_tests(int *num_tests, int *num_fail)TEST (vma_flags_empty);TEST (vma_flags_diff);TEST (vma_flags_and); +TEST (append_vma_flags); }
Signature · 1 line
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 9/23 mm: unexport vm_brk_flags() and eliminate vm_flags parameter
In reply to Lorenzo Stoakes (Oracle)This function is only used by elf_load(), and that is a static function that doesn't need an exported symbol to invoke an internal function, so un-EXPORT_SYMBOLS() it.
Also, the vm_flags parameter is unnecessary, as we only ever set VM_EXEC, so simply make this parameter a boolean.
While we're here, clean up the mm.h definitions for the various vm_xxx() helpers so we actually specify parameter names and elide the redundant extern's.
--- fs/binfmt_elf.c | 3 +-- include/linux/mm.h | 12 ++++++------ mm/mmap.c | 8 ++------ 3 files changed, 9 insertions(+), 14 deletions(-)
fs/binfmt_elf.c +1 -2
@@ -453,14 +453,13 @@ static unsigned long elf_load(struct file *filep, unsigned long addr, zero_end =ELF_PAGEALIGN (zero_end); error =vm_brk_flags (zero_start, zero_end - zero_start, - prot &PROT_EXEC ?VM_EXEC :0 ); + prot &PROT_EXEC );if (error) map_addr = error; }return map_addr; } -static unsigned long total_mapping_size (const struct elf_phdr *phdr,int nr) {elf_addr_t min_addr =-1 ;
include/linux/mm.h +6 -6
@@ -4005,12 +4005,12 @@ static inline void mm_populate(unsigned long addr, unsigned long len) {}#endif /* This takes the mm semaphore itself */ -extern int __must_check vm_brk_flags (unsigned long ,unsigned long ,unsigned long ); -extern int vm_munmap (unsigned long ,size_t ); -extern unsigned long __must_check vm_mmap (struct file *,unsigned long , -unsigned long ,unsigned long , -unsigned long ,unsigned long ); -extern unsigned long __must_check vm_mmap_shadow_stack (unsigned long addr, +int __must_check vm_brk_flags (unsigned long addr,unsigned long request,bool is_exec); +int vm_munmap (unsigned long start,size_t len); +unsigned long __must_check vm_mmap (struct file *file,unsigned long addr, +unsigned long len,unsigned long prot, +unsigned long flag,unsigned long offset); +unsigned long __must_check vm_mmap_shadow_stack (unsigned long addr,unsigned long len,unsigned long flags);struct vm_unmapped_area_info {
mm/mmap.c +2 -6
@@ -1201,8 +1201,9 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size,return ret; } -int vm_brk_flags (unsigned long addr,unsigned long request,vm_flags_t vm_flags) +int vm_brk_flags (unsigned long addr,unsigned long request,bool is_exec) { +const vm_flags_t vm_flags = is_exec ?VM_EXEC :0 ;struct mm_struct *mm = current->mm;struct vm_area_struct *vma =NULL ;unsigned long len;
@@ -1217,10 +1218,6 @@ int vm_brk_flags(unsigned long addr, unsigned long request, vm_flags_t vm_flags)if (!len)return 0 ; -/* Until we need other flags, refuse anything except VM_EXEC. */ -if ((vm_flags & (~VM_EXEC )) !=0 ) -return -EINVAL ; -if (mmap_write_lock_killable (mm))return -EINTR ;
@@ -1246,7 +1243,6 @@ int vm_brk_flags(unsigned long addr, unsigned long request, vm_flags_t vm_flags)mmap_write_unlock (mm);return ret; } -EXPORT_SYMBOL (vm_brk_flags); static unsigned long tear_down_vmas(struct mm_struct *mm, struct vma_iterator *vmi,
Signature · 1 line
2.53.0
Re: [PATCH v3 09/23] mm: unexport vm_brk_flags() and eliminate vm_flags parameter
Quoted 4 lines · Lorenzo Stoakes (Oracle)
This function is only used by
elf_load(), and that is a static function that doesn't need an exported symbol to invoke an internal function, so un-EXPORT_SYMBOLS() it.
Quoted 8 lines
Also, the
vm_flagsparameter is unnecessary, as we only ever setVM_EXEC, so simply make this parameter a boolean.While we're here, clean up the
mm.hdefinitions for the variousvm_xxx()helpers so we actually specify parameter names and elide the redundant extern's.Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Quoted 83 lines
--- fs/binfmt_elf.c | 3 +-- include/linux/mm.h | 12 ++++++------ mm/mmap.c | 8 ++------ 3 files changed, 9 insertions(+), 14 deletions(-)fs/binfmt_elf.c +1
-2@@ -453,14 +453,13 @@ static unsigned long elf_load(struct file *filep, unsigned long addr, zero_end =ELF_PAGEALIGN (zero_end); error =vm_brk_flags (zero_start, zero_end - zero_start, - prot &PROT_EXEC ?VM_EXEC :0 ); + prot &PROT_EXEC );if (error) map_addr = error; }return map_addr; } -static unsigned long total_mapping_size (const struct elf_phdr *phdr,int nr) {elf_addr_t min_addr =-1 ;include/linux/mm.h +6
-6@@ -4005,12 +4005,12 @@ static inline void mm_populate(unsigned long addr, unsigned long len) {}#endif /* This takes the mm semaphore itself */ -extern int __must_check vm_brk_flags (unsigned long ,unsigned long ,unsigned long ); -extern int vm_munmap (unsigned long ,size_t ); -extern unsigned long __must_check vm_mmap (struct file *,unsigned long , -unsigned long ,unsigned long , -unsigned long ,unsigned long ); -extern unsigned long __must_check vm_mmap_shadow_stack (unsigned long addr, +int __must_check vm_brk_flags (unsigned long addr,unsigned long request,bool is_exec); +int vm_munmap (unsigned long start,size_t len); +unsigned long __must_check vm_mmap (struct file *file,unsigned long addr, +unsigned long len,unsigned long prot, +unsigned long flag,unsigned long offset); +unsigned long __must_check vm_mmap_shadow_stack (unsigned long addr,unsigned long len,unsigned long flags);struct vm_unmapped_area_info {mm/mmap.c +2
-6@@ -1201,8 +1201,9 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size,return ret; } -int vm_brk_flags (unsigned long addr,unsigned long request,vm_flags_t vm_flags) +int vm_brk_flags (unsigned long addr,unsigned long request,bool is_exec) { +const vm_flags_t vm_flags = is_exec ?VM_EXEC :0 ;struct mm_struct *mm = current->mm;struct vm_area_struct *vma =NULL ;unsigned long len;@@ -1217,10 +1218,6 @@ int vm_brk_flags(unsigned long addr, unsigned long request, vm_flags_t vm_flags)if (!len)return 0 ; -/* Until we need other flags, refuse anything except VM_EXEC. */ -if ((vm_flags & (~VM_EXEC )) !=0 ) -return -EINVAL ; -if (mmap_write_lock_killable (mm))return -EINTR ;@@ -1246,7 +1243,6 @@ int vm_brk_flags(unsigned long addr, unsigned long request, vm_flags_t vm_flags)mmap_write_unlock (mm);return ret; } -EXPORT_SYMBOL (vm_brk_flags); static unsigned long tear_down_vmas(struct mm_struct *mm, struct vma_iterator *vmi,
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 10/23 mm/vma: introduce vma_flags_same[_mask/_pair]()
In reply to Lorenzo Stoakes (Oracle)Add helpers to determine if two sets of VMA flags are precisely the same, that is - that every flag set one is set in another, and neither contain any flags not set in the other.
We also introduce vma_flags_same_pair() for cases where we want to compare two sets of VMA flags which are both non-const values.
Also update the VMA tests to reflect the change, we already implicitly test that this functions correctly having used it for testing purposes previously.
--- include/linux/mm.h | 28 ++++++++++++++++++++++++++++ tools/testing/vma/include/custom.h | 11 ----------- tools/testing/vma/include/dup.h | 21 +++++++++++++++++++++ 3 files changed, 49 insertions(+), 11 deletions(-)
include/linux/mm.h +28 -0
@@ -1207,6 +1207,34 @@ static __always_inline vma_flags_t vma_flags_diff_pair(const vma_flags_t *flags,return dst; } +/* Determine if flags and flags_other have precisely the same flags set. */ +static __always_inline bool vma_flags_same_pair(const vma_flags_t *flags, +const vma_flags_t *flags_other) +{ +const unsigned long *bitmap = flags->__vma_flags; +const unsigned long *bitmap_other = flags_other->__vma_flags; + +return bitmap_equal (bitmap, bitmap_other,NUM_VMA_FLAG_BITS ); +} + +/* Determine if flags and flags_other have precisely the same flags set. */ +static __always_inline bool vma_flags_same_mask(const vma_flags_t *flags, +vma_flags_t flags_other) +{ +const unsigned long *bitmap = flags->__vma_flags; +const unsigned long *bitmap_other = flags_other.__vma_flags; + +return bitmap_equal (bitmap, bitmap_other,NUM_VMA_FLAG_BITS ); +} + +/* +* Helper macro to determine if only the specific flags are set, e.g.: +* +* if (vma_flags_same(&flags, VMA_WRITE_BIT) { ... } +*/ +#define vma_flags_same (flags, ...) \ + vma_flags_same_mask(flags, mk_vma_flags(__VA_ARGS__)) +/* * Helper to test that ALL specified flags are set in a VMA. *
tools/testing/vma/include/custom.h +0 -11
@@ -102,16 +102,5 @@ static inline unsigned long vma_kernel_pagesize(struct vm_area_struct *vma)return PAGE_SIZE ; } -/* Place here until needed in the kernel code. */ -static __always_inline bool vma_flags_same_mask(vma_flags_t *flags, -vma_flags_t flags_other) -{ -const unsigned long *bitmap = flags->__vma_flags; -const unsigned long *bitmap_other = flags_other.__vma_flags; - -return bitmap_equal (bitmap, bitmap_other,NUM_VMA_FLAG_BITS ); -} -#define vma_flags_same (flags, ...) \ - vma_flags_same_mask(flags, mk_vma_flags(__VA_ARGS__))#define VMA_SPECIAL_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_DONTEXPAND_BIT, \ VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT)
tools/testing/vma/include/dup.h +21 -0
@@ -954,6 +954,27 @@ static __always_inline vma_flags_t vma_flags_diff_pair(const vma_flags_t *flags,return dst; } +static __always_inline bool vma_flags_same_pair(const vma_flags_t *flags, +const vma_flags_t *flags_other) +{ +const unsigned long *bitmap = flags->__vma_flags; +const unsigned long *bitmap_other = flags_other->__vma_flags; + +return bitmap_equal (bitmap, bitmap_other,NUM_VMA_FLAG_BITS ); +} + +static __always_inline bool vma_flags_same_mask(const vma_flags_t *flags, +vma_flags_t flags_other) +{ +const unsigned long *bitmap = flags->__vma_flags; +const unsigned long *bitmap_other = flags_other.__vma_flags; + +return bitmap_equal (bitmap, bitmap_other,NUM_VMA_FLAG_BITS ); +} + +#define vma_flags_same (flags, ...) \ + vma_flags_same_mask(flags, mk_vma_flags(__VA_ARGS__)) +static inline bool vma_test_all_mask (const struct vm_area_struct *vma,vma_flags_t flags) {
Signature · 1 line
2.53.0
Re: [PATCH v3 10/23] mm/vma: introduce vma_flags_same[_mask/_pair]()
Quoted 4 lines · Lorenzo Stoakes (Oracle)
Add helpers to determine if two sets of VMA flags are precisely the same, that is - that every flag set one is set in another, and neither contain any flags not set in the other.
Quoted 8 lines
We also introduce
vma_flags_same_pair()for cases where we want to compare two sets of VMA flags which are both non-const values.Also update the VMA tests to reflect the change, we already implicitly test that this functions correctly having used it for testing purposes previously.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 11/23 mm/vma: introduce [vma_flags,legacy]_to_[legacy,vma_flags]() helpers
In reply to Lorenzo Stoakes (Oracle)While we are still converting VMA flags from vma_flags_t to vm_flags_t, introduce helpers to convert between the two to allow for iterative development without having to 'change the world' in a single commit'.
Also update VMA flags tests to reflect the change.
Finally, refresh vma_flags_overwrite_word(), vma_flag_overwrite_word_once(), vma_flags_set_word() and vma_flags_clear_word() in the VMA tests to reflect current kernel implementations - this should make no functional difference, but keeps the logic consistent between the two.
--- include/linux/mm_types.h | 26 ++++++++++++++++++++++++ tools/testing/vma/include/dup.h | 36 +++++++++++++++++++++++++++++---- 2 files changed, 58 insertions(+), 4 deletions(-)
include/linux/mm_types.h +26 -0
@@ -1069,6 +1069,18 @@ static __always_inline void vma_flags_clear_all(vma_flags_t *flags)bitmap_zero (flags->__vma_flags,NUM_VMA_FLAG_BITS ); } +/* +* Helper function which converts a vma_flags_t value to a legacy vm_flags_t +* value. This is only valid if the input flags value can be expressed in a +* system word. +* +* Will be removed once the conversion to VMA flags is complete. +*/ +static __always_inline vm_flags_t vma_flags_to_legacy(vma_flags_t flags) +{ +return (vm_flags_t )flags.__vma_flags[0 ]; +} +/* * Copy value to the first system word of VMA flags, non-atomically. *
@@ -1082,6 +1094,20 @@ static inline void vma_flags_overwrite_word(vma_flags_t *flags, unsigned long va bitmap[0 ] = value; } +/* +* Helper function which converts a legacy vm_flags_t value to a vma_flags_t +* value. +* +* Will be removed once the conversion to VMA flags is complete. +*/ +static __always_inline vma_flags_t legacy_to_vma_flags(vm_flags_t flags) +{ +vma_flags_t ret =EMPTY_VMA_FLAGS ; + +vma_flags_overwrite_word (&ret, flags); +return ret; +} +/* * Copy value to the first system word of VMA flags ONCE, non-atomically. *
tools/testing/vma/include/dup.h +32 -4
@@ -766,7 +766,9 @@ static inline bool mm_flags_test(int flag, const struct mm_struct *mm)*/ static inline void vma_flags_overwrite_word (vma_flags_t *flags,unsigned long value) { - *ACCESS_PRIVATE (flags, __vma_flags) = value; +unsigned long *bitmap = flags->__vma_flags; + + bitmap[0 ] = value; }/*
@@ -777,7 +779,7 @@ static inline void vma_flags_overwrite_word(vma_flags_t *flags, unsigned long va*/ static inline void vma_flags_overwrite_word_once (vma_flags_t *flags,unsigned long value) { -unsigned long *bitmap =ACCESS_PRIVATE (flags, __vma_flags); +unsigned long *bitmap = flags->__vma_flags;WRITE_ONCE (*bitmap, value); }
@@ -785,7 +787,7 @@ static inline void vma_flags_overwrite_word_once(vma_flags_t *flags, unsigned lo/* Update the first system word of VMA flags setting bits, non-atomically. */ static inline void vma_flags_set_word (vma_flags_t *flags,unsigned long value) { -unsigned long *bitmap =ACCESS_PRIVATE (flags, __vma_flags); +unsigned long *bitmap = flags->__vma_flags; *bitmap |= value; }
@@ -793,7 +795,7 @@ static inline void vma_flags_set_word(vma_flags_t *flags, unsigned long value)/* Update the first system word of VMA flags clearing bits, non-atomically. */ static inline void vma_flags_clear_word (vma_flags_t *flags,unsigned long value) { -unsigned long *bitmap =ACCESS_PRIVATE (flags, __vma_flags); +unsigned long *bitmap = flags->__vma_flags; *bitmap &= ~value; }
@@ -803,6 +805,32 @@ static __always_inline void vma_flags_clear_all(vma_flags_t *flags)bitmap_zero (ACCESS_PRIVATE (flags, __vma_flags),NUM_VMA_FLAG_BITS ); } +/* +* Helper function which converts a vma_flags_t value to a legacy vm_flags_t +* value. This is only valid if the input flags value can be expressed in a +* system word. +* +* Will be removed once the conversion to VMA flags is complete. +*/ +static __always_inline vm_flags_t vma_flags_to_legacy(vma_flags_t flags) +{ +return (vm_flags_t )flags.__vma_flags[0 ]; +} + +/* +* Helper function which converts a legacy vm_flags_t value to a vma_flags_t +* value. +* +* Will be removed once the conversion to VMA flags is complete. +*/ +static __always_inline vma_flags_t legacy_to_vma_flags(vm_flags_t flags) +{ +vma_flags_t ret =EMPTY_VMA_FLAGS ; + +vma_flags_overwrite_word (&ret, flags); +return ret; +} + static __always_inline void vma_flags_set_flag(vma_flags_t *flags,vma_flag_t bit) {
Signature · 1 line
2.53.0
Re: [PATCH v3 11/23] mm/vma: introduce [vma_flags,legacy]_to_[legacy,vma_flags]() helpers
Quoted 6 lines · Lorenzo Stoakes (Oracle)
While we are still converting VMA flags from
vma_flags_ttovm_flags_t, introduce helpers to convert between the two to allow for iterative development without having to 'change the world' in a single commit'.Also update VMA flags tests to reflect the change.
Quoted 7 lines
Finally, refresh
vma_flags_overwrite_word(),vma_flag_overwrite_word_once(),vma_flags_set_word()andvma_flags_clear_word()in the VMA tests to reflect current kernel implementations - this should make no functional difference, but keeps the logic consistent between the two.Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 12/23 tools/testing/vma: test that legacy flag helpers work correctly
In reply to Lorenzo Stoakes (Oracle)Update the existing compare_legacy_flags() predicate function to assert that legacy_to_vma_flags() and vma_flags_to_legacy() behave as expected.
--- tools/testing/vma/tests/vma.c | 6 ++++++ 1 file changed, 6 insertions(+)
tools/testing/vma/tests/vma.c +6 -0
@@ -5,6 +5,7 @@ static bool compare_legacy_flags(vm_flags_t legacy_flags, vma_flags_t flags)const unsigned long legacy_val = legacy_flags;/* The lower word should contain the precise same value. */ const unsigned long flags_lower = flags.__vma_flags[0 ]; +vma_flags_t converted_flags;#if NUM_VMA_FLAG_BITS >BITS_PER_LONG int i;
@@ -17,6 +18,11 @@ static bool compare_legacy_flags(vm_flags_t legacy_flags, vma_flags_t flags)static_assert (sizeof (legacy_flags) ==sizeof (unsigned long )); +/* Assert that legacy flag helpers work correctly. */ + converted_flags =legacy_to_vma_flags (legacy_flags); +ASSERT_FLAGS_SAME_MASK (&converted_flags, flags); +ASSERT_EQ (vma_flags_to_legacy (flags), legacy_flags); +return legacy_val == flags_lower; }
Signature · 1 line
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 13/23 mm/vma: introduce vma_test[_any[_mask]](), and make inlining consistent
In reply to Lorenzo Stoakes (Oracle)Introduce helper functions and macros to make it convenient to test flags and flag masks for VMAs, specifically:
* vma_test() - determine if a single VMA flag is set in a VMA.
* vma_test_any_mask() - determine if any flags in a vma_flags_t value are set in a VMA.
* vma_test_any() - Helper macro to test if any of specific flags are set.
Also, there are a mix of 'inline's and '__always_inline's in VMA helper function declarations, update to consistently use __always_inline.
Finally, update the VMA tests to reflect the changes.
--- include/linux/mm.h | 49 +++++++++++++++++++++----- include/linux/mm_types.h | 12 ++++--- tools/testing/vma/include/dup.h | 61 +++++++++++++++++++++------------ 3 files changed, 88 insertions(+), 34 deletions(-)
include/linux/mm.h +40 -9
@@ -999,7 +999,8 @@ static inline void vm_flags_mod(struct vm_area_struct *vma,__vm_flags_mod (vma, set, clear); } -static inline bool __vma_atomic_valid_flag (struct vm_area_struct *vma,vma_flag_t bit) +static __always_inline bool __vma_atomic_valid_flag(struct vm_area_struct *vma, +vma_flag_t bit) {const vm_flags_t mask =BIT ((__force int )bit);
@@ -1014,7 +1015,8 @@ static inline bool __vma_atomic_valid_flag(struct vm_area_struct *vma, vma_flag_* Set VMA flag atomically. Requires only VMA/mmap read lock. Only specific * valid flags are allowed to do this. */ -static inline void vma_set_atomic_flag (struct vm_area_struct *vma,vma_flag_t bit) +static __always_inline void vma_set_atomic_flag(struct vm_area_struct *vma, +vma_flag_t bit) {unsigned long *bitmap = vma->flags.__vma_flags;
@@ -1030,7 +1032,8 @@ static inline void vma_set_atomic_flag(struct vm_area_struct *vma, vma_flag_t bi* This is necessarily racey, so callers must ensure that serialisation is * achieved through some other means, or that races are permissible. */ -static inline bool vma_test_atomic_flag (struct vm_area_struct *vma,vma_flag_t bit) +static __always_inline bool vma_test_atomic_flag(struct vm_area_struct *vma, +vma_flag_t bit) {if (__vma_atomic_valid_flag (vma, bit))return test_bit ((__force int )bit, &vma->vm_flags);
@@ -1235,13 +1238,41 @@ static __always_inline bool vma_flags_same_mask(const vma_flags_t *flags,#define vma_flags_same (flags, ...) \ vma_flags_same_mask(flags, mk_vma_flags(__VA_ARGS__)) +/* +* Test whether a specific flag in the VMA is set, e.g.: +* +* if (vma_test(vma, VMA_READ_BIT)) { ... } +*/ +static __always_inline bool vma_test(const struct vm_area_struct *vma, +vma_flag_t bit) +{ +return vma_flags_test (&vma->flags, bit); +} + +/* Helper to test any VMA flags in a VMA . */ +static __always_inline bool vma_test_any_mask(const struct vm_area_struct *vma, +vma_flags_t flags) +{ +return vma_flags_test_any_mask (&vma->flags, flags); +} + +/* +* Helper macro for testing whether any VMA flags are set in a VMA, +* e.g.: +* +* if (vma_test_any(vma, VMA_IO_BIT, VMA_PFNMAP_BIT, +* VMA_DONTEXPAND_BIT, VMA_DONTDUMP_BIT)) { ... } +*/ +#define vma_test_any (vma, ...) \ + vma_test_any_mask(vma, mk_vma_flags(__VA_ARGS__)) +/* * Helper to test that ALL specified flags are set in a VMA. * * Note: appropriate locks must be held, this function does not acquire them for * you. */ -static inline bool vma_test_all_mask (const struct vm_area_struct *vma, +static __always_inline bool vma_test_all_mask(const struct vm_area_struct *vma,vma_flags_t flags) {return vma_flags_test_all_mask (&vma->flags, flags);
@@ -1261,7 +1292,7 @@ static inline bool vma_test_all_mask(const struct vm_area_struct *vma,* Note: appropriate locks must be held, this function does not acquire them for * you. */ -static inline void vma_set_flags_mask (struct vm_area_struct *vma, +static __always_inline void vma_set_flags_mask(struct vm_area_struct *vma,vma_flags_t flags) {vma_flags_set_mask (&vma->flags, flags);
@@ -1291,7 +1322,7 @@ static __always_inline bool vma_desc_test(const struct vm_area_desc *desc, }/* Helper to test any VMA flags in a VMA descriptor. */ -static inline bool vma_desc_test_any_mask (const struct vm_area_desc *desc, +static __always_inline bool vma_desc_test_any_mask(const struct vm_area_desc *desc,vma_flags_t flags) {return vma_flags_test_any_mask (&desc->vma_flags, flags);
@@ -1308,7 +1339,7 @@ static inline bool vma_desc_test_any_mask(const struct vm_area_desc *desc,vma_desc_test_any_mask (desc,mk_vma_flags (__VA_ARGS__)) /* Helper to test all VMA flags in a VMA descriptor. */ -static inline bool vma_desc_test_all_mask (const struct vm_area_desc *desc, +static __always_inline bool vma_desc_test_all_mask(const struct vm_area_desc *desc,vma_flags_t flags) {return vma_flags_test_all_mask (&desc->vma_flags, flags);
@@ -1324,7 +1355,7 @@ static inline bool vma_desc_test_all_mask(const struct vm_area_desc *desc,vma_desc_test_all_mask (desc,mk_vma_flags (__VA_ARGS__)) /* Helper to set all VMA flags in a VMA descriptor. */ -static inline void vma_desc_set_flags_mask (struct vm_area_desc *desc, +static __always_inline void vma_desc_set_flags_mask(struct vm_area_desc *desc,vma_flags_t flags) {vma_flags_set_mask (&desc->vma_flags, flags);
@@ -1341,7 +1372,7 @@ static inline void vma_desc_set_flags_mask(struct vm_area_desc *desc,vma_desc_set_flags_mask (desc,mk_vma_flags (__VA_ARGS__)) /* Helper to clear all VMA flags in a VMA descriptor. */ -static inline void vma_desc_clear_flags_mask (struct vm_area_desc *desc, +static __always_inline void vma_desc_clear_flags_mask(struct vm_area_desc *desc,vma_flags_t flags) {vma_flags_clear_mask (&desc->vma_flags, flags);
include/linux/mm_types.h +8 -4
@@ -1087,7 +1087,8 @@ static __always_inline vm_flags_t vma_flags_to_legacy(vma_flags_t flags)* IMPORTANT: This does not overwrite bytes past the first system word. The * caller must account for this. */ -static inline void vma_flags_overwrite_word (vma_flags_t *flags,unsigned long value) +static __always_inline void vma_flags_overwrite_word(vma_flags_t *flags, +unsigned long value) {unsigned long *bitmap = flags->__vma_flags;
@@ -1114,7 +1115,8 @@ static __always_inline vma_flags_t legacy_to_vma_flags(vm_flags_t flags)* IMPORTANT: This does not overwrite bytes past the first system word. The * caller must account for this. */ -static inline void vma_flags_overwrite_word_once (vma_flags_t *flags,unsigned long value) +static __always_inline void vma_flags_overwrite_word_once(vma_flags_t *flags, +unsigned long value) {unsigned long *bitmap = flags->__vma_flags;
@@ -1122,7 +1124,8 @@ static inline void vma_flags_overwrite_word_once(vma_flags_t *flags, unsigned lo }/* Update the first system word of VMA flags setting bits, non-atomically. */ -static inline void vma_flags_set_word (vma_flags_t *flags,unsigned long value) +static __always_inline void vma_flags_set_word(vma_flags_t *flags, +unsigned long value) {unsigned long *bitmap = flags->__vma_flags;
@@ -1130,7 +1133,8 @@ static inline void vma_flags_set_word(vma_flags_t *flags, unsigned long value) }/* Update the first system word of VMA flags clearing bits, non-atomically. */ -static inline void vma_flags_clear_word (vma_flags_t *flags,unsigned long value) +static __always_inline void vma_flags_clear_word(vma_flags_t *flags, +unsigned long value) {unsigned long *bitmap = flags->__vma_flags;
tools/testing/vma/include/dup.h +40 -21
128 lines not on this page · raw message
Signature · 1 line
2.53.0
Re: [PATCH v3 13/23] mm/vma: introduce vma_test[_any[_mask]](), and make inlining consistent
Quoted 8 lines · Lorenzo Stoakes (Oracle)
Introduce helper functions and macros to make it convenient to test flags and flag masks for VMAs, specifically:
*
vma_test()- determine if a single VMA flag is set in a VMA.
*vma_test_any_mask()- determine if any flags in avma_flags_tvalue are set in a VMA.
*vma_test_any()- Helper macro to test if any of specific flags are set.
Quoted 6 lines
Also, there are a mix of 'inline's and '__always_inline's in VMA helper function declarations, update to consistently use
__always_inline.Finally, update the VMA tests to reflect the changes.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 14/23 tools/testing/vma: update VMA flag tests to test vma_test[_any_mask]()
In reply to Lorenzo Stoakes (Oracle)Update the existing test logic to assert that vma_test(), vma_test_any() and vma_test_any_mask() (implicitly tested via vma_test_any()) are functioning correctly.
We already have tests for other variants like this, so it's simply a matter of expanding those tests to also include tests for the VMA-specific helpers.
--- tools/testing/vma/tests/vma.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
tools/testing/vma/tests/vma.c +13 -6
@@ -183,13 +183,18 @@ static bool test_vma_flags_test(void)struct vm_area_desc desc = { .vma_flags = flags, }; +struct vm_area_struct vma = { + .flags = flags, + };#define do_test (_flag) \ ASSERT_TRUE(vma_flags_test(&flags, _flag)); \ + ASSERT_TRUE(vma_test(&vma, _flag)); \ ASSERT_TRUE(vma_desc_test(&desc, _flag))#define do_test_false (_flag) \ ASSERT_FALSE(vma_flags_test(&flags, _flag)); \ + ASSERT_FALSE(vma_test(&vma, _flag)); \ ASSERT_FALSE(vma_desc_test(&desc, _flag))do_test (VMA_READ_BIT );
@@ -219,15 +224,17 @@ static bool test_vma_flags_test_any(void) , 64, 65#endif ); -struct vm_area_struct vma; -struct vm_area_desc desc; - - vma.flags = flags; - desc.vma_flags = flags; +struct vm_area_struct vma = { + .flags = flags, + }; +struct vm_area_desc desc = { + .vma_flags = flags, + };#define do_test (...) \ ASSERT_TRUE(vma_flags_test_any(&flags, __VA_ARGS__)); \ - ASSERT_TRUE(vma_desc_test_any(&desc, __VA_ARGS__)) + ASSERT_TRUE(vma_desc_test_any(&desc, __VA_ARGS__)); \ + ASSERT_TRUE(vma_test_any(&vma, __VA_ARGS__));#define do_test_all_true (...) \ ASSERT_TRUE(vma_flags_test_all(&flags, __VA_ARGS__)); \
Signature · 1 line
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 15/23 mm: introduce vma_flags_count() and vma[_flags]_test_single_mask()
In reply to Lorenzo Stoakes (Oracle)vma_flags_count() determines how many bits are set in VMA flags, using bitmap_weight().
vma_flags_test_single_mask() determines if a vma_flags_t set of flags contains a single flag specified as another vma_flags_t value, or if the sought flag mask is empty, it is defined to return false.
This is useful when we want to declare a VMA flag as optionally a single flag in a mask or empty depending on kernel configuration.
This allows us to have VM_NONE-like semantics when checking whether the flag is set.
In a subsequent patch, we introduce the use of VMA_DROPPABLE of type vma_flags_t using precisely these semantics.
It would be actively confusing to use vma_flags_test_any_single_mask() for this (and vma_flags_test_all_mask() is not correct to use here, as it trivially returns true when tested against an empty vma flags mask).
We introduce vma_flags_count() to be able to assert that the compared flag mask is singular or empty, checked when CONFIG_DEBUG_VM is enabled.
Also update the VMA tests as part of this change.
--- include/linux/mm.h | 46 ++++++++++++++++++++++++++++++ tools/testing/vma/include/custom.h | 6 ---- tools/testing/vma/include/dup.h | 21 ++++++++++++++ tools/testing/vma/vma_internal.h | 6 ++++ 4 files changed, 73 insertions(+), 6 deletions(-)
include/linux/mm.h +46 -0
@@ -1083,6 +1083,14 @@ static __always_inline vma_flags_t __mk_vma_flags(vma_flags_t flags,#define append_vma_flags (flags, ...) __mk_vma_flags(flags, \ COUNT_ARGS(__VA_ARGS__), (const vma_flag_t []){__VA_ARGS__}) +/* Calculates the number of set bits in the specified VMA flags. */ +static __always_inline int vma_flags_count(const vma_flags_t *flags) +{ +const unsigned long *bitmap = flags->__vma_flags; + +return bitmap_weight (bitmap,NUM_VMA_FLAG_BITS ); +} +/* * Test whether a specific VMA flag is set, e.g.: *
@@ -1158,6 +1166,26 @@ static __always_inline bool vma_flags_test_all_mask(const vma_flags_t *flags,#define vma_flags_test_all (flags, ...) \ vma_flags_test_all_mask(flags, mk_vma_flags(__VA_ARGS__)) +/* +* Helper to test that a flag mask of type vma_flags_t has a SINGLE flag set +* (returning false if flagmask has no flags set). +* +* This is defined to make the semantics clearer when testing an optionally +* defined VMA flags mask, e.g.: +* +* if (vma_flags_test_single_mask(&flags, VMA_DROPPABLE)) { ... } +* +* When VMA_DROPPABLE is defined if available, or set to EMPTY_VMA_FLAGS +* otherwise. +*/ +static __always_inline bool vma_flags_test_single_mask(const vma_flags_t *flags, +vma_flags_t flagmask) +{ +VM_WARN_ON_ONCE (vma_flags_count (&flagmask) >1 ); + +return vma_flags_test_any_mask (flags, flagmask); +} +/* Set each of the to_set flags in flags, non-atomically. */ static __always_inline void vma_flags_set_mask(vma_flags_t *flags,vma_flags_t to_set)
@@ -1286,6 +1314,24 @@ static __always_inline bool vma_test_all_mask(const struct vm_area_struct *vma,#define vma_test_all (vma, ...) \ vma_test_all_mask(vma, mk_vma_flags(__VA_ARGS__)) +/* +* Helper to test that a flag mask of type vma_flags_t has a SINGLE flag set +* (returning false if flagmask has no flags set). +* +* This is useful when a flag needs to be either defined or not depending upon +* kernel configuration, e.g.: +* +* if (vma_test_single_mask(vma, VMA_DROPPABLE)) { ... } +* +* When VMA_DROPPABLE is defined if available, or set to EMPTY_VMA_FLAGS +* otherwise. +*/ +static __always_inline bool +vma_test_single_mask (const struct vm_area_struct *vma,vma_flags_t flagmask) +{ +return vma_flags_test_single_mask (&vma->flags, flagmask); +} +/* * Helper to set all VMA flags in a VMA. *
tools/testing/vma/include/custom.h +0 -6
@@ -15,12 +15,6 @@ extern unsigned long dac_mmap_min_addr;#define dac_mmap_min_addr 0UL#endif -#define VM_WARN_ON (_expr) (WARN_ON(_expr)) -#define VM_WARN_ON_ONCE (_expr) (WARN_ON_ONCE(_expr)) -#define VM_WARN_ON_VMG (_expr, _vmg) (WARN_ON(_expr)) -#define VM_BUG_ON (_expr) (BUG_ON(_expr)) -#define VM_BUG_ON_VMA (_expr, _vma) (BUG_ON(_expr)) -#define TASK_SIZE ((1ul << 47)-PAGE_SIZE)/*
tools/testing/vma/include/dup.h +21 -0
@@ -905,6 +905,13 @@ static __always_inline vma_flags_t __mk_vma_flags(vma_flags_t flags,#define append_vma_flags (flags, ...) __mk_vma_flags(flags, \ COUNT_ARGS(__VA_ARGS__), (const vma_flag_t []){__VA_ARGS__}) +static __always_inline int vma_flags_count(const vma_flags_t *flags) +{ +const unsigned long *bitmap = flags->__vma_flags; + +return bitmap_weight (bitmap,NUM_VMA_FLAG_BITS ); +} + static __always_inline bool vma_flags_test(const vma_flags_t *flags,vma_flag_t bit) {
@@ -952,6 +959,14 @@ static __always_inline bool vma_flags_test_all_mask(const vma_flags_t *flags,#define vma_flags_test_all (flags, ...) \ vma_flags_test_all_mask(flags, mk_vma_flags(__VA_ARGS__)) +static __always_inline bool vma_flags_test_single_mask(const vma_flags_t *flags, +vma_flags_t flagmask) +{ +VM_WARN_ON_ONCE (vma_flags_count (&flagmask) >1 ); + +return vma_flags_test_any_mask (flags, flagmask); +} + static __always_inline void vma_flags_set_mask(vma_flags_t *flags, vma_flags_t to_set) {unsigned long *bitmap = flags->__vma_flags;
@@ -1031,6 +1046,12 @@ static __always_inline bool vma_test_all_mask(const struct vm_area_struct *vma,#define vma_test_all (vma, ...) \ vma_test_all_mask(vma, mk_vma_flags(__VA_ARGS__)) +static __always_inline bool +vma_test_single_mask (const struct vm_area_struct *vma,vma_flags_t flagmask) +{ +return vma_flags_test_single_mask (&vma->flags, flagmask); +} + static __always_inline void vma_set_flags_mask(struct vm_area_struct *vma,vma_flags_t flags) {
tools/testing/vma/vma_internal.h +6 -0
@@ -51,6 +51,12 @@ typedef unsigned long pgprotval_t;typedef struct pgprot {pgprotval_t pgprot; }pgprot_t ;typedef __bitwise unsigned int vm_fault_t ; +#define VM_WARN_ON (_expr) (WARN_ON(_expr)) +#define VM_WARN_ON_ONCE (_expr) (WARN_ON_ONCE(_expr)) +#define VM_WARN_ON_VMG (_expr, _vmg) (WARN_ON(_expr)) +#define VM_BUG_ON (_expr) (BUG_ON(_expr)) +#define VM_BUG_ON_VMA (_expr, _vma) (BUG_ON(_expr)) +#include "include/stubs.h" #include "include/dup.h" #include "include/custom.h"
Signature · 1 line
2.53.0
Re: [PATCH v3 15/23] mm: introduce vma_flags_count() and vma[_flags]_test_single_mask()
Quoted 20 lines · Lorenzo Stoakes (Oracle)
vma_flags_count()determines how many bits are set in VMA flags, usingbitmap_weight().
vma_flags_test_single_mask()determines if avma_flags_tset of flags contains a single flag specified as anothervma_flags_tvalue, or if the sought flag mask is empty, it is defined to return false.This is useful when we want to declare a VMA flag as optionally a single flag in a mask or empty depending on kernel configuration.
This allows us to have VM_NONE-like semantics when checking whether the flag is set.
In a subsequent patch, we introduce the use of
VMA_DROPPABLEof typevma_flags_tusing precisely these semantics.It would be actively confusing to use
vma_flags_test_any_single_mask()for this (andvma_flags_test_all_mask()is not correct to use here, as it trivially returns true when tested against an empty vma flags mask).
Quoted 6 lines
We introduce
vma_flags_count()to be able to assert that the compared flag mask is singular or empty, checked whenCONFIG_DEBUG_VMis enabled.Also update the VMA tests as part of this change.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 16/23 tools/testing/vma: test vma_flags_count,vma[_flags]_test_single_mask
In reply to Lorenzo Stoakes (Oracle)Update the VMA tests to assert that vma_flags_count() behaves as expected, as well as vma_flags_test_single_mask() and vma_test_single_mask().
For the test functions we can simply update the existing vma_test(), et al. test to also test the single_mask variants.
We also add some explicit testing of an empty VMA flag to this test to ensure this is handled properly.
In order to test vma_flags_count() we simply take an existing set of flags and gradually remove flags ensuring the count remains as expected throughout.
We also update the vma[_flags]_test_all() tests to make clear the semantics that we expect vma[_flags]_test_all(..., EMPTY_VMA_FLAGS) to return true, as trivially, all flags of none are always set in VMA flags.
--- tools/testing/vma/tests/vma.c | 63 ++++++++++++++++++++++++++++++----- 1 file changed, 54 insertions(+), 9 deletions(-)
tools/testing/vma/tests/vma.c +54 -9
@@ -174,10 +174,10 @@ static bool test_vma_flags_word(void)/* Ensure that vma_flags_test() and friends works correctly. */ static bool test_vma_flags_test (void ) { -const vma_flags_t flags =mk_vma_flags (VMA_READ_BIT ,VMA_WRITE_BIT , -VMA_EXEC_BIT +vma_flags_t flags =mk_vma_flags (VMA_READ_BIT ,VMA_WRITE_BIT , +VMA_EXEC_BIT #if NUM_VMA_FLAG_BITS > 64 - ,64 ,65 + ,64 ,65 #endif );struct vm_area_desc desc = {
@@ -187,14 +187,18 @@ static bool test_vma_flags_test(void) .flags = flags, }; -#define do_test (_flag) \ - ASSERT_TRUE(vma_flags_test(&flags, _flag)); \ - ASSERT_TRUE(vma_test(&vma, _flag)); \ +#define do_test (_flag) \ + ASSERT_TRUE(vma_flags_test(&flags, _flag)); \ + ASSERT_TRUE(vma_flags_test_single_mask(&flags, mk_vma_flags(_flag))); \ + ASSERT_TRUE(vma_test(&vma, _flag)); \ + ASSERT_TRUE(vma_test_single_mask(&vma, mk_vma_flags(_flag))); \ ASSERT_TRUE(vma_desc_test(&desc, _flag)) -#define do_test_false (_flag) \ - ASSERT_FALSE(vma_flags_test(&flags, _flag)); \ - ASSERT_FALSE(vma_test(&vma, _flag)); \ +#define do_test_false (_flag) \ + ASSERT_FALSE(vma_flags_test(&flags, _flag)); \ + ASSERT_FALSE(vma_flags_test_single_mask(&flags, mk_vma_flags(_flag))); \ + ASSERT_FALSE(vma_test(&vma, _flag)); \ + ASSERT_FALSE(vma_test_single_mask(&vma, mk_vma_flags(_flag))); \ ASSERT_FALSE(vma_desc_test(&desc, _flag))do_test (VMA_READ_BIT );
@@ -212,6 +216,15 @@ static bool test_vma_flags_test(void)#undef do_test#undef do_test_false +/* We define the _single_mask() variants to return false if empty. */ +ASSERT_FALSE (vma_flags_test_single_mask (&flags,EMPTY_VMA_FLAGS )); +ASSERT_FALSE (vma_test_single_mask (&vma,EMPTY_VMA_FLAGS )); +/* Even when both flags and tested flag mask are empty! */ + flags =EMPTY_VMA_FLAGS ; + vma.flags =EMPTY_VMA_FLAGS ; +ASSERT_FALSE (vma_flags_test_single_mask (&flags,EMPTY_VMA_FLAGS )); +ASSERT_FALSE (vma_test_single_mask (&vma,EMPTY_VMA_FLAGS )); +return true; }
@@ -309,6 +322,10 @@ static bool test_vma_flags_test_any(void)do_test (VMA_READ_BIT ,VMA_WRITE_BIT ,VMA_EXEC_BIT ,64 ,65 );#endif +/* Testing all flags against none trivially succeeds. */ +ASSERT_TRUE (vma_flags_test_all_mask (&flags,EMPTY_VMA_FLAGS )); +ASSERT_TRUE (vma_test_all_mask (&vma,EMPTY_VMA_FLAGS )); +#undef do_test#undef do_test_all_true#undef do_test_all_false
@@ -592,6 +609,33 @@ static bool test_append_vma_flags(void)return true; } +/* Assert that vma_flags_count() behaves as expected. */ +static bool test_vma_flags_count (void ) +{ +vma_flags_t flags =mk_vma_flags (VMA_READ_BIT ,VMA_WRITE_BIT , +VMA_EXEC_BIT +#if NUM_VMA_FLAG_BITS > 64 + ,64 ,65 +#endif + ); + +#if NUM_VMA_FLAG_BITS >64 +ASSERT_EQ (vma_flags_count (&flags),5 ); +vma_flags_clear (&flags,64 ); +ASSERT_EQ (vma_flags_count (&flags),4 ); +vma_flags_clear (&flags,65 ); +#endif +ASSERT_EQ (vma_flags_count (&flags),3 ); +vma_flags_clear (&flags,VMA_EXEC_BIT ); +ASSERT_EQ (vma_flags_count (&flags),2 ); +vma_flags_clear (&flags,VMA_WRITE_BIT ); +ASSERT_EQ (vma_flags_count (&flags),1 ); +vma_flags_clear (&flags,VMA_READ_BIT ); +ASSERT_EQ (vma_flags_count (&flags),0 ); + +return true; +} +static void run_vma_tests (int *num_tests,int *num_fail) {TEST (copy_vma);
@@ -607,4 +651,5 @@ static void run_vma_tests(int *num_tests, int *num_fail)TEST (vma_flags_diff);TEST (vma_flags_and);TEST (append_vma_flags); +TEST (vma_flags_count); }
Signature · 1 line
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 17/23 mm: convert do_brk_flags() to use vma_flags_t
In reply to Lorenzo Stoakes (Oracle)In order to be able to do this, we need to change VM_DATA_DEFAULT_FLAGS and friends and update the architecture-specific definitions also.
We then have to update some KSM logic to handle VMA flags, and introduce VMA_STACK_FLAGS to define the vma_flags_t equivalent of VM_STACK_FLAGS.
We also introduce two helper functions for use during the time we are converting legacy flags to vma_flags_t values - vma_flags_to_legacy() and legacy_to_vma_flags().
This enables us to iteratively make changes to break these changes up into separate parts.
We use these explicitly here to keep VM_STACK_FLAGS around for certain users which need to maintain the legacy vm_flags_t values for the time being.
We are no longer able to rely on the simple VM_xxx being set to zero if the feature is not enabled, so in the case of VM_DROPPABLE we introduce VMA_DROPPABLE as the vma_flags_t equivalent, which is set to EMPTY_VMA_FLAGS if the droppable flag is not available.
While we're here, we make the description of do_brk_flags() into a kdoc comment, as it almost was already.
We use vma_flags_to_legacy() to not need to update the vm_get_page_prot() logic as this time.
Note that in create_init_stack_vma() we have to replace the BUILD_BUG_ON() with a VM_WARN_ON_ONCE() as the tested values are no longer build time available.
We also update mprotect_fixup() to use VMA flags where possible, though we have to live with a little duplication between vm_flags_t and vma_flags_t values for the time being until further conversions are made.
Finally, we update the VMA tests to reflect these changes.
--- arch/arc/include/asm/page.h | 2 +- arch/arm/include/asm/page.h | 2 +- arch/arm64/include/asm/page.h | 7 ++++- arch/hexagon/include/asm/page.h | 2 +- arch/loongarch/include/asm/page.h | 2 +- arch/mips/include/asm/page.h | 2 +- arch/nios2/include/asm/page.h | 2 +- arch/powerpc/include/asm/page.h | 4 +-- arch/powerpc/include/asm/page_32.h | 2 +- arch/powerpc/include/asm/page_64.h | 12 ++++---- arch/riscv/include/asm/page.h | 2 +- arch/s390/include/asm/page.h | 2 +- arch/x86/include/asm/page_types.h | 2 +- arch/x86/um/asm/vm-flags.h | 4 +-- include/linux/ksm.h | 10 +++---- include/linux/mm.h | 47 ++++++++++++++++++------------ mm/internal.h | 3 ++ mm/ksm.c | 43 ++++++++++++++------------- mm/mmap.c | 13 +++++---- mm/mprotect.c | 46 +++++++++++++++++------------ mm/mremap.c | 6 ++-- mm/vma.c | 34 +++++++++++---------- mm/vma.h | 14 +++++++-- mm/vma_exec.c | 5 ++-- security/selinux/hooks.c | 4 ++- tools/testing/vma/include/custom.h | 3 -- tools/testing/vma/include/dup.h | 42 ++++++++++++++------------ tools/testing/vma/include/stubs.h | 9 +++--- tools/testing/vma/tests/merge.c | 3 +- 29 files changed, 190 insertions(+), 139 deletions(-)
arch/arc/include/asm/page.h +1 -1
@@ -131,7 +131,7 @@ static inline unsigned long virt_to_pfn(const void *kaddr)#define virt_addr_valid (kaddr) pfn_valid(virt_to_pfn(kaddr))/* Default Permissions for stack/heaps pages (Non Executable) */ -#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_NON_EXEC +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_NON_EXEC#define WANT_PAGE_VIRTUAL 1
arch/arm/include/asm/page.h +1 -1
@@ -184,7 +184,7 @@ extern int pfn_valid(unsigned long);#include <asm/memory.h> -#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_TSK_EXEC +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_TSK_EXEC#include <asm-generic/getorder.h> #include <asm-generic/memory_model.h>
arch/arm64/include/asm/page.h +6 -1
@@ -46,7 +46,12 @@ int pfn_is_map_memory(unsigned long pfn);#endif /* !__ASSEMBLER__ */ -#define VM_DATA_DEFAULT_FLAGS (VM_DATA_FLAGS_TSK_EXEC | VM_MTE_ALLOWED) +#ifdef CONFIG_ARM64_MTE +#define VMA_DATA_DEFAULT_FLAGS append_vma_flags(VMA_DATA_FLAGS_TSK_EXEC, \ + VMA_MTE_ALLOWED_BIT) +#else +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_TSK_EXEC +#endif #include <asm-generic/getorder.h>
arch/hexagon/include/asm/page.h +1 -1
@@ -90,7 +90,7 @@ struct page;#define virt_to_page (kaddr) pfn_to_page(PFN_DOWN(__pa(kaddr)))/* Default vm area behavior is non-executable. */ -#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_NON_EXEC +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_NON_EXEC#define virt_addr_valid (kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
arch/loongarch/include/asm/page.h +1 -1
@@ -104,7 +104,7 @@ struct page *tlb_virt_to_page(unsigned long kaddr);extern int __virt_addr_valid (volatile void *kaddr);#define virt_addr_valid (kaddr) __virt_addr_valid((volatile void *)(kaddr)) -#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_TSK_EXEC +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_TSK_EXEC#include <asm-generic/memory_model.h> #include <asm-generic/getorder.h>
arch/mips/include/asm/page.h +1 -1
@@ -213,7 +213,7 @@ extern bool __virt_addr_valid(const volatile void *kaddr);#define virt_addr_valid (kaddr) \ __virt_addr_valid((const volatile void *) (kaddr)) -#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_TSK_EXEC +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_TSK_EXECextern unsigned long __kaslr_offset;static inline unsigned long kaslr_offset (void )
arch/nios2/include/asm/page.h +1 -1
@@ -85,7 +85,7 @@ extern struct page *mem_map;# define virt_to_page (vaddr) pfn_to_page(PFN_DOWN(virt_to_phys(vaddr)))# define virt_addr_valid (vaddr) pfn_valid(PFN_DOWN(virt_to_phys(vaddr))) -# define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_NON_EXEC +# define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_NON_EXEC#include <asm-generic/memory_model.h>
arch/powerpc/include/asm/page.h +2 -2
@@ -240,8 +240,8 @@ static inline const void *pfn_to_kaddr(unsigned long pfn)* and needs to be executable. This means the whole heap ends * up being executable. */ -#define VM_DATA_DEFAULT_FLAGS32 VM_DATA_FLAGS_TSK_EXEC -#define VM_DATA_DEFAULT_FLAGS64 VM_DATA_FLAGS_NON_EXEC +#define VMA_DATA_DEFAULT_FLAGS32 VMA_DATA_FLAGS_TSK_EXEC +#define VMA_DATA_DEFAULT_FLAGS64 VMA_DATA_FLAGS_NON_EXEC#ifdef __powerpc64__#include <asm/page_64.h>
arch/powerpc/include/asm/page_32.h +1 -1
@@ -10,7 +10,7 @@#endif #endif -#define VM_DATA_DEFAULT_FLAGS VM_DATA_DEFAULT_FLAGS32 +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_DEFAULT_FLAGS32#if defined(CONFIG_PPC_256K_PAGES ) || \ (defined(CONFIG_PPC_8xx) && defined(CONFIG_PPC_16K_PAGES ))
arch/powerpc/include/asm/page_64.h +6 -6
@@ -84,9 +84,9 @@ extern u64 ppc64_pft_size;#endif /* __ASSEMBLER__ */ -#define VM_DATA_DEFAULT_FLAGS \ +#define VMA_DATA_DEFAULT_FLAGS \ (is_32bit_task() ? \ - VM_DATA_DEFAULT_FLAGS32 : VM_DATA_DEFAULT_FLAGS64) + VMA_DATA_DEFAULT_FLAGS32 : VMA_DATA_DEFAULT_FLAGS64)/* * This is the default if a program doesn't have a PT_GNU_STACK
@@ -94,12 +94,12 @@ extern u64 ppc64_pft_size;* stack by default, so in the absence of a PT_GNU_STACK program header * we turn execute permission off. */ -#define VM_STACK_DEFAULT_FLAGS32 VM_DATA_FLAGS_EXEC -#define VM_STACK_DEFAULT_FLAGS64 VM_DATA_FLAGS_NON_EXEC +#define VMA_STACK_DEFAULT_FLAGS32 VMA_DATA_FLAGS_EXEC +#define VMA_STACK_DEFAULT_FLAGS64 VMA_DATA_FLAGS_NON_EXEC -#define VM_STACK_DEFAULT_FLAGS \ +#define VMA_STACK_DEFAULT_FLAGS \ (is_32bit_task() ? \ - VM_STACK_DEFAULT_FLAGS32 : VM_STACK_DEFAULT_FLAGS64) + VMA_STACK_DEFAULT_FLAGS32 : VMA_STACK_DEFAULT_FLAGS64)#include <asm-generic/getorder.h>
arch/riscv/include/asm/page.h +1 -1
@@ -204,7 +204,7 @@ static __always_inline void *pfn_to_kaddr(unsigned long pfn) (unsigned long )(_addr) >=PAGE_OFFSET &&pfn_valid (virt_to_pfn (_addr)); \ }) -#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_NON_EXEC +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_NON_EXEC#include <asm-generic/memory_model.h> #include <asm-generic/getorder.h>
arch/s390/include/asm/page.h +1 -1
@@ -277,7 +277,7 @@ static inline unsigned long virt_to_pfn(const void *kaddr)#define virt_addr_valid (kaddr) pfn_valid(phys_to_pfn(__pa_nodebug((unsigned long)(kaddr)))) -#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_NON_EXEC +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_NON_EXEC#endif /* !__ASSEMBLER__ */
arch/x86/include/asm/page_types.h +1 -1
@@ -26,7 +26,7 @@#define PAGE_OFFSET ((unsigned long)__PAGE_OFFSET) -#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_TSK_EXEC +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_TSK_EXEC/* Physical address where kernel should be loaded. */ #define LOAD_PHYSICAL_ADDR __ALIGN_KERNEL_MASK(CONFIG_PHYSICAL_START, CONFIG_PHYSICAL_ALIGN - 1)
arch/x86/um/asm/vm-flags.h +2 -2
@@ -9,11 +9,11 @@#ifdef CONFIG_X86_32 -#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_TSK_EXEC +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_TSK_EXEC#else -#define VM_STACK_DEFAULT_FLAGS (VM_GROWSDOWN | VM_DATA_FLAGS_EXEC) +#define VMA_STACK_DEFAULT_FLAGS append_vma_flags(VMA_DATA_FLAGS_EXEC, VMA_GROWSDOWN_BIT)#endif #endif
include/linux/ksm.h +5 -5
@@ -17,8 +17,8 @@#ifdef CONFIG_KSM int ksm_madvise (struct vm_area_struct *vma,unsigned long start,unsigned long end,int advice,vm_flags_t *vm_flags); -vm_flags_t ksm_vma_flags (struct mm_struct *mm,const struct file *file, -vm_flags_t vm_flags); +vma_flags_t ksm_vma_flags (struct mm_struct *mm,const struct file *file, +vma_flags_t vma_flags);int ksm_enable_merge_any (struct mm_struct *mm);int ksm_disable_merge_any (struct mm_struct *mm);int ksm_disable (struct mm_struct *mm);
@@ -103,10 +103,10 @@ bool ksm_process_mergeable(struct mm_struct *mm);#else /* !CONFIG_KSM */ -static inline vm_flags_t ksm_vma_flags (struct mm_struct *mm, -const struct file *file,vm_flags_t vm_flags) +static inline vma_flags_t ksm_vma_flags (struct mm_struct *mm, +const struct file *file,vma_flags_t vma_flags) { -return vm_flags; +return vma_flags; }static inline int ksm_disable (struct mm_struct *mm)
include/linux/mm.h +29 -18
97 lines not on this page · raw message
mm/internal.h +3 -0
@@ -1870,4 +1870,7 @@ static inline int get_sysctl_max_map_count(void) return READ_ONCE(sysctl_max_map_count); } +bool may_expand_vm (struct mm_struct *mm,const vma_flags_t *vma_flags, +unsigned long npages); +#endif /* __MM_INTERNAL_H */
mm/ksm.c +23 -20
91 lines not on this page · raw message
mm/mmap.c +8 -5
@@ -192,7 +192,8 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) brkvma =vma_prev_limit (&vmi, mm->start_brk);/* Ok, looks good - let it rip. */ -if (do_brk_flags (&vmi, brkvma, oldbrk, newbrk - oldbrk,0 ) <0 ) +if (do_brk_flags (&vmi, brkvma, oldbrk, newbrk - oldbrk, +EMPTY_VMA_FLAGS ) <0 ) goto out; mm->brk = brk;
@@ -1203,7 +1204,8 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size,int vm_brk_flags (unsigned long addr,unsigned long request,bool is_exec) { -const vm_flags_t vm_flags = is_exec ?VM_EXEC :0 ; +const vma_flags_t vma_flags = is_exec ? +mk_vma_flags (VMA_EXEC_BIT ) :EMPTY_VMA_FLAGS ;struct mm_struct *mm = current->mm;struct vm_area_struct *vma =NULL ;unsigned long len;
@@ -1230,7 +1232,7 @@ int vm_brk_flags(unsigned long addr, unsigned long request, bool is_exec) goto munmap_failed; vma =vma_prev (&vmi); - ret =do_brk_flags (&vmi, vma, addr, len, vm_flags); + ret =do_brk_flags (&vmi, vma, addr, len, vma_flags); populate = ((mm->def_flags &VM_LOCKED ) !=0 );mmap_write_unlock (mm);userfaultfd_unmap_complete (mm, &uf);
@@ -1328,12 +1330,13 @@ void exit_mmap(struct mm_struct *mm)* Return true if the calling process may expand its vm space by the passed * number of pages */ -bool may_expand_vm (struct mm_struct *mm,vm_flags_t flags,unsigned long npages) +bool may_expand_vm (struct mm_struct *mm,const vma_flags_t *vma_flags, +unsigned long npages) {if (mm->total_vm + npages >rlimit (RLIMIT_AS ) >>PAGE_SHIFT )return false; -if (is_data_mapping (flags) && +if (is_data_mapping_vma_flags (vma_flags) && mm->data_vm + npages >rlimit (RLIMIT_DATA ) >>PAGE_SHIFT ) {/* Workaround for Valgrind */ if (rlimit(RLIMIT_DATA) == 0 &&
mm/mprotect.c +28 -18
97 lines not on this page · raw message
mm/mremap.c +3 -3
@@ -1472,10 +1472,10 @@ static unsigned long mremap_to(struct vma_remap_struct *vrm)/* MREMAP_DONTUNMAP expands by old_len since old_len == new_len */ if (vrm->flags &MREMAP_DONTUNMAP ) { -vm_flags_t vm_flags = vrm->vma->vm_flags; +vma_flags_t vma_flags = vrm->vma->flags;unsigned long pages = vrm->old_len >>PAGE_SHIFT ; -if (!may_expand_vm (mm, vm_flags, pages)) +if (!may_expand_vm (mm, &vma_flags, pages))return -ENOMEM ; }
@@ -1813,7 +1813,7 @@ static int check_prep_vma(struct vma_remap_struct *vrm)if (!mlock_future_ok (mm, vma->vm_flags &VM_LOCKED , vrm->delta))return -EAGAIN ; -if (!may_expand_vm (mm, vma->vm_flags, vrm->delta >>PAGE_SHIFT )) +if (!may_expand_vm (mm, &vma->flags, vrm->delta >>PAGE_SHIFT ))return -ENOMEM ;return 0 ;
mm/vma.c +19 -15
@@ -2385,7 +2385,7 @@ static void vms_abort_munmap_vmas(struct vma_munmap_struct *vms,static void update_ksm_flags (struct mmap_state *map) { - map->vm_flags =ksm_vma_flags (map->mm, map->file, map->vm_flags); + map->vma_flags =ksm_vma_flags (map->mm, map->file, map->vma_flags); } static void set_desc_from_map(struct vm_area_desc *desc,
@@ -2446,7 +2446,7 @@ static int __mmap_setup(struct mmap_state *map, struct vm_area_desc *desc, }/* Check against address space limit. */ -if (!may_expand_vm (map->mm, map->vm_flags, map->pglen - vms->nr_pages)) +if (!may_expand_vm (map->mm, &map->vma_flags, map->pglen - vms->nr_pages))return -ENOMEM ;/* Private writable mapping: check memory availability. */
@@ -2866,20 +2866,22 @@ unsigned long mmap_region(struct file *file, unsigned long addr,return ret; } -/* +/** * do_brk_flags() - Increase the brk vma if the flags match. * @vmi: The vma iterator * @addr: The start address * @len: The length of the increase * @vma: The vma, -* @vm_flags: The VMA Flags +* @vma_flags: The VMA Flags * * Extend the brk VMA from addr to addr + len. If the VMA is NULL or the flags * do not match then create a new anonymous VMA. Eventually we may be able to * do some brk-specific accounting here. +* +* Returns: %0 on success, or otherwise an error. */ int do_brk_flags (struct vma_iterator *vmi,struct vm_area_struct *vma, -unsigned long addr,unsigned long len,vm_flags_t vm_flags) +unsigned long addr,unsigned long len,vma_flags_t vma_flags) {struct mm_struct *mm = current->mm;
@@ -2887,9 +2889,12 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma,* Check against address space limits by the changed size * Note: This happens *after* clearing old mappings in some code paths. */ - vm_flags |=VM_DATA_DEFAULT_FLAGS |VM_ACCOUNT | mm->def_flags; - vm_flags =ksm_vma_flags (mm,NULL , vm_flags); -if (!may_expand_vm (mm, vm_flags, len >>PAGE_SHIFT )) +vma_flags_set_mask (&vma_flags,VMA_DATA_DEFAULT_FLAGS ); +vma_flags_set (&vma_flags,VMA_ACCOUNT_BIT ); +vma_flags_set_mask (&vma_flags, mm->def_vma_flags); + + vma_flags =ksm_vma_flags (mm,NULL , vma_flags); +if (!may_expand_vm (mm, &vma_flags, len >>PAGE_SHIFT ))return -ENOMEM ;if (mm->map_count >get_sysctl_max_map_count ())
@@ -2903,7 +2908,7 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma,* occur after forking, so the expand will only happen on new VMAs. */ if (vma && vma->vm_end == addr) { -VMG_STATE (vmg, mm, vmi, addr, addr + len, vm_flags,PHYS_PFN (addr)); +VMG_STATE (vmg, mm, vmi, addr, addr + len, vma_flags,PHYS_PFN (addr)); vmg.prev = vma;/* vmi is positioned at prev, which this mode expects. */
@@ -2924,8 +2929,8 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma,vma_set_anonymous (vma);vma_set_range (vma, addr, addr + len, addr >>PAGE_SHIFT ); -vm_flags_init (vma, vm_flags); - vma->vm_page_prot =vm_get_page_prot (vm_flags); + vma->flags = vma_flags; + vma->vm_page_prot =vm_get_page_prot (vma_flags_to_legacy (vma_flags));vma_start_write (vma);if (vma_iter_store_gfp (vmi, vma,GFP_KERNEL )) goto mas_store_fail;
@@ -2936,10 +2941,10 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma,perf_event_mmap (vma); mm->total_vm += len >>PAGE_SHIFT ; mm->data_vm += len >>PAGE_SHIFT ; -if (vm_flags &VM_LOCKED ) +if (vma_flags_test (&vma_flags,VMA_LOCKED_BIT )) mm->locked_vm += (len >>PAGE_SHIFT );if (pgtable_supports_soft_dirty ()) -vm_flags_set (vma,VM_SOFTDIRTY ); +vma_set_flags (vma,VMA_SOFTDIRTY_BIT );return 0 ; mas_store_fail:
@@ -3070,7 +3075,7 @@ static int acct_stack_growth(struct vm_area_struct *vma,unsigned long new_start;/* address space limit tests */ -if (!may_expand_vm (mm, vma->vm_flags, grow)) +if (!may_expand_vm (mm, &vma->flags, grow))return -ENOMEM ;/* Stack limit test */
@@ -3289,7 +3294,6 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma) {unsigned long charged =vma_pages (vma); -if (find_vma_intersection (mm, vma->vm_start, vma->vm_end))return -ENOMEM ;
mm/vma.h +11 -3
@@ -237,13 +237,13 @@ static inline pgoff_t vma_pgoff_offset(struct vm_area_struct *vma,return vma->vm_pgoff +PHYS_PFN (addr - vma->vm_start); } -#define VMG_STATE (name, mm_, vmi_, start_, end_, vm_flags_, pgoff_) \ +#define VMG_STATE (name, mm_, vmi_, start_, end_, vma_flags_, pgoff_) \ struct vma_merge_struct name = { \ .mm = mm_, \ .vmi = vmi_, \ .start = start_, \ .end = end_, \ - .vm_flags = vm_flags_, \ + .vma_flags = vma_flags_, \ .pgoff = pgoff_, \ .state = VMA_MERGE_START, \ }
@@ -465,7 +465,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, struct list_head *uf);int do_brk_flags (struct vma_iterator *vmi,struct vm_area_struct *brkvma, -unsigned long addr,unsigned long request,unsigned long flags); +unsigned long addr,unsigned long request, +vma_flags_t vma_flags);unsigned long unmapped_area (struct vm_unmapped_area_info *info);unsigned long unmapped_area_topdown (struct vm_unmapped_area_info *info);
@@ -527,6 +528,13 @@ static inline bool is_data_mapping(vm_flags_t flags)return (flags & (VM_WRITE |VM_SHARED |VM_STACK )) ==VM_WRITE ; } +static inline bool is_data_mapping_vma_flags (const vma_flags_t *vma_flags) +{ +const vma_flags_t mask =vma_flags_and (vma_flags, +VMA_WRITE_BIT ,VMA_SHARED_BIT ,VMA_STACK_BIT ); + +return vma_flags_same (&mask,VMA_WRITE_BIT ); +}static inline void vma_iter_config (struct vma_iterator *vmi,unsigned long index,unsigned long last)
mm/vma_exec.c +3 -2
@@ -36,7 +36,8 @@ int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift)unsigned long new_start = old_start - shift;unsigned long new_end = old_end - shift;VMA_ITERATOR (vmi, mm, new_start); -VMG_STATE (vmg, mm, &vmi, new_start, old_end,0 , vma->vm_pgoff); +VMG_STATE (vmg, mm, &vmi, new_start, old_end,EMPTY_VMA_FLAGS , + vma->vm_pgoff);struct vm_area_struct *next;struct mmu_gather tlb;PAGETABLE_MOVE (pmc, vma, vma, old_start, new_start, length);
@@ -135,7 +136,7 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,* use STACK_TOP because that can depend on attributes which aren't * configured yet. */ -BUILD_BUG_ON (VM_STACK_FLAGS &VM_STACK_INCOMPLETE_SETUP ); +VM_WARN_ON_ONCE (VM_STACK_FLAGS &VM_STACK_INCOMPLETE_SETUP ); vma->vm_end =STACK_TOP_MAX ; vma->vm_start = vma->vm_end -PAGE_SIZE ;if (pgtable_supports_soft_dirty ())
security/selinux/hooks.c +3 -1
@@ -7713,6 +7713,8 @@ static struct security_hook_list selinux_hooks[] __ro_after_init = {static __init int selinux_init (void ) { +vma_flags_t data_default_flags =VMA_DATA_DEFAULT_FLAGS ; +pr_info ("SELinux: Initializing.\n" );memset (&selinux_state,0 ,sizeof (selinux_state));
@@ -7729,7 +7731,7 @@ static __init int selinux_init(void)AUDIT_CFG_LSM_SECCTX_SUBJECT | AUDIT_CFG_LSM_SECCTX_OBJECT); - default_noexec = !(VM_DATA_DEFAULT_FLAGS &VM_EXEC ); + default_noexec = !vma_flags_test (&data_default_flags,VMA_EXEC_BIT );if (!default_noexec)pr_notice ("SELinux: virtual memory is executable by default\n" );
tools/testing/vma/include/custom.h +0 -3
@@ -95,6 +95,3 @@ static inline unsigned long vma_kernel_pagesize(struct vm_area_struct *vma) {return PAGE_SIZE ; } - -#define VMA_SPECIAL_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_DONTEXPAND_BIT, \ - VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT)
tools/testing/vma/include/dup.h +23 -19
67 lines not on this page · raw message
tools/testing/vma/include/stubs.h +5 -4
@@ -101,10 +101,10 @@ static inline bool shmem_file(struct file *file)return false; } -static inline vm_flags_t ksm_vma_flags (const struct mm_struct *mm, -const struct file *file,vm_flags_t vm_flags) +static inline vma_flags_t ksm_vma_flags (struct mm_struct *mm, +const struct file *file,vma_flags_t vma_flags) { -return vm_flags; +return vma_flags; }static inline void remap_pfn_range_prepare (struct vm_area_desc *desc,unsigned long pfn)
@@ -239,7 +239,8 @@ static inline int security_vm_enough_memory_mm(struct mm_struct *mm, long pages)return 0 ; } -static inline bool may_expand_vm (struct mm_struct *mm,vm_flags_t flags, +static inline bool may_expand_vm (struct mm_struct *mm, +const vma_flags_t *vma_flags,unsigned long npages) {return true;
tools/testing/vma/tests/merge.c +1 -2
@@ -1429,11 +1429,10 @@ static bool test_expand_only_mode(void) {vma_flags_t vma_flags =mk_vma_flags (VMA_READ_BIT ,VMA_WRITE_BIT ,VMA_MAYREAD_BIT ,VMA_MAYWRITE_BIT ); -vm_flags_t legacy_flags =VM_READ |VM_WRITE |VM_MAYREAD |VM_MAYWRITE ;struct mm_struct mm = {};VMA_ITERATOR (vmi, &mm,0 );struct vm_area_struct *vma_prev, *vma; -VMG_STATE (vmg, &mm, &vmi,0x5000 ,0x9000 , legacy_flags,5 ); +VMG_STATE (vmg, &mm, &vmi,0x5000 ,0x9000 , vma_flags,5 );/* * Place a VMA prior to the one we're expanding so we assert that we do
Signature · 1 line
2.53.0
Re: [PATCH v3 17/23] mm: convert do_brk_flags() to use vma_flags_t
Quoted 9 lines · Lorenzo Stoakes (Oracle)
In order to be able to do this, we need to change
VM_DATA_DEFAULT_FLAGSand friends and update the architecture-specific definitions also.We then have to update some KSM logic to handle VMA flags, and introduce
VMA_STACK_FLAGSto define thevma_flags_tequivalent ofVM_STACK_FLAGS.We also introduce two helper functions for use during the time we are converting legacy flags to
vma_flags_tvalues -vma_flags_to_legacy()andlegacy_to_vma_flags().
Nit: this was done by an earlier patch.
Quoted 22 lines
This enables us to iteratively make changes to break these changes up into separate parts.
We use these explicitly here to keep
VM_STACK_FLAGSaround for certain users which need to maintain the legacyvm_flags_tvalues for the time being.We are no longer able to rely on the simple
VM_xxxbeing set to zero if the feature is not enabled, so in the case ofVM_DROPPABLEwe introduceVMA_DROPPABLEas thevma_flags_tequivalent, which is set toEMPTY_VMA_FLAGSif the droppable flag is not available.While we're here, we make the description of
do_brk_flags()into a kdoc comment, as it almost was already.We use
vma_flags_to_legacy()to not need to update thevm_get_page_prot()logic as this time.Note that in
create_init_stack_vma()we have to replace theBUILD_BUG_ON()with aVM_WARN_ON_ONCE()as the tested values are no longer build time available.
Quoted 8 lines
We also update
mprotect_fixup()to use VMA flags where possible, though we have to live with a little duplication betweenvm_flags_tandvma_flags_tvalues for the time being until further conversions are made.Finally, we update the VMA tests to reflect these changes.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
More nits below:
arch/arm64/include/asm/page.h int pfn_is_map_memory(unsigned long pfn); · line 51
Quoted 12 lines
arch/arm64/include/asm/page.h +3
-1@@ -46,7 +46,12 @@ int pfn_is_map_memory(unsigned long pfn);#endif /* !__ASSEMBLER__ */ -#define VM_DATA_DEFAULT_FLAGS (VM_DATA_FLAGS_TSK_EXEC | VM_MTE_ALLOWED) +#ifdef CONFIG_ARM64_MTE +#define VMA_DATA_DEFAULT_FLAGS append_vma_flags(VMA_DATA_FLAGS_TSK_EXEC, \ + VMA_MTE_ALLOWED_BIT)
I wonder what's the bloat-o-meter impact of these #define's (this arm64-specific one isn't the only one) being no longer compile-time-constants?
Quoted 30 lines
arch/arm64/include/asm/page.h +3
-0+#else +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_TSK_EXEC +#endif #include <asm-generic/getorder.h> include/linux/mm.h +6
-2+#define VMA_STACK_FLAGS append_vma_flags(VMA_STACK_DEFAULT_FLAGS, \ + VMA_STACK_BIT, VMA_ACCOUNT_BIT) + +/* Temporary until VMA flags conversion complete. */ +#define VM_STACK_FLAGS vma_flags_to_legacy(VMA_STACK_FLAGS) +#define VM_STARTGAP_FLAGS (VM_GROWSDOWN | VM_SHADOW_STACK)#ifdef CONFIG_MSEAL_SYSTEM_MAPPINGS @@ -536,8 +547,6 @@ enum {#define VM_SEALED_SYSMAP VM_NONE#endif -#define VM_STACK_FLAGS (VM_STACK | VM_STACK_DEFAULT_FLAGS | VM_ACCOUNT) -/* VMA basic access permission flags */ #define VM_ACCESS_FLAGS (VM_READ | VM_WRITE | VM_EXEC)#define VMA_ACCESS_FLAGS mk_vma_flags(VMA_READ_BIT, VMA_WRITE_BIT, VMA_EXEC_BIT)
include/linux/mm.h enum { · line 560
Quoted 5 lines
include/linux/mm.h +2
-0@@ -547,6 +556,9 @@ enum {*/ #define VM_SPECIAL (VM_IO | VM_DONTEXPAND | VM_PFNMAP | VM_MIXEDMAP) +#define VMA_SPECIAL_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_DONTEXPAND_BIT, \ + VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT)
Should VM_SPECIAL be also redefined using vma_flags_to_legacy()?
include/linux/mm.h static __always_inline void vma_desc_set_flags_mask(struct vm_area_desc *desc, · line 1427
Quoted 10 lines
include/linux/mm.h +2
-1+/* * Physically remapped pages are special. Tell the * rest of the world about it: @@ -1412,7 +1424,7 @@ static __always_inline void vma_desc_set_flags_mask(struct vm_area_desc *desc,* vm_area_desc object describing a proposed VMA, e.g.: * * vma_desc_set_flags(desc, VMA_IO_BIT, VMA_PFNMAP_BIT, VMA_DONTEXPAND_BIT, -* VMA_DONTDUMP_BIT); +* VMA_DONTDUMP_BIT);
Looks like spurious tabs-to-space change inconsistent with other instances.
Quoted 31 lines
include/linux/mm.h +0
-1*/ #define vma_desc_set_flags (desc, ...) \ vma_desc_set_flags_mask(desc, mk_vma_flags(__VA_ARGS__))@@ -4059,7 +4071,6 @@ extern int replace_mm_exe_file(struct mm_struct *mm, struct file *new_exe_file);extern struct file *get_mm_exe_file (struct mm_struct *mm);extern struct file *get_task_exe_file (struct task_struct *task); -extern bool may_expand_vm (struct mm_struct *,vm_flags_t ,unsigned long npages);extern void vm_stat_account (struct mm_struct *,vm_flags_t ,long npages); extern bool vma_is_special_mapping(const struct vm_area_struct *vma,mm/internal.h +0
-0mm/mprotect.c +2
-1@@ -773,19 +778,24 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,change_protection (tlb, vma, start, end, mm_cp_flags); -if ((oldflags &VM_ACCOUNT ) && !(newflags &VM_ACCOUNT )) +if (vma_flags_test (&old_vma_flags,VMA_ACCOUNT_BIT ) && + !vma_flags_test (&new_vma_flags,VMA_ACCOUNT_BIT ))vm_unacct_memory (nrpages);/* * Private VM_LOCKED VMA becoming writable: trigger COW to avoid major * fault on access.
mm/mprotect.c mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, · line 794
Quoted 10 lines
mm/mprotect.c +6
-3*/ -if ((oldflags & (VM_WRITE |VM_SHARED |VM_LOCKED )) ==VM_LOCKED && - (newflags &VM_WRITE )) { -populate_vma_page_range (vma, start, end,NULL ); +if (vma_flags_test (&new_vma_flags,VMA_WRITE_BIT )) { +const vma_flags_t mask = +vma_flags_and (&old_vma_flags,VMA_WRITE_BIT , +VMA_SHARED_BIT ,VMA_LOCKED_BIT ); + +if (vma_flags_same (&mask,VMA_LOCKED_BIT ))
That converts the original logic 1:1, but I wonder if it's now feasible to write it more obviously as "VMA_LOCKED_BIT must be set, VM_WRITE_BIT and VM_SHARED_BIT must not" ?
Quoted 13 lines
mm/mprotect.c +2
-1+populate_vma_page_range (vma, start, end,NULL ); } -vm_stat_account (mm, oldflags, -nrpages); +vm_stat_account (mm,vma_flags_to_legacy (old_vma_flags), -nrpages);vm_stat_account (mm, newflags, nrpages);perf_event_mmap (vma);return 0 ;mm/vma.h +0
-0
mm/vma.h static inline bool is_data_mapping(vm_flags_t flags) · line 536
Quoted 7 lines
mm/vma.h +6
-0+static inline bool is_data_mapping_vma_flags (const vma_flags_t *vma_flags) +{ +const vma_flags_t mask =vma_flags_and (vma_flags, +VMA_WRITE_BIT ,VMA_SHARED_BIT ,VMA_STACK_BIT ); + +return vma_flags_same (&mask,VMA_WRITE_BIT );
Ditto?
Quoted 6 lines
mm/vma.h +1
-0+}static inline void vma_iter_config (struct vma_iterator *vmi,unsigned long index,unsigned long last)mm/vma_exec.c +0
-0
Lorenzo Stoakes (Oracle)
maintainerRe: [PATCH v3 17/23] mm: convert do_brk_flags() to use vma_flags_t
Quoted 4 lines · Vlastimil Babka (SUSE)
Quoted 3 lines · Lorenzo Stoakes (Oracle)
In order to be able to do this, we need to change
VM_DATA_DEFAULT_FLAGSand friends and update the architecture-specific definitions also.
Quoted 8 lines
Quoted 6 lines
We then have to update some KSM logic to handle VMA flags, and introduce
VMA_STACK_FLAGSto define thevma_flags_tequivalent ofVM_STACK_FLAGS.We also introduce two helper functions for use during the time we are converting legacy flags to
vma_flags_tvalues -vma_flags_to_legacy()andlegacy_to_vma_flags().Nit: this was done by an earlier patch.
Ack will fix up.
Quoted 27 lines
Quoted 27 lines
This enables us to iteratively make changes to break these changes up into separate parts.
We use these explicitly here to keep
VM_STACK_FLAGSaround for certain users which need to maintain the legacyvm_flags_tvalues for the time being.We are no longer able to rely on the simple
VM_xxxbeing set to zero if the feature is not enabled, so in the case ofVM_DROPPABLEwe introduceVMA_DROPPABLEas thevma_flags_tequivalent, which is set toEMPTY_VMA_FLAGSif the droppable flag is not available.While we're here, we make the description of
do_brk_flags()into a kdoc comment, as it almost was already.We use
vma_flags_to_legacy()to not need to update thevm_get_page_prot()logic as this time.Note that in
create_init_stack_vma()we have to replace theBUILD_BUG_ON()with aVM_WARN_ON_ONCE()as the tested values are no longer build time available.We also update
mprotect_fixup()to use VMA flags where possible, though we have to live with a little duplication betweenvm_flags_tandvma_flags_tvalues for the time being until further conversions are made.
Quoted 6 lines
Quoted 4 lines
Finally, we update the VMA tests to reflect these changes.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Thanks!
Quoted 11 lines
More nits below:
Quoted 8 lines
arch/arm64/include/asm/page.h +0
-0@@ -46,7 +46,12 @@ int pfn_is_map_memory(unsigned long pfn);#endif /* !__ASSEMBLER__ */
Quoted 7 lines
arch/arm64/include/asm/page.h int pfn_is_map_memory(unsigned long pfn); · line 51
Quoted 4 lines
arch/arm64/include/asm/page.h +3
-1-#define VM_DATA_DEFAULT_FLAGS (VM_DATA_FLAGS_TSK_EXEC | VM_MTE_ALLOWED) +#ifdef CONFIG_ARM64_MTE +#define VMA_DATA_DEFAULT_FLAGS append_vma_flags(VMA_DATA_FLAGS_TSK_EXEC, \ + VMA_MTE_ALLOWED_BIT)I wonder what's the bloat-o-meter impact of these #define's (this arm64-specific one isn't the only one) being no longer compile-time-constants?
I mean there's a precedent for this, but the compiler _should_ figure out this as a constant value, I have repeatedly confirmed that it's good at that in godbolt, via make foo/bar.S etc.
So it should have no measureable impact at 64-bit VMA flags anyway, but I can give it a go and see before/after.
Quoted 34 lines
Quoted 34 lines
arch/arm64/include/asm/page.h +3
-0+#else +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_TSK_EXEC +#endif #include <asm-generic/getorder.h> include/linux/mm.h +6
-2+#define VMA_STACK_FLAGS append_vma_flags(VMA_STACK_DEFAULT_FLAGS, \ + VMA_STACK_BIT, VMA_ACCOUNT_BIT) + +/* Temporary until VMA flags conversion complete. */ +#define VM_STACK_FLAGS vma_flags_to_legacy(VMA_STACK_FLAGS) +#define VM_STARTGAP_FLAGS (VM_GROWSDOWN | VM_SHADOW_STACK)#ifdef CONFIG_MSEAL_SYSTEM_MAPPINGS @@ -536,8 +547,6 @@ enum {#define VM_SEALED_SYSMAP VM_NONE#endif -#define VM_STACK_FLAGS (VM_STACK | VM_STACK_DEFAULT_FLAGS | VM_ACCOUNT) -/* VMA basic access permission flags */ #define VM_ACCESS_FLAGS (VM_READ | VM_WRITE | VM_EXEC)#define VMA_ACCESS_FLAGS mk_vma_flags(VMA_READ_BIT, VMA_WRITE_BIT, VMA_EXEC_BIT)@@ -547,6 +556,9 @@ enum {*/ #define VM_SPECIAL (VM_IO | VM_DONTEXPAND | VM_PFNMAP | VM_MIXEDMAP)
Quoted 4 lines
include/linux/mm.h enum { · line 560
Quoted 2 lines
include/linux/mm.h +2
-0+#define VMA_SPECIAL_FLAGS mk_vma_flags(VMA_IO_BIT, VMA_DONTEXPAND_BIT, \ + VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT)Should
VM_SPECIALbe also redefined usingvma_flags_to_legacy()?
Could do! It should be a pretty clear indicator this is legacy.
Quoted 13 lines
include/linux/mm.h static __always_inline void vma_desc_set_flags_mask(struct vm_area_desc *desc, · line 1427
Quoted 11 lines
include/linux/mm.h +2
-1+/* * Physically remapped pages are special. Tell the * rest of the world about it: @@ -1412,7 +1424,7 @@ static __always_inline void vma_desc_set_flags_mask(struct vm_area_desc *desc,* vm_area_desc object describing a proposed VMA, e.g.: * * vma_desc_set_flags(desc, VMA_IO_BIT, VMA_PFNMAP_BIT, VMA_DONTEXPAND_BIT, -* VMA_DONTDUMP_BIT); +* VMA_DONTDUMP_BIT); Looks like spurious tabs-to-space change inconsistent with other instances.
Yeah that's a mistake, will fixup.
Quoted 38 lines
Quoted 38 lines
include/linux/mm.h +0
-1*/ #define vma_desc_set_flags (desc, ...) \ vma_desc_set_flags_mask(desc, mk_vma_flags(__VA_ARGS__))@@ -4059,7 +4071,6 @@ extern int replace_mm_exe_file(struct mm_struct *mm, struct file *new_exe_file);extern struct file *get_mm_exe_file (struct mm_struct *mm);extern struct file *get_task_exe_file (struct task_struct *task); -extern bool may_expand_vm (struct mm_struct *,vm_flags_t ,unsigned long npages);extern void vm_stat_account (struct mm_struct *,vm_flags_t ,long npages); extern bool vma_is_special_mapping(const struct vm_area_struct *vma,mm/internal.h +0
-0mm/mprotect.c +4
-4@@ -773,19 +778,24 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,change_protection (tlb, vma, start, end, mm_cp_flags); -if ((oldflags &VM_ACCOUNT ) && !(newflags &VM_ACCOUNT )) +if (vma_flags_test (&old_vma_flags,VMA_ACCOUNT_BIT ) && + !vma_flags_test (&new_vma_flags,VMA_ACCOUNT_BIT ))vm_unacct_memory (nrpages);/* * Private VM_LOCKED VMA becoming writable: trigger COW to avoid major * fault on access. */ -if ((oldflags & (VM_WRITE |VM_SHARED |VM_LOCKED )) ==VM_LOCKED && - (newflags &VM_WRITE )) { -populate_vma_page_range (vma, start, end,NULL ); +if (vma_flags_test (&new_vma_flags,VMA_WRITE_BIT )) { + const vma_flags_t mask =
Quoted 8 lines
mm/mprotect.c mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, · line 794
Quoted 4 lines
mm/mprotect.c +4
-0+vma_flags_and (&old_vma_flags,VMA_WRITE_BIT , +VMA_SHARED_BIT ,VMA_LOCKED_BIT ); + +if (vma_flags_same (&mask,VMA_LOCKED_BIT ))That converts the original logic 1:1, but I wonder if it's now feasible to write it more obviously as "
VMA_LOCKED_BITmust be set,VM_WRITE_BITandVM_SHARED_BITmust not" ?
Hmm, I'm not sure if I can express this more clearly, it's a pain either way. Could do:
if (vma_flags_test(&new_vma_flags, VMA_WRITE_BIT) && !vma_flags_test_any(&old_vma_flags, VMA_WRITE_BIT, VMA_SHARED_BIT)) populate_vma_page_range(vma, start, end, NULL);
Quoted 15 lines
Quoted 14 lines
mm/mprotect.c +2
-1+populate_vma_page_range (vma, start, end,NULL ); } -vm_stat_account (mm, oldflags, -nrpages); +vm_stat_account (mm,vma_flags_to_legacy (old_vma_flags), -nrpages);vm_stat_account (mm, newflags, nrpages);perf_event_mmap (vma);return 0 ;mm/vma.h +0
-0
Quoted 8 lines
mm/vma.h static inline bool is_data_mapping(vm_flags_t flags) · line 536
Quoted 6 lines
mm/vma.h +6
-0+static inline bool is_data_mapping_vma_flags (const vma_flags_t *vma_flags) +{ +const vma_flags_t mask =vma_flags_and (vma_flags, +VMA_WRITE_BIT ,VMA_SHARED_BIT ,VMA_STACK_BIT ); + +return vma_flags_same (&mask,VMA_WRITE_BIT );Ditto?
I may do both as a follow up patch given the series is a pain to rework at this point and I want at least the first version to be like-for-like intentionally.
Quoted 7 lines
mm/vma_exec.c
Quoted 7 lines
mm/vma.h +1
-0+}static inline void vma_iter_config (struct vma_iterator *vmi,unsigned long index,unsigned long last)mm/vma_exec.c +0
-0
Thanks, Lorenzo
Vlastimil Babka (SUSE)
reviewerRe: [PATCH v3 17/23] mm: convert do_brk_flags() to use vma_flags_t
Quoted 19 lines · Lorenzo Stoakes (Oracle)
Quoted 11 lines
More nits below:
Quoted 8 lines
arch/arm64/include/asm/page.h +0
-0@@ -46,7 +46,12 @@ int pfn_is_map_memory(unsigned long pfn);#endif /* !__ASSEMBLER__ */ Quoted 7 lines
arch/arm64/include/asm/page.h int pfn_is_map_memory(unsigned long pfn); · line 51
Quoted 4 lines
arch/arm64/include/asm/page.h +3
-1-#define VM_DATA_DEFAULT_FLAGS (VM_DATA_FLAGS_TSK_EXEC | VM_MTE_ALLOWED) +#ifdef CONFIG_ARM64_MTE +#define VMA_DATA_DEFAULT_FLAGS append_vma_flags(VMA_DATA_FLAGS_TSK_EXEC, \ + VMA_MTE_ALLOWED_BIT)I wonder what's the bloat-o-meter impact of these #define's (this arm64-specific one isn't the only one) being no longer compile-time-constants?
Quoted 3 lines
I mean there's a precedent for this, but the compiler _should_ figure out this as a constant value, I have repeatedly confirmed that it's good at that in godbolt, via make
foo/bar.Setc.
Great, thanks!
Quoted 48 lines
Quoted 39 lines
Quoted 39 lines
arch/arm64/include/asm/page.h +0
-1*/ #define vma_desc_set_flags (desc, ...) \ vma_desc_set_flags_mask(desc, mk_vma_flags(__VA_ARGS__))@@ -4059,7 +4071,6 @@ extern int replace_mm_exe_file(struct mm_struct *mm, struct file *new_exe_file);extern struct file *get_mm_exe_file (struct mm_struct *mm);extern struct file *get_task_exe_file (struct task_struct *task); -extern bool may_expand_vm (struct mm_struct *,vm_flags_t ,unsigned long npages);extern void vm_stat_account (struct mm_struct *,vm_flags_t ,long npages); extern bool vma_is_special_mapping(const struct vm_area_struct *vma,mm/internal.h +0
-0mm/mprotect.c +4
-4@@ -773,19 +778,24 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,change_protection (tlb, vma, start, end, mm_cp_flags); -if ((oldflags &VM_ACCOUNT ) && !(newflags &VM_ACCOUNT )) +if (vma_flags_test (&old_vma_flags,VMA_ACCOUNT_BIT ) && + !vma_flags_test (&new_vma_flags,VMA_ACCOUNT_BIT ))vm_unacct_memory (nrpages);/* * Private VM_LOCKED VMA becoming writable: trigger COW to avoid major * fault on access. */ -if ((oldflags & (VM_WRITE |VM_SHARED |VM_LOCKED )) ==VM_LOCKED && - (newflags &VM_WRITE )) { -populate_vma_page_range (vma, start, end,NULL ); +if (vma_flags_test (&new_vma_flags,VMA_WRITE_BIT )) { + const vma_flags_t mask =Quoted 8 lines
mm/mprotect.c mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, · line 794
Quoted 4 lines
mm/mprotect.c +4
-0+vma_flags_and (&old_vma_flags,VMA_WRITE_BIT , +VMA_SHARED_BIT ,VMA_LOCKED_BIT ); + +if (vma_flags_same (&mask,VMA_LOCKED_BIT ))That converts the original logic 1:1, but I wonder if it's now feasible to write it more obviously as "
VMA_LOCKED_BITmust be set,VM_WRITE_BITandVM_SHARED_BITmust not" ?
Quoted 6 lines
Hmm, I'm not sure if I can express this more clearly, it's a pain either way. Could do:
if (vma_flags_test(&new_vma_flags, VMA_WRITE_BIT) && !vma_flags_test_any(&old_vma_flags, VMA_WRITE_BIT, VMA_SHARED_BIT)) populate_vma_page_range(vma, start, end, NULL);
It would be a bit more:
if (vma_flags_test(&new_vma_flags, VMA_WRITE_BIT) && vma_flags_test(&old_vma_flags, VMA_LOCKED_BIT) &&
!vma_flags_test_any(&old_vma_flags, VMA_WRITE_BIT, VMA_SHARED_BIT))
populate_vma_page_range(vma, start, end, NULL);
(or reordered in whatever way the short circuting works best here, the original code tested oldflags first).
But yeah, at least to me it's more clear what that's testing and doesn't need to set up the intermediate mask variable, and VMA_LOCKED_BIT is there only once in the code now. The number of vma_flags_* operations is the same.
Quoted 24 lines
Quoted 15 lines
Quoted 14 lines
mm/mprotect.c +2
-1+populate_vma_page_range (vma, start, end,NULL ); } -vm_stat_account (mm, oldflags, -nrpages); +vm_stat_account (mm,vma_flags_to_legacy (old_vma_flags), -nrpages);vm_stat_account (mm, newflags, nrpages);perf_event_mmap (vma);return 0 ;mm/vma.h +0
-0Quoted 8 lines
mm/vma.h static inline bool is_data_mapping(vm_flags_t flags) · line 536
Quoted 6 lines
mm/vma.h +6
-0+static inline bool is_data_mapping_vma_flags (const vma_flags_t *vma_flags) +{ +const vma_flags_t mask =vma_flags_and (vma_flags, +VMA_WRITE_BIT ,VMA_SHARED_BIT ,VMA_STACK_BIT ); + +return vma_flags_same (&mask,VMA_WRITE_BIT );Ditto?
Quoted 2 lines
I may do both as a follow up patch given the series is a pain to rework at this point and I want at least the first version to be like-for-like intentionally.
OK sure.
Quoted 9 lines
Quoted 7 lines
mm/vma_exec.c
Quoted 7 lines
mm/vma.h +1
-0+}static inline void vma_iter_config (struct vma_iterator *vmi,unsigned long index,unsigned long last)mm/vma_exec.c +0
-0Thanks, Lorenzo
Lorenzo Stoakes (Oracle)
maintainerRe: [PATCH v3 17/23] mm: convert do_brk_flags() to use vma_flags_t
Quoted 85 lines · Vlastimil Babka (SUSE)
Quoted 19 lines · Lorenzo Stoakes (Oracle)
Quoted 11 lines
More nits below:
Quoted 8 lines
arch/arm64/include/asm/page.h +0
-0@@ -46,7 +46,12 @@ int pfn_is_map_memory(unsigned long pfn);#endif /* !__ASSEMBLER__ */ Quoted 7 lines
arch/arm64/include/asm/page.h int pfn_is_map_memory(unsigned long pfn); · line 51
Quoted 4 lines
arch/arm64/include/asm/page.h +3
-1-#define VM_DATA_DEFAULT_FLAGS (VM_DATA_FLAGS_TSK_EXEC | VM_MTE_ALLOWED) +#ifdef CONFIG_ARM64_MTE +#define VMA_DATA_DEFAULT_FLAGS append_vma_flags(VMA_DATA_FLAGS_TSK_EXEC, \ + VMA_MTE_ALLOWED_BIT)I wonder what's the bloat-o-meter impact of these #define's (this arm64-specific one isn't the only one) being no longer compile-time-constants?
Quoted 3 lines
I mean there's a precedent for this, but the compiler _should_ figure out this as a constant value, I have repeatedly confirmed that it's good at that in godbolt, via make
foo/bar.Setc.Great, thanks!
Quoted 48 lines
Quoted 39 lines
Quoted 39 lines
arch/arm64/include/asm/page.h +0
-1*/ #define vma_desc_set_flags (desc, ...) \ vma_desc_set_flags_mask(desc, mk_vma_flags(__VA_ARGS__))@@ -4059,7 +4071,6 @@ extern int replace_mm_exe_file(struct mm_struct *mm, struct file *new_exe_file);extern struct file *get_mm_exe_file (struct mm_struct *mm);extern struct file *get_task_exe_file (struct task_struct *task); -extern bool may_expand_vm (struct mm_struct *,vm_flags_t ,unsigned long npages);extern void vm_stat_account (struct mm_struct *,vm_flags_t ,long npages); extern bool vma_is_special_mapping(const struct vm_area_struct *vma,mm/internal.h +0
-0mm/mprotect.c +4
-4@@ -773,19 +778,24 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,change_protection (tlb, vma, start, end, mm_cp_flags); -if ((oldflags &VM_ACCOUNT ) && !(newflags &VM_ACCOUNT )) +if (vma_flags_test (&old_vma_flags,VMA_ACCOUNT_BIT ) && + !vma_flags_test (&new_vma_flags,VMA_ACCOUNT_BIT ))vm_unacct_memory (nrpages);/* * Private VM_LOCKED VMA becoming writable: trigger COW to avoid major * fault on access. */ -if ((oldflags & (VM_WRITE |VM_SHARED |VM_LOCKED )) ==VM_LOCKED && - (newflags &VM_WRITE )) { -populate_vma_page_range (vma, start, end,NULL ); +if (vma_flags_test (&new_vma_flags,VMA_WRITE_BIT )) { + const vma_flags_t mask =Quoted 8 lines
mm/mprotect.c mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, · line 794
Quoted 4 lines
mm/mprotect.c +4
-0+vma_flags_and (&old_vma_flags,VMA_WRITE_BIT , +VMA_SHARED_BIT ,VMA_LOCKED_BIT ); + +if (vma_flags_same (&mask,VMA_LOCKED_BIT ))That converts the original logic 1:1, but I wonder if it's now feasible to write it more obviously as "
VMA_LOCKED_BITmust be set,VM_WRITE_BITandVM_SHARED_BITmust not" ?Quoted 6 lines
Hmm, I'm not sure if I can express this more clearly, it's a pain either way. Could do:
if (vma_flags_test(&new_vma_flags, VMA_WRITE_BIT) && !vma_flags_test_any(&old_vma_flags, VMA_WRITE_BIT, VMA_SHARED_BIT)) populate_vma_page_range(vma, start, end, NULL);It would be a bit more:
if (vma_flags_test(&new_vma_flags, VMA_WRITE_BIT) && vma_flags_test(&old_vma_flags, VMA_LOCKED_BIT) &&
Review
Quoted 2 lines
(quoted diff) +0
-0!vma_flags_test_any(&old_vma_flags, VMA_WRITE_BIT, VMA_SHARED_BIT)) populate_vma_page_range(vma, start, end, NULL);
Right yeah sorry :)
That is clearer but still pretty verbose. Will think about how to do it but I think as a follow up still makes more sense.
Quoted 32 lines
(or reordered in whatever way the short circuting works best here, the original code tested oldflags first).
But yeah, at least to me it's more clear what that's testing and doesn't need to set up the intermediate mask variable, and
VMA_LOCKED_BITis there only once in the code now. The number of vma_flags_* operations is the same.Quoted 24 lines
Quoted 23 lines
Quoted 14 lines
mm/mprotect.c +2
-1+populate_vma_page_range (vma, start, end,NULL ); } -vm_stat_account (mm, oldflags, -nrpages); +vm_stat_account (mm,vma_flags_to_legacy (old_vma_flags), -nrpages);vm_stat_account (mm, newflags, nrpages);perf_event_mmap (vma);return 0 ;mm/vma.h +0
-0mm/vma.h static inline bool is_data_mapping(vm_flags_t flags) · line 536
Quoted 7 lines
mm/vma.h +6
-0+static inline bool is_data_mapping_vma_flags (const vma_flags_t *vma_flags) +{ +const vma_flags_t mask =vma_flags_and (vma_flags, +VMA_WRITE_BIT ,VMA_SHARED_BIT ,VMA_STACK_BIT ); + +return vma_flags_same (&mask,VMA_WRITE_BIT );Ditto?
Quoted 4 lines
Quoted 2 lines
I may do both as a follow up patch given the series is a pain to rework at this point and I want at least the first version to be like-for-like intentionally.
OK sure.
Cheers!
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
mm/vma_exec.c
Quoted 8 lines
mm/vma.h +1
-0+}static inline void vma_iter_config (struct vma_iterator *vmi,unsigned long index,unsigned long last)mm/vma_exec.c +0
-0Thanks, Lorenzo
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 18/23 mm: update vma_supports_mlock() to use new VMA flags
In reply to Lorenzo Stoakes (Oracle)We now have the ability to test all of this using the new vma_flags_t approach, so let's do so.
--- mm/internal.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/internal.h +3 -1
@@ -1252,7 +1252,9 @@ static inline struct file *maybe_unlock_mmap_for_io(struct vm_fault *vmf,static inline bool vma_supports_mlock (const struct vm_area_struct *vma) { -if (vma->vm_flags & (VM_SPECIAL |VM_DROPPABLE )) +if (vma_test_any_mask (vma,VMA_SPECIAL_FLAGS )) +return false; +if (vma_test_single_mask (vma,VMA_DROPPABLE ))return false;if (vma_is_dax (vma) ||is_vm_hugetlb_page (vma))return false;
Signature · 1 line
2.53.0
Re: [PATCH v3 18/23] mm: update vma_supports_mlock() to use new VMA flags
Quoted 4 lines · Lorenzo Stoakes (Oracle)
We now have the ability to test all of this using the new
vma_flags_tapproach, so let's do so.Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Quoted 19 lines
--- mm/internal.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/internal.h +3
-1@@ -1252,7 +1252,9 @@ static inline struct file *maybe_unlock_mmap_for_io(struct vm_fault *vmf,static inline bool vma_supports_mlock (const struct vm_area_struct *vma) { -if (vma->vm_flags & (VM_SPECIAL |VM_DROPPABLE )) +if (vma_test_any_mask (vma,VMA_SPECIAL_FLAGS )) +return false; +if (vma_test_single_mask (vma,VMA_DROPPABLE ))return false;if (vma_is_dax (vma) ||is_vm_hugetlb_page (vma))return false;
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 19/23 mm/vma: introduce vma_clear_flags[_mask]()
In reply to Lorenzo Stoakes (Oracle)Introduce a helper function and helper macro to easily clear a VMA's flags using the new vma_flags_t vma->flags field:
* vma_clear_flags_mask() - Clears all of the flags in a specified mask in the VMA's flags field.
* vma_clear_flags() - Clears all of the specified individual VMA flag bits in a VMA's flags field.
Also update the VMA tests to reflect the change.
--- include/linux/mm.h | 16 ++++++++++++++++ tools/testing/vma/include/dup.h | 9 +++++++++ 2 files changed, 25 insertions(+)
include/linux/mm.h +16 -0
@@ -1368,6 +1368,22 @@ static __always_inline void vma_set_flags_mask(struct vm_area_struct *vma,#define vma_set_flags (vma, ...) \ vma_set_flags_mask(vma, mk_vma_flags(__VA_ARGS__)) +/* Helper to clear all VMA flags in a VMA. */ +static __always_inline void vma_clear_flags_mask(struct vm_area_struct *vma, +vma_flags_t flags) +{ +vma_flags_clear_mask (&vma->flags, flags); +} + +/* +* Helper macro for clearing VMA flags, e.g.: +* +* vma_clear_flags(vma, VMA_IO_BIT, VMA_PFNMAP_BIT, VMA_DONTEXPAND_BIT, +* VMA_DONTDUMP_BIT); +*/ +#define vma_clear_flags (vma, ...) \ + vma_clear_flags_mask(vma, mk_vma_flags(__VA_ARGS__)) +/* * Test whether a specific VMA flag is set in a VMA descriptor, e.g.: *
tools/testing/vma/include/dup.h +9 -0
@@ -1065,6 +1065,15 @@ static __always_inline void vma_set_flags_mask(struct vm_area_struct *vma,#define vma_set_flags (vma, ...) \ vma_set_flags_mask(vma, mk_vma_flags(__VA_ARGS__)) +static __always_inline void vma_clear_flags_mask(struct vm_area_struct *vma, +vma_flags_t flags) +{ +vma_flags_clear_mask (&vma->flags, flags); +} + +#define vma_clear_flags (vma, ...) \ + vma_clear_flags_mask(vma, mk_vma_flags(__VA_ARGS__)) + static __always_inline bool vma_desc_test(const struct vm_area_desc *desc,vma_flag_t bit) {
Signature · 1 line
2.53.0
Re: [PATCH v3 19/23] mm/vma: introduce vma_clear_flags[_mask]()
Quoted 11 lines · Lorenzo Stoakes (Oracle)
Introduce a helper function and helper macro to easily clear a VMA's flags using the new
vma_flags_tvma->flagsfield:*
vma_clear_flags_mask()- Clears all of the flags in a specified mask in the VMA's flags field.
*vma_clear_flags()- Clears all of the specified individual VMA flag bits in a VMA's flags field.Also update the VMA tests to reflect the change.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 20/23 tools/testing/vma: update VMA tests to test vma_clear_flags[_mask]()
In reply to Lorenzo Stoakes (Oracle)The tests have existing flag clearing logic, so simply expand this to use the new VMA-specific flag clearing helpers.
Also correct some trivial formatting issue in a macro define.
--- tools/testing/vma/tests/vma.c | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-)
tools/testing/vma/tests/vma.c +17 -16
@@ -347,19 +347,20 @@ static bool test_vma_flags_clear(void) , 64#endif ); -struct vm_area_struct vma; -struct vm_area_desc desc; - - vma.flags = flags; - desc.vma_flags = flags; +struct vm_area_struct vma = { + .flags = flags, + }; +struct vm_area_desc desc = { + .vma_flags = flags, + };/* Cursory check of _mask() variant, as the helper macros imply. */ vma_flags_clear_mask (&flags, mask); -vma_flags_clear_mask (&vma.flags, mask); +vma_clear_flags_mask (&vma, mask);vma_desc_clear_flags_mask (&desc, mask);#if NUM_VMA_FLAG_BITS >64 ASSERT_FALSE (vma_flags_test_any (&flags,VMA_EXEC_BIT ,64 )); -ASSERT_FALSE (vma_flags_test_any (&vma.flags,VMA_EXEC_BIT ,64 )); +ASSERT_FALSE (vma_test_any (&vma,VMA_EXEC_BIT ,64 ));ASSERT_FALSE (vma_desc_test_any (&desc,VMA_EXEC_BIT ,64 ));/* Reset. */ vma_flags_set (&flags,VMA_EXEC_BIT ,64 );
@@ -371,15 +372,15 @@ static bool test_vma_flags_clear(void)* Clear the flags and assert clear worked, then reset flags back to * include specified flags. */ -#define do_test_and_reset (...) \ - vma_flags_clear(&flags, __VA_ARGS__); \ - vma_flags_clear(&vma.flags, __VA_ARGS__); \ - vma_desc_clear_flags(&desc, __VA_ARGS__); \ - ASSERT_FALSE(vma_flags_test_any(&flags, __VA_ARGS__)); \ - ASSERT_FALSE(vma_flags_test_any(&vma.flags, __VA_ARGS__)); \ - ASSERT_FALSE(vma_desc_test_any(&desc, __VA_ARGS__)); \ - vma_flags_set(&flags, __VA_ARGS__); \ - vma_set_flags(&vma, __VA_ARGS__); \ +#define do_test_and_reset (...) \ + vma_flags_clear(&flags, __VA_ARGS__); \ + vma_clear_flags(&vma, __VA_ARGS__); \ + vma_desc_clear_flags(&desc, __VA_ARGS__); \ + ASSERT_FALSE(vma_flags_test_any(&flags, __VA_ARGS__)); \ + ASSERT_FALSE(vma_test_any(&vma, __VA_ARGS__)); \ + ASSERT_FALSE(vma_desc_test_any(&desc, __VA_ARGS__)); \ + vma_flags_set(&flags, __VA_ARGS__); \ + vma_set_flags(&vma, __VA_ARGS__); \ vma_desc_set_flags(&desc, __VA_ARGS__)/* Single flags. */
Signature · 1 line
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 21/23 mm/vma: convert as much as we can in mm/vma.c to vma_flags_t
In reply to Lorenzo Stoakes (Oracle)Now we have established a good foundation for vm_flags_t to vma_flags_t changes, update mm/vma.c to utilise vma_flags_t wherever possible.
We are able to convert VM_STARTGAP_FLAGS entirely as this is only used in mm/vma.c, and to account for the fact we can't use VM_NONE to make life easier, place the definition of this within existing #ifdef's to be cleaner.
Generally the remaining changes are mechanical.
Also update the VMA tests to reflect the changes.
--- include/linux/mm.h | 6 ++- mm/vma.c | 89 +++++++++++++++++-------------- tools/testing/vma/include/dup.h | 4 ++ tools/testing/vma/include/stubs.h | 2 +- 4 files changed, 59 insertions(+), 42 deletions(-)
include/linux/mm.h +4 -2
@@ -463,8 +463,10 @@ enum {#if defined(CONFIG_X86_USER_SHADOW_STACK ) || defined(CONFIG_ARM64_GCS ) || \ defined(CONFIG_RISCV_USER_CFI )#define VM_SHADOW_STACK INIT_VM_FLAG(SHADOW_STACK) +#define VMA_STARTGAP_FLAGS mk_vma_flags(VMA_GROWSDOWN_BIT, VMA_SHADOW_STACK_BIT)#else #define VM_SHADOW_STACK VM_NONE +#define VMA_STARTGAP_FLAGS mk_vma_flags(VMA_GROWSDOWN_BIT)#endif #if defined(CONFIG_PPC64 )#define VM_SAO INIT_VM_FLAG(SAO)
@@ -539,8 +541,6 @@ enum {/* Temporary until VMA flags conversion complete. */ #define VM_STACK_FLAGS vma_flags_to_legacy(VMA_STACK_FLAGS) -#define VM_STARTGAP_FLAGS (VM_GROWSDOWN | VM_SHADOW_STACK) -#ifdef CONFIG_MSEAL_SYSTEM_MAPPINGS #define VM_SEALED_SYSMAP VM_SEALED#else
@@ -584,6 +584,8 @@ enum {/* This mask represents all the VMA flag bits used by mlock */ #define VM_LOCKED_MASK (VM_LOCKED | VM_LOCKONFAULT) +#define VMA_LOCKED_MASK mk_vma_flags(VMA_LOCKED_BIT, VMA_LOCKONFAULT_BIT) +/* These flags can be updated atomically via VMA/mmap read lock. */ #define VM_ATOMIC_SET_ALLOWED VM_MAYBE_GUARD
mm/vma.c +50 -39
283 lines not on this page · raw message
tools/testing/vma/include/dup.h +4 -0
@@ -267,8 +267,10 @@ enum {#endif /* CONFIG_ARCH_HAS_PKEYS */ #if defined(CONFIG_X86_USER_SHADOW_STACK ) || defined(CONFIG_ARM64_GCS )#define VM_SHADOW_STACK INIT_VM_FLAG(SHADOW_STACK) +#define VMA_STARTGAP_FLAGS mk_vma_flags(VMA_GROWSDOWN_BIT, VMA_SHADOW_STACK_BIT)#else #define VM_SHADOW_STACK VM_NONE +#define VMA_STARTGAP_FLAGS mk_vma_flags(VMA_GROWSDOWN_BIT)#endif #if defined(CONFIG_PPC64 )#define VM_SAO INIT_VM_FLAG(SAO)
@@ -366,6 +368,8 @@ enum {/* This mask represents all the VMA flag bits used by mlock */ #define VM_LOCKED_MASK (VM_LOCKED | VM_LOCKONFAULT) +#define VMA_LOCKED_MASK mk_vma_flags(VMA_LOCKED_BIT, VMA_LOCKONFAULT_BIT) +#define RLIMIT_STACK 3/* max stack size */ #define RLIMIT_MEMLOCK 8/* max locked-in-memory address space */
tools/testing/vma/include/stubs.h +1 -1
@@ -229,7 +229,7 @@ static inline bool signal_pending(void *p)return false; } -static inline bool is_file_hugepages (struct file *file) +static inline bool is_file_hugepages (const struct file *file) {return false; }
Signature · 1 line
2.53.0
Re: [PATCH v3 21/23] mm/vma: convert as much as we can in mm/vma.c to vma_flags_t
Quoted 8 lines · Lorenzo Stoakes (Oracle)
Now we have established a good foundation for
vm_flags_ttovma_flags_tchanges, updatemm/vma.cto utilisevma_flags_twherever possible.We are able to convert
VM_STARTGAP_FLAGSentirely as this is only used inmm/vma.c, and to account for the fact we can't useVM_NONEto make life easier, place the definition of this within existing #ifdef's to be cleaner.
Quoted 5 lines
Generally the remaining changes are mechanical.
Also update the VMA tests to reflect the changes.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Nits:
Quoted 14 lines
mm/vma.c +4
-1@@ -2338,8 +2339,11 @@ void mm_drop_all_locks(struct mm_struct *mm)* We account for memory if it's a private writeable mapping, * not hugepages and VM_NORESERVE wasn't set. */ -static bool accountable_mapping (struct file *file,vm_flags_t vm_flags) +static bool accountable_mapping (struct mmap_state *map) { +const struct file *file = map->file; +vma_flags_t mask; +/* * hugetlb has its own accounting separate from the core VM * VM_HUGETLB may not be set yet so we cannot check for that flag.
mm/vma.c static bool accountable_mapping(struct file *file, vm_flags_t vm_flags) · line 2356
Quoted 7 lines
mm/vma.c +3
-1@@ -2347,7 +2351,9 @@ static bool accountable_mapping(struct file *file, vm_flags_t vm_flags)if (file &&is_file_hugepages (file))return false; -return (vm_flags & (VM_NORESERVE |VM_SHARED |VM_WRITE )) ==VM_WRITE ; + mask =vma_flags_and (&map->vma_flags,VMA_NORESERVE_BIT ,VMA_SHARED_BIT , +VMA_WRITE_BIT ); +return vma_flags_same (&mask,VMA_WRITE_BIT );
Another case of possible refactor, if you agree with those pointed out in earlier patch.
mm/vma.c unsigned long unmapped_area(struct vm_unmapped_area_info *info) · line 3002
Quoted 11 lines
mm/vma.c +2
-1}/* @@ -2993,7 +2998,8 @@ unsigned long unmapped_area(struct vm_unmapped_area_info *info) gap =vma_iter_addr (&vmi) + info->start_gap; gap += (info->align_offset - gap) & info->align_mask; tmp =vma_next (&vmi); -if (tmp && (tmp->vm_flags &VM_STARTGAP_FLAGS )) {/* Avoid prev check if possible */ +/* Avoid prev check if possible */ +if (tmp && (vma_test_any_mask (tmp,VMA_STARTGAP_FLAGS ))) {
The parentheses around function call not necessary?
mm/vma.c unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info) · line 3055
Quoted 10 lines
mm/vma.c +2
-1if (vm_start_gap (tmp) < gap + length -1 ) { low_limit = tmp->vm_end;vma_iter_reset (&vmi);@@ -3045,7 +3051,8 @@ unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info) gap -= (gap - info->align_offset) & info->align_mask; gap_end =vma_iter_end (&vmi); tmp =vma_next (&vmi); -if (tmp && (tmp->vm_flags &VM_STARTGAP_FLAGS )) {/* Avoid prev check if possible */ +/* Avoid prev check if possible */ +if (tmp && (vma_test_any_mask (tmp,VMA_STARTGAP_FLAGS ))) {
Same.
Quoted 3 lines
mm/vma.c +0
-0if (vm_start_gap (tmp) < gap_end) { high_limit =vm_start_gap (tmp);vma_iter_reset (&vmi);
Lorenzo Stoakes (Oracle)
maintainerRe: [PATCH v3 21/23] mm/vma: convert as much as we can in mm/vma.c to vma_flags_t
Quoted 9 lines · Vlastimil Babka (SUSE)
Quoted 8 lines · Lorenzo Stoakes (Oracle)
Now we have established a good foundation for
vm_flags_ttovma_flags_tchanges, updatemm/vma.cto utilisevma_flags_twherever possible.We are able to convert
VM_STARTGAP_FLAGSentirely as this is only used inmm/vma.c, and to account for the fact we can't useVM_NONEto make life easier, place the definition of this within existing #ifdef's to be cleaner.
Quoted 7 lines
Quoted 5 lines
Generally the remaining changes are mechanical.
Also update the VMA tests to reflect the changes.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Thanks!
Quoted 20 lines
Nits:
Quoted 17 lines
mm/vma.c +4
-1@@ -2338,8 +2339,11 @@ void mm_drop_all_locks(struct mm_struct *mm)* We account for memory if it's a private writeable mapping, * not hugepages and VM_NORESERVE wasn't set. */ -static bool accountable_mapping (struct file *file,vm_flags_t vm_flags) +static bool accountable_mapping (struct mmap_state *map) { +const struct file *file = map->file; +vma_flags_t mask; +/* * hugetlb has its own accounting separate from the core VM * VM_HUGETLB may not be set yet so we cannot check for that flag. @@ -2347,7 +2351,9 @@ static bool accountable_mapping(struct file *file, vm_flags_t vm_flags)if (file &&is_file_hugepages (file))return false;
Quoted 7 lines
mm/vma.c static bool accountable_mapping(struct file *file, vm_flags_t vm_flags) · line 2356
Quoted 4 lines
mm/vma.c +3
-1-return (vm_flags & (VM_NORESERVE |VM_SHARED |VM_WRITE )) ==VM_WRITE ; + mask =vma_flags_and (&map->vma_flags,VMA_NORESERVE_BIT ,VMA_SHARED_BIT , +VMA_WRITE_BIT ); +return vma_flags_same (&mask,VMA_WRITE_BIT );Another case of possible refactor, if you agree with those pointed out in earlier patch.
Ack
Quoted 5 lines
Quoted 4 lines
(quoted diff) +0
-0} /*
Quoted 9 lines
mm/vma.c unsigned long unmapped_area(struct vm_unmapped_area_info *info) · line 3002
Quoted 7 lines
mm/vma.c +2
-1@@ -2993,7 +2998,8 @@ unsigned long unmapped_area(struct vm_unmapped_area_info *info) gap =vma_iter_addr (&vmi) + info->start_gap; gap += (info->align_offset - gap) & info->align_mask; tmp =vma_next (&vmi); -if (tmp && (tmp->vm_flags &VM_STARTGAP_FLAGS )) {/* Avoid prev check if possible */ +/* Avoid prev check if possible */ +if (tmp && (vma_test_any_mask (tmp,VMA_STARTGAP_FLAGS ))) {The parentheses around function call not necessary?
True, can fix up.
Quoted 13 lines
mm/vma.c unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info) · line 3055
Quoted 11 lines
mm/vma.c +2
-1if (vm_start_gap (tmp) < gap + length -1 ) { low_limit = tmp->vm_end;vma_iter_reset (&vmi);@@ -3045,7 +3051,8 @@ unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info) gap -= (gap - info->align_offset) & info->align_mask; gap_end =vma_iter_end (&vmi); tmp =vma_next (&vmi); -if (tmp && (tmp->vm_flags &VM_STARTGAP_FLAGS )) {/* Avoid prev check if possible */ +/* Avoid prev check if possible */ +if (tmp && (vma_test_any_mask (tmp,VMA_STARTGAP_FLAGS ))) {Same.
True, will fix up.
Quoted 4 lines
mm/vma.c unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info) · line 3058
Quoted 4 lines
mm/vma.c +0
-0if (vm_start_gap (tmp) < gap_end) { high_limit =vm_start_gap (tmp);vma_iter_reset (&vmi);
Cheers, Lorenzo
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 22/23 mm/vma: convert vma_modify_flags[_uffd]() to use vma_flags_t
In reply to Lorenzo Stoakes (Oracle)Update the vma_modify_flags() and vma_modify_flags_uffd() functions to accept a vma_flags_t parameter rather than a vm_flags_t one, and propagate the changes as needed to implement this change.
Finally, update the VMA tests to reflect this.
--- include/linux/userfaultfd_k.h | 3 +++ mm/madvise.c | 10 +++++---- mm/mlock.c | 38 ++++++++++++++++++--------------- mm/mprotect.c | 7 +++--- mm/mseal.c | 11 ++++++---- mm/userfaultfd.c | 21 ++++++++++++------ mm/vma.c | 15 +++++++------ mm/vma.h | 15 ++++++------- tools/testing/vma/tests/merge.c | 3 +-- 9 files changed, 70 insertions(+), 53 deletions(-)
include/linux/userfaultfd_k.h +3 -0
@@ -23,6 +23,9 @@/* The set of all possible UFFD-related VM flags. */ #define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR) +#define __VMA_UFFD_FLAGS mk_vma_flags(VMA_UFFD_MISSING_BIT, VMA_UFFD_WP_BIT, \ + VMA_UFFD_MINOR_BIT) +/* * CAREFUL: Check include/uapi/asm-generic/fcntl.h when defining * new flags, since they might collide with O_* ones. We want
mm/madvise.c +6 -4
@@ -151,13 +151,15 @@ static int madvise_update_vma(vm_flags_t new_flags, struct madvise_behavior *madv_behavior) {struct vm_area_struct *vma = madv_behavior->vma; +vma_flags_t new_vma_flags =legacy_to_vma_flags (new_flags);struct madvise_behavior_range *range = &madv_behavior->range;struct anon_vma_name *anon_name = madv_behavior->anon_name;bool set_new_anon_name = madv_behavior->behavior == __MADV_SET_ANON_VMA_NAME;VMA_ITERATOR (vmi, madv_behavior->mm, range->start); -if (new_flags == vma->vm_flags && (!set_new_anon_name || -anon_vma_name_eq (anon_vma_name (vma), anon_name))) +if (vma_flags_same_mask (&vma->flags, new_vma_flags) && + (!set_new_anon_name || +anon_vma_name_eq (anon_vma_name (vma), anon_name)))return 0 ;if (set_new_anon_name)
@@ -165,7 +167,7 @@ static int madvise_update_vma(vm_flags_t new_flags, range->start, range->end, anon_name); else vma =vma_modify_flags (&vmi, madv_behavior->prev, vma, - range->start, range->end, &new_flags); + range->start, range->end, &new_vma_flags);if (IS_ERR (vma))return PTR_ERR (vma);
@@ -174,7 +176,7 @@ static int madvise_update_vma(vm_flags_t new_flags,/* vm_flags is protected by the mmap_lock held in write mode. */ vma_start_write (vma); -vm_flags_reset (vma, new_flags); + vma->flags = new_vma_flags;if (set_new_anon_name)return replace_anon_vma_name (vma, anon_name);
mm/mlock.c +21 -17
@@ -415,13 +415,14 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr,* @vma - vma containing range to be mlock()ed or munlock()ed * @start - start address in @vma of the range * @end - end of range in @vma -* @newflags - the new set of flags for @vma. +* @new_vma_flags - the new set of flags for @vma. * * Called for mlock(), mlock2() and mlockall(), to set @vma VM_LOCKED; * called for munlock() and munlockall(), to clear VM_LOCKED from @vma. */ static void mlock_vma_pages_range (struct vm_area_struct *vma, -unsigned long start,unsigned long end,vm_flags_t newflags) +unsigned long start,unsigned long end, +vma_flags_t *new_vma_flags) {static const struct mm_walk_ops mlock_walk_ops = { .pmd_entry = mlock_pte_range,
@@ -439,18 +440,18 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma,* combination should not be visible to other mmap_lock users; * but WRITE_ONCE so rmap walkers must see VM_IO if VM_LOCKED. */ -if (newflags &VM_LOCKED ) - newflags |=VM_IO ; +if (vma_flags_test (new_vma_flags,VMA_LOCKED_BIT )) +vma_flags_set (new_vma_flags,VMA_IO_BIT );vma_start_write (vma); -vm_flags_reset_once (vma, newflags); +WRITE_ONCE (vma->flags, *new_vma_flags);lru_add_drain ();walk_page_range (vma->vm_mm, start, end, &mlock_walk_ops,NULL );lru_add_drain (); -if (newflags &VM_IO ) { - newflags &= ~VM_IO ; -vm_flags_reset_once (vma, newflags); +if (vma_flags_test (new_vma_flags,VMA_IO_BIT )) { +vma_flags_clear (new_vma_flags,VMA_IO_BIT ); +WRITE_ONCE (vma->flags, *new_vma_flags); } }
@@ -467,20 +468,22 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, struct vm_area_struct **prev, unsigned long start, unsigned long end, vm_flags_t newflags) { + vma_flags_t new_vma_flags = legacy_to_vma_flags(newflags); +const vma_flags_t old_vma_flags = vma->flags;struct mm_struct *mm = vma->vm_mm;int nr_pages;int ret =0 ; -vm_flags_t oldflags = vma->vm_flags; -if (newflags == oldflags ||vma_is_secretmem (vma) || - !vma_supports_mlock (vma)) +if (vma_flags_same_pair (&old_vma_flags, &new_vma_flags) || +vma_is_secretmem (vma) || !vma_supports_mlock (vma)) {/* * Don't set VM_LOCKED or VM_LOCKONFAULT and don't count. * For secretmem, don't allow the memory to be unlocked. */ goto out; + } - vma =vma_modify_flags (vmi, *prev, vma, start, end, &newflags); + vma =vma_modify_flags (vmi, *prev, vma, start, end, &new_vma_flags);if (IS_ERR (vma)) { ret =PTR_ERR (vma); goto out;
@@ -490,9 +493,9 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma,* Keep track of amount of locked VM. */ nr_pages = (end - start) >>PAGE_SHIFT ; -if (!(newflags &VM_LOCKED )) +if (!vma_flags_test (&new_vma_flags,VMA_LOCKED_BIT )) nr_pages = -nr_pages; -else if (oldflags &VM_LOCKED ) +else if (vma_flags_test (&old_vma_flags,VMA_LOCKED_BIT )) nr_pages =0 ; mm->locked_vm += nr_pages;
@@ -501,12 +504,13 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma,* It's okay if try_to_unmap_one unmaps a page just after we * set VM_LOCKED, populate_vma_page_range will bring it back. */ -if ((newflags &VM_LOCKED ) && (oldflags &VM_LOCKED )) { +if (vma_flags_test (&new_vma_flags,VMA_LOCKED_BIT ) && +vma_flags_test (&old_vma_flags,VMA_LOCKED_BIT )) {/* No work to do, and mlocking twice would be wrong */ vma_start_write (vma); -vm_flags_reset (vma, newflags); + vma->flags = new_vma_flags; }else { -mlock_vma_pages_range (vma, start, end, newflags); +mlock_vma_pages_range (vma, start, end, &new_vma_flags); } out: *prev = vma;
mm/mprotect.c +3 -4
@@ -756,13 +756,11 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,vma_flags_clear (&new_vma_flags,VMA_ACCOUNT_BIT ); } - newflags =vma_flags_to_legacy (new_vma_flags); - vma =vma_modify_flags (vmi, *pprev, vma, start, end, &newflags); + vma =vma_modify_flags (vmi, *pprev, vma, start, end, &new_vma_flags);if (IS_ERR (vma)) { error =PTR_ERR (vma); goto fail; } - new_vma_flags =legacy_to_vma_flags (newflags); *pprev = vma;
@@ -771,7 +769,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,* held in write mode. */ vma_start_write (vma); -vm_flags_reset_once (vma, newflags); +WRITE_ONCE (vma->flags, new_vma_flags);if (vma_wants_manual_pte_write_upgrade (vma)) mm_cp_flags |=MM_CP_TRY_CHANGE_WRITABLE ;vma_set_page_prot (vma);
@@ -796,6 +794,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, }vm_stat_account (mm,vma_flags_to_legacy (old_vma_flags), -nrpages); + newflags =vma_flags_to_legacy (new_vma_flags);vm_stat_account (mm, newflags, nrpages);perf_event_mmap (vma);return 0 ;
mm/mseal.c +7 -4
@@ -68,14 +68,17 @@ static int mseal_apply(struct mm_struct *mm,for_each_vma_range (vmi, vma, end) {const unsigned long curr_end =MIN (vma->vm_end, end); -if (!(vma->vm_flags &VM_SEALED )) { -vm_flags_t vm_flags = vma->vm_flags |VM_SEALED ; +if (!vma_test (vma,VMA_SEALED_BIT )) { +vma_flags_t vma_flags = vma->flags; + +vma_flags_set (&vma_flags,VMA_SEALED_BIT ); vma =vma_modify_flags (&vmi, prev, vma, curr_start, - curr_end, &vm_flags); + curr_end, &vma_flags);if (IS_ERR (vma))return PTR_ERR (vma); -vm_flags_set (vma,VM_SEALED ); +vma_start_write (vma); +vma_set_flags (vma,VMA_SEALED_BIT ); } prev = vma;
mm/userfaultfd.c +14 -7
@@ -2093,6 +2093,9 @@ struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi, {struct vm_area_struct *ret;bool give_up_on_oom = false; +vma_flags_t new_vma_flags = vma->flags; + +vma_flags_clear_mask (&new_vma_flags, __VMA_UFFD_FLAGS);/* * If we are modifying only and not splitting, just give up on the merge
@@ -2106,8 +2109,8 @@ struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,uffd_wp_range (vma, start, end - start, false); ret =vma_modify_flags_uffd (vmi, prev, vma, start, end, - vma->vm_flags & ~__VM_UFFD_FLAGS, -NULL_VM_UFFD_CTX , give_up_on_oom); + &new_vma_flags,NULL_VM_UFFD_CTX , + give_up_on_oom);/* * In the vma_merge() successful mprotect-like case 8:
@@ -2127,10 +2130,11 @@ int userfaultfd_register_range(struct userfaultfd_ctx *ctx, unsigned long start, unsigned long end, bool wp_async) { + vma_flags_t vma_flags = legacy_to_vma_flags(vm_flags);VMA_ITERATOR (vmi, ctx->mm, start);struct vm_area_struct *prev =vma_prev (&vmi);unsigned long vma_end; -vm_flags_t new_flags; +vma_flags_t new_vma_flags;if (vma->vm_start < start) prev = vma;
@@ -2141,23 +2145,26 @@ int userfaultfd_register_range(struct userfaultfd_ctx *ctx,VM_WARN_ON_ONCE (!vma_can_userfault (vma, vm_flags, wp_async));VM_WARN_ON_ONCE (vma->vm_userfaultfd_ctx.ctx && vma->vm_userfaultfd_ctx.ctx != ctx); -VM_WARN_ON_ONCE (!(vma->vm_flags &VM_MAYWRITE )); +VM_WARN_ON_ONCE (!vma_test (vma,VMA_MAYWRITE_BIT ));/* * Nothing to do: this vma is already registered into this * userfaultfd and with the right tracking mode too. */ if (vma->vm_userfaultfd_ctx.ctx == ctx && - (vma->vm_flags & vm_flags) == vm_flags) +vma_test_all_mask (vma, vma_flags)) goto skip;if (vma->vm_start > start) start = vma->vm_start; vma_end =min (end, vma->vm_end); - new_flags = (vma->vm_flags & ~__VM_UFFD_FLAGS) | vm_flags; + new_vma_flags = vma->flags; +vma_flags_clear_mask (&new_vma_flags, __VMA_UFFD_FLAGS); +vma_flags_set_mask (&new_vma_flags, vma_flags); + vma =vma_modify_flags_uffd (&vmi, prev, vma, start, vma_end, - new_flags, + &new_vma_flags, (struct vm_userfaultfd_ctx ){ctx},/* give_up_on_oom = */ false);if (IS_ERR (vma))
mm/vma.c +8 -7
@@ -1710,13 +1710,13 @@ static struct vm_area_struct *vma_modify(struct vma_merge_struct *vmg)struct vm_area_struct *vma_modify_flags (struct vma_iterator *vmi,struct vm_area_struct *prev,struct vm_area_struct *vma,unsigned long start,unsigned long end, -vm_flags_t *vm_flags_ptr) +vma_flags_t *vma_flags_ptr) {VMG_VMA_STATE (vmg, vmi, prev, vma, start, end); -const vm_flags_t vm_flags = *vm_flags_ptr; +const vma_flags_t vma_flags = *vma_flags_ptr;struct vm_area_struct *ret; - vmg.vm_flags = vm_flags; + vmg.vma_flags = vma_flags; ret =vma_modify (&vmg);if (IS_ERR (ret))
@@ -1728,7 +1728,7 @@ struct vm_area_struct *vma_modify_flags(struct vma_iterator *vmi,* them to the caller. */ if (vmg.state ==VMA_MERGE_SUCCESS ) - *vm_flags_ptr = ret->vm_flags; + *vma_flags_ptr = ret->flags;return ret; }
@@ -1758,12 +1758,13 @@ struct vm_area_struct *vma_modify_policy(struct vma_iterator *vmi,struct vm_area_struct *vma_modify_flags_uffd (struct vma_iterator *vmi,struct vm_area_struct *prev,struct vm_area_struct *vma, -unsigned long start,unsigned long end,vm_flags_t vm_flags, -struct vm_userfaultfd_ctx new_ctx,bool give_up_on_oom) +unsigned long start,unsigned long end, +const vma_flags_t *vma_flags,struct vm_userfaultfd_ctx new_ctx, +bool give_up_on_oom) {VMG_VMA_STATE (vmg, vmi, prev, vma, start, end); - vmg.vm_flags = vm_flags; + vmg.vma_flags = *vma_flags; vmg.uffd_ctx = new_ctx;if (give_up_on_oom) vmg.give_up_on_oom = true;
mm/vma.h +7 -8
@@ -342,24 +342,23 @@ void unmap_region(struct unmap_desc *unmap);* @vma: The VMA containing the range @start to @end to be updated. * @start: The start of the range to update. May be offset within @vma. * @end: The exclusive end of the range to update, may be offset within @vma. -* @vm_flags_ptr: A pointer to the VMA flags that the @start to @end range is +* @vma_flags_ptr: A pointer to the VMA flags that the @start to @end range is * about to be set to. On merge, this will be updated to include sticky flags. * * IMPORTANT: The actual modification being requested here is NOT applied, * rather the VMA is perhaps split, perhaps merged to accommodate the change, * and the caller is expected to perform the actual modification. * -* In order to account for sticky VMA flags, the @vm_flags_ptr parameter points +* In order to account for sticky VMA flags, the @vma_flags_ptr parameter points * to the requested flags which are then updated so the caller, should they * overwrite any existing flags, correctly retains these. * * Returns: A VMA which contains the range @start to @end ready to have its -* flags altered to *@vm_flags. +* flags altered to *@vma_flags. */ __must_check struct vm_area_struct *vma_modify_flags (struct vma_iterator *vmi,struct vm_area_struct *prev,struct vm_area_struct *vma, -unsigned long start,unsigned long end, -vm_flags_t *vm_flags_ptr); +unsigned long start,unsigned long end,vma_flags_t *vma_flags_ptr);/** * vma_modify_name() - Perform any necessary split/merge in preparation for
@@ -418,7 +417,7 @@ __must_check struct vm_area_struct *vma_modify_policy(struct vma_iterator *vmi,* @vma: The VMA containing the range @start to @end to be updated. * @start: The start of the range to update. May be offset within @vma. * @end: The exclusive end of the range to update, may be offset within @vma. -* @vm_flags: The VMA flags that the @start to @end range is about to be set to. +* @vma_flags: The VMA flags that the @start to @end range is about to be set to. * @new_ctx: The userfaultfd context that the @start to @end range is about to * be set to. * @give_up_on_oom: If an out of memory condition occurs on merge, simply give
@@ -429,11 +428,11 @@ __must_check struct vm_area_struct *vma_modify_policy(struct vma_iterator *vmi,* and the caller is expected to perform the actual modification. * * Returns: A VMA which contains the range @start to @end ready to have its VMA -* flags changed to @vm_flags and its userfaultfd context changed to @new_ctx. +* flags changed to @vma_flags and its userfaultfd context changed to @new_ctx. */ __must_check struct vm_area_struct *vma_modify_flags_uffd (struct vma_iterator *vmi,struct vm_area_struct *prev,struct vm_area_struct *vma, -unsigned long start,unsigned long end,vm_flags_t vm_flags, +unsigned long start,unsigned long end,const vma_flags_t *vma_flags,struct vm_userfaultfd_ctx new_ctx,bool give_up_on_oom);__must_check struct vm_area_struct *vma_merge_new_range (struct vma_merge_struct *vmg);
tools/testing/vma/tests/merge.c +1 -2
@@ -132,7 +132,6 @@ static bool test_simple_modify(void)struct vm_area_struct *vma;vma_flags_t vma_flags =mk_vma_flags (VMA_READ_BIT ,VMA_WRITE_BIT ,VMA_MAYREAD_BIT ,VMA_MAYWRITE_BIT ); -vm_flags_t legacy_flags =VM_READ |VM_WRITE ;struct mm_struct mm = {};struct vm_area_struct *init_vma =alloc_vma (&mm,0 ,0x3000 ,0 , vma_flags);VMA_ITERATOR (vmi, &mm,0x1000 );
@@ -144,7 +143,7 @@ static bool test_simple_modify(void)* performs the merge/split only. */ vma =vma_modify_flags (&vmi, init_vma, init_vma, -0x1000 ,0x2000 , &legacy_flags); +0x1000 ,0x2000 , &vma_flags);ASSERT_NE (vma,NULL );/* We modify the provided VMA, and on split allocate new VMAs. */ ASSERT_EQ (vma, init_vma);
Signature · 1 line
2.53.0
Vlastimil Babka (SUSE)
reviewerRe: [PATCH v3 22/23] mm/vma: convert vma_modify_flags[_uffd]() to use vma_flags_t
Quoted 28 lines · Lorenzo Stoakes (Oracle)
Update the
vma_modify_flags()andvma_modify_flags_uffd()functions to accept avma_flags_tparameter rather than avm_flags_tone, and propagate the changes as needed to implement this change.Finally, update the VMA tests to reflect this.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>mm/mlock.c +3
-2@@ -415,13 +415,14 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr,* @vma - vma containing range to be mlock()ed or munlock()ed * @start - start address in @vma of the range * @end - end of range in @vma -* @newflags - the new set of flags for @vma. +* @new_vma_flags - the new set of flags for @vma. * * Called for mlock(), mlock2() and mlockall(), to set @vma VM_LOCKED; * called for munlock() and munlockall(), to clear VM_LOCKED from @vma. */ static void mlock_vma_pages_range (struct vm_area_struct *vma, -unsigned long start,unsigned long end,vm_flags_t newflags) +unsigned long start,unsigned long end, +vma_flags_t *new_vma_flags) {static const struct mm_walk_ops mlock_walk_ops = { .pmd_entry = mlock_pte_range,
mm/mlock.c static void mlock_vma_pages_range(struct vm_area_struct *vma, · line 446
Quoted 10 lines
mm/mlock.c +3
-3@@ -439,18 +440,18 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma,* combination should not be visible to other mmap_lock users; * but WRITE_ONCE so rmap walkers must see VM_IO if VM_LOCKED. */ -if (newflags &VM_LOCKED ) - newflags |=VM_IO ; +if (vma_flags_test (new_vma_flags,VMA_LOCKED_BIT )) +vma_flags_set (new_vma_flags,VMA_IO_BIT );vma_start_write (vma); -vm_flags_reset_once (vma, newflags); +WRITE_ONCE (vma->flags, *new_vma_flags);
It's not clear to me, how is switching from vm_flags_t to vma_flags_t allowing us to simply do WRITE_ONCE() instead of the full logic of vm_flags_reset_once()? Won't it fail to compile once once flags are more than single word? Or worse, will compile but silently allow tearing?
mm/mlock.c static void mlock_vma_pages_range(struct vm_area_struct *vma, · line 454
Quoted 11 lines
mm/mlock.c +3
-3lru_add_drain ();walk_page_range (vma->vm_mm, start, end, &mlock_walk_ops,NULL );lru_add_drain (); -if (newflags &VM_IO ) { - newflags &= ~VM_IO ; -vm_flags_reset_once (vma, newflags); +if (vma_flags_test (new_vma_flags,VMA_IO_BIT )) { +vma_flags_clear (new_vma_flags,VMA_IO_BIT ); +WRITE_ONCE (vma->flags, *new_vma_flags);
Ditto.
Quoted 43 lines
mm/mlock.c +8
-6} }@@ -467,20 +468,22 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, struct vm_area_struct **prev, unsigned long start, unsigned long end, vm_flags_t newflags) { + vma_flags_t new_vma_flags = legacy_to_vma_flags(newflags); +const vma_flags_t old_vma_flags = vma->flags;struct mm_struct *mm = vma->vm_mm;int nr_pages;int ret =0 ; -vm_flags_t oldflags = vma->vm_flags; -if (newflags == oldflags ||vma_is_secretmem (vma) || - !vma_supports_mlock (vma)) +if (vma_flags_same_pair (&old_vma_flags, &new_vma_flags) || +vma_is_secretmem (vma) || !vma_supports_mlock (vma)) {/* * Don't set VM_LOCKED or VM_LOCKONFAULT and don't count. * For secretmem, don't allow the memory to be unlocked. */ goto out; + } - vma =vma_modify_flags (vmi, *prev, vma, start, end, &newflags); + vma =vma_modify_flags (vmi, *prev, vma, start, end, &new_vma_flags);if (IS_ERR (vma)) { ret =PTR_ERR (vma); goto out;@@ -490,9 +493,9 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma,* Keep track of amount of locked VM. */ nr_pages = (end - start) >>PAGE_SHIFT ; -if (!(newflags &VM_LOCKED )) +if (!vma_flags_test (&new_vma_flags,VMA_LOCKED_BIT )) nr_pages = -nr_pages; -else if (oldflags &VM_LOCKED ) +else if (vma_flags_test (&old_vma_flags,VMA_LOCKED_BIT )) nr_pages =0 ; mm->locked_vm += nr_pages;
mm/mlock.c static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 511
Quoted 10 lines
mm/mlock.c +3
-2@@ -501,12 +504,13 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma,* It's okay if try_to_unmap_one unmaps a page just after we * set VM_LOCKED, populate_vma_page_range will bring it back. */ -if ((newflags &VM_LOCKED ) && (oldflags &VM_LOCKED )) { +if (vma_flags_test (&new_vma_flags,VMA_LOCKED_BIT ) && +vma_flags_test (&old_vma_flags,VMA_LOCKED_BIT )) {/* No work to do, and mlocking twice would be wrong */ vma_start_write (vma); -vm_flags_reset (vma, newflags); + vma->flags = new_vma_flags;
This also does lot less than vm_flags_reset()?
Quoted 26 lines
mm/mlock.c +1
-1}else { -mlock_vma_pages_range (vma, start, end, newflags); +mlock_vma_pages_range (vma, start, end, &new_vma_flags); } out: *prev = vma;mm/mprotect.c +1
-3@@ -756,13 +756,11 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,vma_flags_clear (&new_vma_flags,VMA_ACCOUNT_BIT ); } - newflags =vma_flags_to_legacy (new_vma_flags); - vma =vma_modify_flags (vmi, *pprev, vma, start, end, &newflags); + vma =vma_modify_flags (vmi, *pprev, vma, start, end, &new_vma_flags);if (IS_ERR (vma)) { error =PTR_ERR (vma); goto fail; } - new_vma_flags =legacy_to_vma_flags (newflags); *pprev = vma;
mm/mprotect.c mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, · line 772
Quoted 5 lines
mm/mprotect.c +1
-1@@ -771,7 +769,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,* held in write mode. */ vma_start_write (vma); -vm_flags_reset_once (vma, newflags); +WRITE_ONCE (vma->flags, new_vma_flags);
Ditto.
Quoted 13 lines
mm/mprotect.c +1
-0if (vma_wants_manual_pte_write_upgrade (vma)) mm_cp_flags |=MM_CP_TRY_CHANGE_WRITABLE ;vma_set_page_prot (vma);@@ -796,6 +794,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, }vm_stat_account (mm,vma_flags_to_legacy (old_vma_flags), -nrpages); + newflags =vma_flags_to_legacy (new_vma_flags);vm_stat_account (mm, newflags, nrpages);perf_event_mmap (vma);return 0 ;mm/mseal.c +0
-0
Lorenzo Stoakes (Oracle)
maintainerRe: [PATCH v3 22/23] mm/vma: convert vma_modify_flags[_uffd]() to use vma_flags_t
Quoted 36 lines · Vlastimil Babka (SUSE)
Quoted 35 lines · Lorenzo Stoakes (Oracle)
Update the
vma_modify_flags()andvma_modify_flags_uffd()functions to accept avma_flags_tparameter rather than avm_flags_tone, and propagate the changes as needed to implement this change.Finally, update the VMA tests to reflect this.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>mm/mlock.c +5
-4@@ -415,13 +415,14 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr,* @vma - vma containing range to be mlock()ed or munlock()ed * @start - start address in @vma of the range * @end - end of range in @vma -* @newflags - the new set of flags for @vma. +* @new_vma_flags - the new set of flags for @vma. * * Called for mlock(), mlock2() and mlockall(), to set @vma VM_LOCKED; * called for munlock() and munlockall(), to clear VM_LOCKED from @vma. */ static void mlock_vma_pages_range (struct vm_area_struct *vma, -unsigned long start,unsigned long end,vm_flags_t newflags) +unsigned long start,unsigned long end, +vma_flags_t *new_vma_flags) {static const struct mm_walk_ops mlock_walk_ops = { .pmd_entry = mlock_pte_range,@@ -439,18 +440,18 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma,* combination should not be visible to other mmap_lock users; * but WRITE_ONCE so rmap walkers must see VM_IO if VM_LOCKED. */ -if (newflags &VM_LOCKED ) - newflags |=VM_IO ; +if (vma_flags_test (new_vma_flags,VMA_LOCKED_BIT )) +vma_flags_set (new_vma_flags,VMA_IO_BIT );
Quoted 8 lines
mm/mlock.c static void mlock_vma_pages_range(struct vm_area_struct *vma, · line 446
Quoted 3 lines
mm/mlock.c +1
-1vma_start_write (vma); -vm_flags_reset_once (vma, newflags); +WRITE_ONCE (vma->flags, *new_vma_flags);It's not clear to me, how is switching from
vm_flags_ttovma_flags_tallowing us to simply doWRITE_ONCE()instead of the full logic ofvm_flags_reset_once()? Won't it fail to compile once once flags are more than single word? Or worse, will compile but silently allow tearing?
We only care about tearing in the flags that can be contained within a system word, but true we should probably do this more carefully, as I did for vm_flags_reset_once().
I will reimplement this as a new, hideous, helper function.
I am not a fan of this being a thing to handle races, it's a hack. But I guess that should be addressed separately.
Quoted 6 lines
Quoted 6 lines
mm/mlock.c +0
-0lru_add_drain ();walk_page_range (vma->vm_mm, start, end, &mlock_walk_ops,NULL );lru_add_drain ();
Quoted 8 lines
mm/mlock.c static void mlock_vma_pages_range(struct vm_area_struct *vma, · line 454
Quoted 6 lines
mm/mlock.c +3
-3-if (newflags &VM_IO ) { - newflags &= ~VM_IO ; -vm_flags_reset_once (vma, newflags); +if (vma_flags_test (new_vma_flags,VMA_IO_BIT )) { +vma_flags_clear (new_vma_flags,VMA_IO_BIT ); +WRITE_ONCE (vma->flags, *new_vma_flags);Ditto.
Yup.
Quoted 43 lines
Quoted 43 lines
mm/mlock.c +8
-6} }@@ -467,20 +468,22 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, struct vm_area_struct **prev, unsigned long start, unsigned long end, vm_flags_t newflags) { + vma_flags_t new_vma_flags = legacy_to_vma_flags(newflags); +const vma_flags_t old_vma_flags = vma->flags;struct mm_struct *mm = vma->vm_mm;int nr_pages;int ret =0 ; -vm_flags_t oldflags = vma->vm_flags; -if (newflags == oldflags ||vma_is_secretmem (vma) || - !vma_supports_mlock (vma)) +if (vma_flags_same_pair (&old_vma_flags, &new_vma_flags) || +vma_is_secretmem (vma) || !vma_supports_mlock (vma)) {/* * Don't set VM_LOCKED or VM_LOCKONFAULT and don't count. * For secretmem, don't allow the memory to be unlocked. */ goto out; + } - vma =vma_modify_flags (vmi, *prev, vma, start, end, &newflags); + vma =vma_modify_flags (vmi, *prev, vma, start, end, &new_vma_flags);if (IS_ERR (vma)) { ret =PTR_ERR (vma); goto out;@@ -490,9 +493,9 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma,* Keep track of amount of locked VM. */ nr_pages = (end - start) >>PAGE_SHIFT ; -if (!(newflags &VM_LOCKED )) +if (!vma_flags_test (&new_vma_flags,VMA_LOCKED_BIT )) nr_pages = -nr_pages; -else if (oldflags &VM_LOCKED ) +else if (vma_flags_test (&old_vma_flags,VMA_LOCKED_BIT )) nr_pages =0 ; mm->locked_vm += nr_pages;
Quoted 13 lines
mm/mlock.c static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 511
Quoted 11 lines
mm/mlock.c +3
-2@@ -501,12 +504,13 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma,* It's okay if try_to_unmap_one unmaps a page just after we * set VM_LOCKED, populate_vma_page_range will bring it back. */ -if ((newflags &VM_LOCKED ) && (oldflags &VM_LOCKED )) { +if (vma_flags_test (&new_vma_flags,VMA_LOCKED_BIT ) && +vma_flags_test (&old_vma_flags,VMA_LOCKED_BIT )) {/* No work to do, and mlocking twice would be wrong */ vma_start_write (vma); -vm_flags_reset (vma, newflags); + vma->flags = new_vma_flags;This also does lot less than
vm_flags_reset()?
Well let's look:
VM_WARN_ON_ONCE(!pgtable_supports_soft_dirty() && (flags & VM_SOFTDIRTY));
Are we really at a point where this is problematic? Do we hit this? Why are we specifically checking only this case on every single instance of resetting VMA flags?
vma_assert_write_locked(vma);
Note the vma_start_write() line above. I want to separate vma_flags_t helpers from asserts like that, because:
1. We might be operating on a VMA that is not yet added to the tree
2. We might be operating on a VMA that is now detached
3. Really in all but core code, you should be using vma_desc_xxx().
4. Other VMA fields are manipulated with no such checks.
5. It'd be egregious to have to add variants of flag functions just to account for cases such as the above, especially when we don't do so for other VMA fields. Drivers are the problematic cases and why it was especially important (and also for debug as VMA locks were introduced), the mmap_prepare work is solving this generally.
vm_flags_init(vma, flags);
Analysing what's in this function:
VM_WARN_ON_ONCE(!pgtable_supports_soft_dirty() && (flags & VM_SOFTDIRTY));
Duplicated.
vma_flags_clear_all(&vma->flags);
Only necessary while you're only setting the first system word of vma->flags.
vma_flags_overwrite_word(&vma->flags, flags);
Again only necessary when you're only setting the first system word.
So yeah it's doing the equivalent and (intentionally) eliminating some noise.
But I'll add the S/D check back I guess.
Quoted 26 lines
Quoted 26 lines
mm/mlock.c +1
-1}else { -mlock_vma_pages_range (vma, start, end, newflags); +mlock_vma_pages_range (vma, start, end, &new_vma_flags); } out: *prev = vma;mm/mprotect.c +1
-3@@ -756,13 +756,11 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,vma_flags_clear (&new_vma_flags,VMA_ACCOUNT_BIT ); } - newflags =vma_flags_to_legacy (new_vma_flags); - vma =vma_modify_flags (vmi, *pprev, vma, start, end, &newflags); + vma =vma_modify_flags (vmi, *pprev, vma, start, end, &new_vma_flags);if (IS_ERR (vma)) { error =PTR_ERR (vma); goto fail; } - new_vma_flags =legacy_to_vma_flags (newflags); *pprev = vma;
Quoted 8 lines
mm/mprotect.c mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, · line 772
Quoted 6 lines
mm/mprotect.c +1
-1@@ -771,7 +769,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,* held in write mode. */ vma_start_write (vma); -vm_flags_reset_once (vma, newflags); +WRITE_ONCE (vma->flags, new_vma_flags);Ditto.
I mean overall these cases are hacks in my opinion to work around code that should have solved their problem another way.
But sure, as above I'll add a helper function or such.
Quoted 14 lines
Quoted 14 lines
mm/mprotect.c +1
-0if (vma_wants_manual_pte_write_upgrade (vma)) mm_cp_flags |=MM_CP_TRY_CHANGE_WRITABLE ;vma_set_page_prot (vma);@@ -796,6 +794,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, }vm_stat_account (mm,vma_flags_to_legacy (old_vma_flags), -nrpages); + newflags =vma_flags_to_legacy (new_vma_flags);vm_stat_account (mm, newflags, nrpages);perf_event_mmap (vma);return 0 ;mm/mseal.c +0
-0
Vlastimil Babka (SUSE)
reviewerRe: [PATCH v3 22/23] mm/vma: convert vma_modify_flags[_uffd]() to use vma_flags_t
Quoted 46 lines · Lorenzo Stoakes (Oracle)
Quoted 36 lines · Vlastimil Babka (SUSE)
Quoted 35 lines · Lorenzo Stoakes (Oracle)
Update the
vma_modify_flags()andvma_modify_flags_uffd()functions to accept avma_flags_tparameter rather than avm_flags_tone, and propagate the changes as needed to implement this change.Finally, update the VMA tests to reflect this.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>mm/mlock.c +5
-4@@ -415,13 +415,14 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr,* @vma - vma containing range to be mlock()ed or munlock()ed * @start - start address in @vma of the range * @end - end of range in @vma -* @newflags - the new set of flags for @vma. +* @new_vma_flags - the new set of flags for @vma. * * Called for mlock(), mlock2() and mlockall(), to set @vma VM_LOCKED; * called for munlock() and munlockall(), to clear VM_LOCKED from @vma. */ static void mlock_vma_pages_range (struct vm_area_struct *vma, -unsigned long start,unsigned long end,vm_flags_t newflags) +unsigned long start,unsigned long end, +vma_flags_t *new_vma_flags) {static const struct mm_walk_ops mlock_walk_ops = { .pmd_entry = mlock_pte_range,@@ -439,18 +440,18 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma,* combination should not be visible to other mmap_lock users; * but WRITE_ONCE so rmap walkers must see VM_IO if VM_LOCKED. */ -if (newflags &VM_LOCKED ) - newflags |=VM_IO ; +if (vma_flags_test (new_vma_flags,VMA_LOCKED_BIT )) +vma_flags_set (new_vma_flags,VMA_IO_BIT );Quoted 8 lines
mm/mlock.c static void mlock_vma_pages_range(struct vm_area_struct *vma, · line 446
Quoted 3 lines
mm/mlock.c +1
-1vma_start_write (vma); -vm_flags_reset_once (vma, newflags); +WRITE_ONCE (vma->flags, *new_vma_flags);It's not clear to me, how is switching from
vm_flags_ttovma_flags_tallowing us to simply doWRITE_ONCE()instead of the full logic ofvm_flags_reset_once()? Won't it fail to compile once once flags are more than single word? Or worse, will compile but silently allow tearing?
Quoted 8 lines
We only care about tearing in the flags that can be contained within a system word, but true we should probably do this more carefully, as I did for
vm_flags_reset_once().I will reimplement this as a new, hideous, helper function.
I am not a fan of this being a thing to handle races, it's a hack. But I guess that should be addressed separately.
Right, thanks!
Quoted 57 lines
Quoted 43 lines
Quoted 43 lines
mm/mlock.c +8
-6} }@@ -467,20 +468,22 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, struct vm_area_struct **prev, unsigned long start, unsigned long end, vm_flags_t newflags) { + vma_flags_t new_vma_flags = legacy_to_vma_flags(newflags); +const vma_flags_t old_vma_flags = vma->flags;struct mm_struct *mm = vma->vm_mm;int nr_pages;int ret =0 ; -vm_flags_t oldflags = vma->vm_flags; -if (newflags == oldflags ||vma_is_secretmem (vma) || - !vma_supports_mlock (vma)) +if (vma_flags_same_pair (&old_vma_flags, &new_vma_flags) || +vma_is_secretmem (vma) || !vma_supports_mlock (vma)) {/* * Don't set VM_LOCKED or VM_LOCKONFAULT and don't count. * For secretmem, don't allow the memory to be unlocked. */ goto out; + } - vma =vma_modify_flags (vmi, *prev, vma, start, end, &newflags); + vma =vma_modify_flags (vmi, *prev, vma, start, end, &new_vma_flags);if (IS_ERR (vma)) { ret =PTR_ERR (vma); goto out;@@ -490,9 +493,9 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma,* Keep track of amount of locked VM. */ nr_pages = (end - start) >>PAGE_SHIFT ; -if (!(newflags &VM_LOCKED )) +if (!vma_flags_test (&new_vma_flags,VMA_LOCKED_BIT )) nr_pages = -nr_pages; -else if (oldflags &VM_LOCKED ) +else if (vma_flags_test (&old_vma_flags,VMA_LOCKED_BIT )) nr_pages =0 ; mm->locked_vm += nr_pages;Quoted 13 lines
mm/mlock.c static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 511
Quoted 11 lines
mm/mlock.c +3
-2@@ -501,12 +504,13 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma,* It's okay if try_to_unmap_one unmaps a page just after we * set VM_LOCKED, populate_vma_page_range will bring it back. */ -if ((newflags &VM_LOCKED ) && (oldflags &VM_LOCKED )) { +if (vma_flags_test (&new_vma_flags,VMA_LOCKED_BIT ) && +vma_flags_test (&old_vma_flags,VMA_LOCKED_BIT )) {/* No work to do, and mlocking twice would be wrong */ vma_start_write (vma); -vm_flags_reset (vma, newflags); + vma->flags = new_vma_flags;This also does lot less than
vm_flags_reset()?
Quoted 7 lines
Well let's look:
VM_WARN_ON_ONCE(!pgtable_supports_soft_dirty() && (flags & VM_SOFTDIRTY));Are we really at a point where this is problematic? Do we hit this? Why are we specifically checking only this case on every single instance of resetting VMA flags?
I'll admit I don't know, but sounds like we can stop being so paranoid when converting code to the new API.
Quoted 6 lines
vma_assert_write_locked(vma);Note the
vma_start_write()line above. I want to separatevma_flags_thelpers from asserts like that, because:
Quoted 9 lines
1. We might be operating on a VMA that is not yet added to the tree
2. We might be operating on a VMA that is now detached
3. Really in all but core code, you should be usingvma_desc_xxx().
4. Other VMA fields are manipulated with no such checks.
5. It'd be egregious to have to add variants of flag functions just to account for cases such as the above, especially when we don't do so for other VMA fields. Drivers are the problematic cases and why it was especially important (and also for debug as VMA locks were introduced), themmap_preparework is solving this generally.
Perfectly reasonable!
Quoted 13 lines
vm_flags_init(vma, flags);Analysing what's in this function:
VM_WARN_ON_ONCE(!pgtable_supports_soft_dirty() && (flags & VM_SOFTDIRTY));Duplicated.
vma_flags_clear_all(&vma->flags);Only necessary while you're only setting the first system word of
vma->flags.
Quoted 6 lines
vma_flags_overwrite_word(&vma->flags, flags);Again only necessary when you're only setting the first system word.
So yeah it's doing the equivalent and (intentionally) eliminating some noise.
Ack.
Quoted 1 line
But I'll add the S/D check back I guess.
Thanks for the detailed explanation. It's fine to drop legacy stuff, it just wasn't obvious if intentional or by mistake. Maybe just mention the intention in the changelog?
Thanks!
Quoted 35 lines
Quoted 26 lines
Quoted 26 lines
mm/mlock.c +1
-1}else { -mlock_vma_pages_range (vma, start, end, newflags); +mlock_vma_pages_range (vma, start, end, &new_vma_flags); } out: *prev = vma;mm/mprotect.c +1
-3@@ -756,13 +756,11 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,vma_flags_clear (&new_vma_flags,VMA_ACCOUNT_BIT ); } - newflags =vma_flags_to_legacy (new_vma_flags); - vma =vma_modify_flags (vmi, *pprev, vma, start, end, &newflags); + vma =vma_modify_flags (vmi, *pprev, vma, start, end, &new_vma_flags);if (IS_ERR (vma)) { error =PTR_ERR (vma); goto fail; } - new_vma_flags =legacy_to_vma_flags (newflags); *pprev = vma;Quoted 8 lines
mm/mprotect.c mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, · line 772
Quoted 6 lines
mm/mprotect.c +1
-1@@ -771,7 +769,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,* held in write mode. */ vma_start_write (vma); -vm_flags_reset_once (vma, newflags); +WRITE_ONCE (vma->flags, new_vma_flags);Ditto.
Quoted 4 lines
I mean overall these cases are hacks in my opinion to work around code that should have solved their problem another way.
But sure, as above I'll add a helper function or such.
Great.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
mm/mprotect.c +1
-0if (vma_wants_manual_pte_write_upgrade (vma)) mm_cp_flags |=MM_CP_TRY_CHANGE_WRITABLE ;vma_set_page_prot (vma);@@ -796,6 +794,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, }vm_stat_account (mm,vma_flags_to_legacy (old_vma_flags), -nrpages); + newflags =vma_flags_to_legacy (new_vma_flags);vm_stat_account (mm, newflags, nrpages);perf_event_mmap (vma);return 0 ;mm/mseal.c +0
-0
Lorenzo Stoakes (Oracle)
maintainerRe: [PATCH v3 22/23] mm/vma: convert vma_modify_flags[_uffd]() to use vma_flags_t
Quoted 51 lines · Vlastimil Babka (SUSE)
Quoted 50 lines · Lorenzo Stoakes (Oracle)
Quoted 36 lines · Vlastimil Babka (SUSE)
Quoted 35 lines · Lorenzo Stoakes (Oracle)
Update the
vma_modify_flags()andvma_modify_flags_uffd()functions to accept avma_flags_tparameter rather than avm_flags_tone, and propagate the changes as needed to implement this change.Finally, update the VMA tests to reflect this.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>mm/mlock.c +5
-4@@ -415,13 +415,14 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr,* @vma - vma containing range to be mlock()ed or munlock()ed * @start - start address in @vma of the range * @end - end of range in @vma -* @newflags - the new set of flags for @vma. +* @new_vma_flags - the new set of flags for @vma. * * Called for mlock(), mlock2() and mlockall(), to set @vma VM_LOCKED; * called for munlock() and munlockall(), to clear VM_LOCKED from @vma. */ static void mlock_vma_pages_range (struct vm_area_struct *vma, -unsigned long start,unsigned long end,vm_flags_t newflags) +unsigned long start,unsigned long end, +vma_flags_t *new_vma_flags) {static const struct mm_walk_ops mlock_walk_ops = { .pmd_entry = mlock_pte_range,@@ -439,18 +440,18 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma,* combination should not be visible to other mmap_lock users; * but WRITE_ONCE so rmap walkers must see VM_IO if VM_LOCKED. */ -if (newflags &VM_LOCKED ) - newflags |=VM_IO ; +if (vma_flags_test (new_vma_flags,VMA_LOCKED_BIT )) +vma_flags_set (new_vma_flags,VMA_IO_BIT );Quoted 8 lines
mm/mlock.c static void mlock_vma_pages_range(struct vm_area_struct *vma, · line 446
Quoted 3 lines
mm/mlock.c +1
-1vma_start_write (vma); -vm_flags_reset_once (vma, newflags); +WRITE_ONCE (vma->flags, *new_vma_flags);It's not clear to me, how is switching from
vm_flags_ttovma_flags_tallowing us to simply doWRITE_ONCE()instead of the full logic ofvm_flags_reset_once()? Won't it fail to compile once once flags are more than single word? Or worse, will compile but silently allow tearing?We only care about tearing in the flags that can be contained within a system word, but true we should probably do this more carefully, as I did for
vm_flags_reset_once().
Quoted 6 lines
Quoted 4 lines
I will reimplement this as a new, hideous, helper function.
I am not a fan of this being a thing to handle races, it's a hack. But I guess that should be addressed separately.
Right, thanks!
No problem. I might be able to avoid it being too hideous let's see :)
Quoted 60 lines
Quoted 60 lines
Quoted 44 lines
Quoted 44 lines
mm/mlock.c +8
-6} }@@ -467,20 +468,22 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, struct vm_area_struct **prev, unsigned long start, unsigned long end, vm_flags_t newflags) { + vma_flags_t new_vma_flags = legacy_to_vma_flags(newflags); +const vma_flags_t old_vma_flags = vma->flags;struct mm_struct *mm = vma->vm_mm;int nr_pages;int ret =0 ; -vm_flags_t oldflags = vma->vm_flags; -if (newflags == oldflags ||vma_is_secretmem (vma) || - !vma_supports_mlock (vma)) +if (vma_flags_same_pair (&old_vma_flags, &new_vma_flags) || +vma_is_secretmem (vma) || !vma_supports_mlock (vma)) {/* * Don't set VM_LOCKED or VM_LOCKONFAULT and don't count. * For secretmem, don't allow the memory to be unlocked. */ goto out; + } - vma =vma_modify_flags (vmi, *prev, vma, start, end, &newflags); + vma =vma_modify_flags (vmi, *prev, vma, start, end, &new_vma_flags);if (IS_ERR (vma)) { ret =PTR_ERR (vma); goto out;@@ -490,9 +493,9 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma,* Keep track of amount of locked VM. */ nr_pages = (end - start) >>PAGE_SHIFT ; -if (!(newflags &VM_LOCKED )) +if (!vma_flags_test (&new_vma_flags,VMA_LOCKED_BIT )) nr_pages = -nr_pages; -else if (oldflags &VM_LOCKED ) +else if (vma_flags_test (&old_vma_flags,VMA_LOCKED_BIT )) nr_pages =0 ; mm->locked_vm += nr_pages;Quoted 13 lines
mm/mlock.c static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 511
Quoted 11 lines
mm/mlock.c +3
-2@@ -501,12 +504,13 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma,* It's okay if try_to_unmap_one unmaps a page just after we * set VM_LOCKED, populate_vma_page_range will bring it back. */ -if ((newflags &VM_LOCKED ) && (oldflags &VM_LOCKED )) { +if (vma_flags_test (&new_vma_flags,VMA_LOCKED_BIT ) && +vma_flags_test (&old_vma_flags,VMA_LOCKED_BIT )) {/* No work to do, and mlocking twice would be wrong */ vma_start_write (vma); -vm_flags_reset (vma, newflags); + vma->flags = new_vma_flags;This also does lot less than
vm_flags_reset()?Well let's look:
Quoted 8 lines
Quoted 5 lines
VM_WARN_ON_ONCE(!pgtable_supports_soft_dirty() && (flags & VM_SOFTDIRTY));Are we really at a point where this is problematic? Do we hit this? Why are we specifically checking only this case on every single instance of resetting VMA flags?
I'll admit I don't know, but sounds like we can stop being so paranoid when converting code to the new API.
Yeah I think so. I think Suren is ok with it, probably :P
Quoted 39 lines
Quoted 7 lines
vma_assert_write_locked(vma);Note the
vma_start_write()line above. I want to separatevma_flags_thelpers from asserts like that, because:Quoted 9 lines
1. We might be operating on a VMA that is not yet added to the tree
2. We might be operating on a VMA that is now detached
3. Really in all but core code, you should be usingvma_desc_xxx().
4. Other VMA fields are manipulated with no such checks.
5. It'd be egregious to have to add variants of flag functions just to account for cases such as the above, especially when we don't do so for other VMA fields. Drivers are the problematic cases and why it was especially important (and also for debug as VMA locks were introduced), themmap_preparework is solving this generally.Perfectly reasonable!
Quoted 13 lines
vm_flags_init(vma, flags);Analysing what's in this function:
VM_WARN_ON_ONCE(!pgtable_supports_soft_dirty() && (flags & VM_SOFTDIRTY));Duplicated.
vma_flags_clear_all(&vma->flags);Only necessary while you're only setting the first system word of
vma->flags.Quoted 6 lines
vma_flags_overwrite_word(&vma->flags, flags);Again only necessary when you're only setting the first system word.
So yeah it's doing the equivalent and (intentionally) eliminating some noise.
Quoted 7 lines
Ack.
Quoted 1 line
But I'll add the S/D check back I guess.
Thanks for the detailed explanation. It's fine to drop legacy stuff, it just wasn't obvious if intentional or by mistake. Maybe just mention the intention in the changelog?
Yeah that's a fair point will do!
Quoted 52 lines
Thanks!
Quoted 35 lines
Quoted 26 lines
Quoted 26 lines
mm/mlock.c +1
-1}else { -mlock_vma_pages_range (vma, start, end, newflags); +mlock_vma_pages_range (vma, start, end, &new_vma_flags); } out: *prev = vma;mm/mprotect.c +1
-3@@ -756,13 +756,11 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,vma_flags_clear (&new_vma_flags,VMA_ACCOUNT_BIT ); } - newflags =vma_flags_to_legacy (new_vma_flags); - vma =vma_modify_flags (vmi, *pprev, vma, start, end, &newflags); + vma =vma_modify_flags (vmi, *pprev, vma, start, end, &new_vma_flags);if (IS_ERR (vma)) { error =PTR_ERR (vma); goto fail; } - new_vma_flags =legacy_to_vma_flags (newflags); *pprev = vma;Quoted 8 lines
mm/mprotect.c mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, · line 772
Quoted 6 lines
mm/mprotect.c +1
-1@@ -771,7 +769,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,* held in write mode. */ vma_start_write (vma); -vm_flags_reset_once (vma, newflags); +WRITE_ONCE (vma->flags, new_vma_flags);Ditto.
Quoted 4 lines
I mean overall these cases are hacks in my opinion to work around code that should have solved their problem another way.
But sure, as above I'll add a helper function or such.
Great.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
mm/mprotect.c +0
-0if (vma_wants_manual_pte_write_upgrade (vma)) mm_cp_flags |=MM_CP_TRY_CHANGE_WRITABLE ;vma_set_page_prot (vma);@@ -796,6 +794,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, }
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
mm/mseal.c
Quoted 7 lines
mm/mprotect.c +1
-0vm_stat_account (mm,vma_flags_to_legacy (old_vma_flags), -nrpages); + newflags =vma_flags_to_legacy (new_vma_flags);vm_stat_account (mm, newflags, nrpages);perf_event_mmap (vma);return 0 ;mm/mseal.c +0
-0
Cheers, Lorenzo
Lorenzo Stoakes (Oracle)
maintainerPATCH v3 23/23 mm/vma: convert __mmap_region() to use vma_flags_t
In reply to Lorenzo Stoakes (Oracle)Update the mmap() implementation logic implemented in __mmap_region() and functions invoked by it. The mmap_region() function converts its input vm_flags_t parameter to a vma_flags_t value which it then passes to __mmap_region() which uses the vma_flags_t value consistently from then on.
As part of the change, we convert map_deny_write_exec() to using vma_flags_t (it was incorrectly using unsigned long before), and place it in vma.h, as it is only used internal to mm.
With this change, we eliminate the legacy is_shared_maywrite_vm_flags() helper function which is now no longer required.
We are also able to update the MMAP_STATE() and VMG_MMAP_STATE() macros to use the vma_flags_t value.
Finally, we update the VMA tests to reflect the change.
--- include/linux/mm.h | 18 ++++++++---- include/linux/mman.h | 49 ------------------------------- mm/mprotect.c | 4 ++- mm/vma.c | 25 ++++++++-------- mm/vma.h | 51 +++++++++++++++++++++++++++++++++ tools/testing/vma/include/dup.h | 34 +++++----------------- tools/testing/vma/tests/mmap.c | 18 ++++-------- 7 files changed, 92 insertions(+), 107 deletions(-)
include/linux/mm.h +12 -6
@@ -1529,12 +1529,6 @@ static inline bool vma_is_accessible(const struct vm_area_struct *vma)return vma->vm_flags &VM_ACCESS_FLAGS ; } -static inline bool is_shared_maywrite_vm_flags (vm_flags_t vm_flags) -{ -return (vm_flags & (VM_SHARED |VM_MAYWRITE )) == - (VM_SHARED |VM_MAYWRITE ); -} -static inline bool is_shared_maywrite (const vma_flags_t *flags) {return vma_flags_test_all (flags,VMA_SHARED_BIT ,VMA_MAYWRITE_BIT );
@@ -4351,12 +4345,24 @@ static inline bool range_in_vma(const struct vm_area_struct *vma,#ifdef CONFIG_MMU pgprot_t vm_get_page_prot (vm_flags_t vm_flags); + +static inline pgprot_t vma_get_page_prot (vma_flags_t vma_flags) +{ +const vm_flags_t vm_flags =vma_flags_to_legacy (vma_flags); + +return vm_get_page_prot (vm_flags); +} +void vma_set_page_prot (struct vm_area_struct *vma);#else static inline pgprot_t vm_get_page_prot (vm_flags_t vm_flags) {return __pgprot (0 ); } +static inline pgprot_t vma_get_page_prot (vma_flags_t vma_flags) +{ +return __pgprot (0 ); +}static inline void vma_set_page_prot (struct vm_area_struct *vma) { vma->vm_page_prot =vm_get_page_prot (vma->vm_flags);
include/linux/mman.h +0 -49
53 lines not on this page · raw message
mm/mprotect.c +3 -1
@@ -882,6 +882,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len, tmp = vma->vm_start;for_each_vma_range (vmi, vma, end) {vm_flags_t mask_off_old_flags; +vma_flags_t new_vma_flags;vm_flags_t newflags;int new_vma_pkey;
@@ -904,6 +905,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len, new_vma_pkey =arch_override_mprotect_pkey (vma, prot, pkey); newflags =calc_vm_prot_bits (prot, new_vma_pkey); newflags |= (vma->vm_flags & ~mask_off_old_flags); + new_vma_flags =legacy_to_vma_flags (newflags);/* newflags >> 4 shift VM_MAY% in place of VM_% */ if ((newflags & ~(newflags >>4 )) &VM_ACCESS_FLAGS ) {
@@ -911,7 +913,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len,break ; } -if (map_deny_write_exec (vma->vm_flags, newflags)) { +if (map_deny_write_exec (&vma->flags, &new_vma_flags)) { error = -EACCES ;break ; }
mm/vma.c +13 -12
@@ -44,7 +44,7 @@ struct mmap_state { bool file_doesnt_need_get :1; }; -#define MMAP_STATE (name, mm_, vmi_, addr_, len_, pgoff_, vm_flags_, file_) \ +#define MMAP_STATE (name, mm_, vmi_, addr_, len_, pgoff_, vma_flags_, file_) \ struct mmap_state name = { \ .mm = mm_, \ .vmi = vmi_, \
@@ -52,9 +52,9 @@ struct mmap_state { .end = (addr_) + (len_), \ .pgoff = pgoff_, \ .pglen = PHYS_PFN(len_), \ - .vm_flags = vm_flags_, \ + .vma_flags = vma_flags_, \ .file = file_, \ - .page_prot = vm_get_page_prot(vm_flags_), \ + .page_prot = vma_get_page_prot(vma_flags_), \ }#define VMG_MMAP_STATE (name, map_, vma_) \
@@ -63,7 +63,7 @@ struct mmap_state { .vmi = (map_)->vmi, \ .start = (map_)->addr, \ .end = (map_)->end, \ - .vm_flags = (map_)->vm_flags, \ + .vma_flags = (map_)->vma_flags, \ .pgoff = (map_)->pgoff, \ .file = (map_)->file, \ .prev = (map_)->prev, \
@@ -2746,14 +2746,14 @@ static int call_action_complete(struct mmap_state *map, }static unsigned long __mmap_region (struct file *file,unsigned long addr, -unsigned long len,vm_flags_t vm_flags,unsigned long pgoff, -struct list_head *uf) +unsigned long len,vma_flags_t vma_flags, +unsigned long pgoff,struct list_head *uf) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma =NULL ;bool have_mmap_prepare = file && file->f_op->mmap_prepare;VMA_ITERATOR (vmi, mm, addr); -MMAP_STATE (map, mm, &vmi, addr, len, pgoff, vm_flags, file); +MMAP_STATE (map, mm, &vmi, addr, len, pgoff, vma_flags, file);struct vm_area_desc desc = { .mm = mm, .file = file,
@@ -2837,16 +2837,17 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr,* been performed. */ unsigned long mmap_region (struct file *file,unsigned long addr, -unsigned long len,vm_flags_t vm_flags,unsigned long pgoff, -struct list_head *uf) +unsigned long len,vm_flags_t vm_flags, +unsigned long pgoff,struct list_head *uf) {unsigned long ret;bool writable_file_mapping = false; +const vma_flags_t vma_flags =legacy_to_vma_flags (vm_flags);mmap_assert_write_locked (current->mm);/* Check to see if MDWE is applicable. */ -if (map_deny_write_exec (vm_flags, vm_flags)) +if (map_deny_write_exec (&vma_flags, &vma_flags))return -EACCES ;/* Allow architectures to sanity-check the vm_flags. */
@@ -2854,7 +2855,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,return -EINVAL ;/* Map writable and ensure this isn't a sealed memfd. */ -if (file &&is_shared_maywrite_vm_flags (vm_flags)) { +if (file &&is_shared_maywrite (&vma_flags)) {int error =mapping_map_writable (file->f_mapping);if (error)
@@ -2862,7 +2863,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, writable_file_mapping = true; } - ret =__mmap_region (file, addr, len, vm_flags, pgoff, uf); + ret =__mmap_region (file, addr, len, vma_flags, pgoff, uf);/* Clear our write mapping regardless of error. */ if (writable_file_mapping)
mm/vma.h +51 -0
55 lines not on this page · raw message
tools/testing/vma/include/dup.h +7 -27
@@ -1120,12 +1120,6 @@ static __always_inline void vma_desc_clear_flags_mask(struct vm_area_desc *desc,#define vma_desc_clear_flags (desc, ...) \ vma_desc_clear_flags_mask(desc, mk_vma_flags(__VA_ARGS__)) -static inline bool is_shared_maywrite_vm_flags (vm_flags_t vm_flags) -{ -return (vm_flags & (VM_SHARED |VM_MAYWRITE )) == - (VM_SHARED |VM_MAYWRITE ); -} -static inline bool is_shared_maywrite (const vma_flags_t *flags) {return vma_flags_test_all (flags,VMA_SHARED_BIT ,VMA_MAYWRITE_BIT );
@@ -1442,27 +1436,6 @@ static inline bool mlock_future_ok(const struct mm_struct *mm,return locked_pages <= limit_pages; } -static inline bool map_deny_write_exec (unsigned long old,unsigned long new) -{ -/* If MDWE is disabled, we have nothing to deny. */ -if (mm_flags_test (MMF_HAS_MDWE , current->mm)) -return false; - -/* If the new VMA is not executable, we have nothing to deny. */ -if (!(new &VM_EXEC )) -return false; - -/* Under MDWE we do not accept newly writably executable VMAs... */ -if (new &VM_WRITE ) -return true; - -/* ...nor previously non-executable VMAs becoming executable. */ -if (!(old &VM_EXEC )) -return true; - -return false; -} -static inline int mapping_map_writable (struct address_space *mapping) { return atomic_inc_unless_negative(&mapping->i_mmap_writable) ?
@@ -1514,3 +1487,10 @@ static inline int get_sysctl_max_map_count(void)#ifndef pgtable_supports_soft_dirty#define pgtable_supports_soft_dirty () IS_ENABLED(CONFIG_MEM_SOFT_DIRTY)#endif + +static inline pgprot_t vma_get_page_prot (vma_flags_t vma_flags) +{ +const vm_flags_t vm_flags =vma_flags_to_legacy (vma_flags); + +return vm_get_page_prot (vm_flags); +}
tools/testing/vma/tests/mmap.c +6 -12
@@ -2,6 +2,8 @@static bool test_mmap_region_basic (void ) { +const vma_flags_t vma_flags =mk_vma_flags (VMA_READ_BIT ,VMA_WRITE_BIT , +VMA_MAYREAD_BIT ,VMA_MAYWRITE_BIT );struct mm_struct mm = {};unsigned long addr;struct vm_area_struct *vma;
@@ -10,27 +12,19 @@ static bool test_mmap_region_basic(void) current->mm = &mm;/* Map at 0x300000, length 0x3000. */ - addr =__mmap_region (NULL ,0x300000 ,0x3000 , -VM_READ |VM_WRITE |VM_MAYREAD |VM_MAYWRITE , -0x300 ,NULL ); + addr =__mmap_region (NULL ,0x300000 ,0x3000 , vma_flags,0x300 ,NULL );ASSERT_EQ (addr,0x300000 );/* Map at 0x250000, length 0x3000. */ - addr =__mmap_region (NULL ,0x250000 ,0x3000 , -VM_READ |VM_WRITE |VM_MAYREAD |VM_MAYWRITE , -0x250 ,NULL ); + addr =__mmap_region (NULL ,0x250000 ,0x3000 , vma_flags,0x250 ,NULL );ASSERT_EQ (addr,0x250000 );/* Map at 0x303000, merging to 0x300000 of length 0x6000. */ - addr =__mmap_region (NULL ,0x303000 ,0x3000 , -VM_READ |VM_WRITE |VM_MAYREAD |VM_MAYWRITE , -0x303 ,NULL ); + addr =__mmap_region (NULL ,0x303000 ,0x3000 , vma_flags,0x303 ,NULL );ASSERT_EQ (addr,0x303000 );/* Map at 0x24d000, merging to 0x250000 of length 0x6000. */ - addr =__mmap_region (NULL ,0x24d000 ,0x3000 , -VM_READ |VM_WRITE |VM_MAYREAD |VM_MAYWRITE , -0x24d ,NULL ); + addr =__mmap_region (NULL ,0x24d000 ,0x3000 , vma_flags,0x24d ,NULL );ASSERT_EQ (addr,0x24d000 );ASSERT_EQ (mm.map_count,2 );
Signature · 1 line
2.53.0
Re: [PATCH v3 23/23] mm/vma: convert __mmap_region() to use vma_flags_t
Quoted 13 lines · Lorenzo Stoakes (Oracle)
Update the
mmap()implementation logic implemented in__mmap_region()and functions invoked by it. Themmap_region()function converts its inputvm_flags_tparameter to avma_flags_tvalue which it then passes to__mmap_region()which uses thevma_flags_tvalue consistently from then on.As part of the change, we convert
map_deny_write_exec()to usingvma_flags_t(it was incorrectly using unsigned long before), and place it invma.h, as it is only used internal to mm.With this change, we eliminate the legacy
is_shared_maywrite_vm_flags()helper function which is now no longer required.
Quoted 6 lines
We are also able to update the
MMAP_STATE()andVMG_MMAP_STATE()macros to use thevma_flags_tvalue.Finally, we update the VMA tests to reflect the change.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Andrew Morton
maintainerQuoted 2 lines · On Wed, 18 Mar 2026 15:50:11 +0000 "Lorenzo Stoakes (Oracle)" <[email protected]>…
This series converts a lot of the existing use of the legacy
vm_flags_tdata type to the newvma_flags_ttype which replaces it.
Updated, thanks. Below is how v3 altered mm.git:
arch/arm64/include/asm/page.h | 4 +++ include/linux/mm_types.h | 32 +++++++++++++++--------------- mm/mseal.c | 1 mm/vma.c | 27 +++++++++++++------------ tools/testing/vma/include/dup.h | 22 +++++++++++--------- tools/testing/vma/tests/vma.c | 3 -- 6 files changed, 48 insertions(+), 41 deletions(-)
arch/arm64/include/asm/page.h renamed from arch/arm64/include/asm/page.h~b +4 -0
@@ -46,8 +46,12 @@ int pfn_is_map_memory(unsigned long pfn)#endif /* !__ASSEMBLER__ */ +#ifdef CONFIG_ARM64_MTE #define VMA_DATA_DEFAULT_FLAGS append_vma_flags(VMA_DATA_FLAGS_TSK_EXEC, \ VMA_MTE_ALLOWED_BIT) +#else +#define VMA_DATA_DEFAULT_FLAGS VMA_DATA_FLAGS_TSK_EXEC +#endif #include <asm-generic/getorder.h>
include/linux/mm_types.h renamed from include/linux/mm_types.h~b +16 -16
@@ -871,9 +871,9 @@ typedef struct {#define EMPTY_VMA_FLAGS ((vma_flags_t){ })/* Are no flags set in the specified VMA flags? */ -static __always_inline bool vma_flags_empty(vma_flags_t *flags) +static __always_inline bool vma_flags_empty(const vma_flags_t *flags) { -unsigned long *bitmap = flags->__vma_flags; +const unsigned long *bitmap = flags->__vma_flags;return bitmap_empty (bitmap,NUM_VMA_FLAG_BITS ); }
@@ -1082,20 +1082,6 @@ static __always_inline vm_flags_t vma_fl }/* -* Helper function which converts a legacy vm_flags_t value to a vma_flags_t -* value. -* -* Will be removed once the conversion to VMA flags is complete. -*/ -static __always_inline vma_flags_t legacy_to_vma_flags(vm_flags_t flags) -{ -vma_flags_t ret; - - ret.__vma_flags[0 ] = (unsigned long )flags; -return ret; -} - -/* * Copy value to the first system word of VMA flags, non-atomically. * * IMPORTANT: This does not overwrite bytes past the first system word. The
@@ -1110,6 +1096,20 @@ static __always_inline void vma_flags_ov }/* +* Helper function which converts a legacy vm_flags_t value to a vma_flags_t +* value. +* +* Will be removed once the conversion to VMA flags is complete. +*/ +static __always_inline vma_flags_t legacy_to_vma_flags(vm_flags_t flags) +{ +vma_flags_t ret =EMPTY_VMA_FLAGS ; + +vma_flags_overwrite_word (&ret, flags); +return ret; +} + +/* * Copy value to the first system word of VMA flags ONCE, non-atomically. * * IMPORTANT: This does not overwrite bytes past the first system word. The
mm/mseal.c renamed from mm/mseal.c~b +1 -0
@@ -77,6 +77,7 @@ static int mseal_apply(struct mm_struct curr_end, &vma_flags);if (IS_ERR (vma))return PTR_ERR (vma); +vma_start_write (vma);vma_set_flags (vma,VMA_SEALED_BIT ); }
mm/vma.c renamed from mm/vma.c~b +14 -13
@@ -905,20 +905,21 @@ static __must_check struct vm_area_strucvma_start_write (middle);if (merge_right) { -const vma_flags_t next_sticky = -vma_flags_and_mask (&next->flags,VMA_STICKY_FLAGS ); +vma_flags_t next_sticky;vma_start_write (next); vmg->target = next; + next_sticky =vma_flags_and_mask (&next->flags,VMA_STICKY_FLAGS );vma_flags_set_mask (&sticky_flags, next_sticky); }if (merge_left) { -const vma_flags_t prev_sticky = -vma_flags_and_mask (&prev->flags,VMA_STICKY_FLAGS ); +vma_flags_t prev_sticky;vma_start_write (prev); vmg->target = prev; + + prev_sticky =vma_flags_and_mask (&prev->flags,VMA_STICKY_FLAGS );vma_flags_set_mask (&sticky_flags, prev_sticky); }
@@ -1170,13 +1171,14 @@ int vma_expand(struct vma_merge_struct *bool remove_next = false;vma_flags_t sticky_flags =vma_flags_and_mask (&vmg->vma_flags,VMA_STICKY_FLAGS ); -const vma_flags_t target_sticky = -vma_flags_and_mask (&target->flags,VMA_STICKY_FLAGS ); +vma_flags_t target_sticky;int ret =0 ;mmap_assert_write_locked (vmg->mm);vma_start_write (target); + target_sticky =vma_flags_and_mask (&target->flags,VMA_STICKY_FLAGS ); +if (next && target != next && vmg->end == next->vm_end) remove_next = true;
@@ -1192,12 +1194,6 @@ int vma_expand(struct vma_merge_struct * target->vm_end > vmg->end, vmg);vma_flags_set_mask (&sticky_flags, target_sticky); -if (remove_next) { -const vma_flags_t next_sticky = -vma_flags_and_mask (&next->flags,VMA_STICKY_FLAGS ); - -vma_flags_set_mask (&sticky_flags, next_sticky); - }/* * If we are removing the next VMA or copying from a VMA
@@ -1214,8 +1210,13 @@ int vma_expand(struct vma_merge_struct *return ret;if (remove_next) { +vma_flags_t next_sticky; +vma_start_write (next); vmg->__remove_next = true; + + next_sticky =vma_flags_and_mask (&next->flags,VMA_STICKY_FLAGS ); +vma_flags_set_mask (&sticky_flags, next_sticky); }if (commit_merge (vmg)) goto nomem;
@@ -2950,7 +2951,7 @@ out:if (vma_flags_test (&vma_flags,VMA_LOCKED_BIT )) mm->locked_vm += (len >>PAGE_SHIFT );if (pgtable_supports_soft_dirty ()) -vma_flags_set (&vma_flags,VMA_SOFTDIRTY_BIT ); +vma_set_flags (vma,VMA_SOFTDIRTY_BIT );return 0 ; mas_store_fail:
tools/testing/vma/include/dup.h renamed from tools/testing/vma/include/dup.h~b +12 -10
@@ -441,9 +441,9 @@ struct vma_iterator {#define MAPCOUNT_ELF_CORE_MARGIN (5)#define DEFAULT_MAX_MAP_COUNT (USHRT_MAX - MAPCOUNT_ELF_CORE_MARGIN) -static __always_inline bool vma_flags_empty(vma_flags_t *flags) +static __always_inline bool vma_flags_empty(const vma_flags_t *flags) { -unsigned long *bitmap = flags->__vma_flags; +const unsigned long *bitmap = flags->__vma_flags;return bitmap_empty (bitmap,NUM_VMA_FLAG_BITS ); }
@@ -775,7 +775,9 @@ static inline bool mm_flags_test(int fla static __always_inline void vma_flags_overwrite_word(vma_flags_t *flags,unsigned long value) { - *ACCESS_PRIVATE (flags, __vma_flags) = value; +unsigned long *bitmap = flags->__vma_flags; + + bitmap[0 ] = value; }/*
@@ -787,7 +789,7 @@ static __always_inline void vma_flags_ov static __always_inline void vma_flags_overwrite_word_once(vma_flags_t *flags,unsigned long value) { -unsigned long *bitmap =ACCESS_PRIVATE (flags, __vma_flags); +unsigned long *bitmap = flags->__vma_flags;WRITE_ONCE (*bitmap, value); }
@@ -796,7 +798,7 @@ static __always_inline void vma_flags_ov static __always_inline void vma_flags_set_word(vma_flags_t *flags,unsigned long value) { -unsigned long *bitmap =ACCESS_PRIVATE (flags, __vma_flags); +unsigned long *bitmap = flags->__vma_flags; *bitmap |= value; }
@@ -805,7 +807,7 @@ static __always_inline void vma_flags_se static __always_inline void vma_flags_clear_word(vma_flags_t *flags,unsigned long value) { -unsigned long *bitmap =ACCESS_PRIVATE (flags, __vma_flags); +unsigned long *bitmap = flags->__vma_flags; *bitmap &= ~value; }
@@ -835,9 +837,9 @@ static __always_inline vm_flags_t vma_fl*/ static __always_inline vma_flags_t legacy_to_vma_flags(vm_flags_t flags) { -vma_flags_t ret; +vma_flags_t ret =EMPTY_VMA_FLAGS ; - ret.__vma_flags[0 ] = (unsigned long )flags; +vma_flags_overwrite_word (&ret, flags);return ret; }
@@ -1073,8 +1075,8 @@ static __always_inline void vma_clear_flvma_flags_clear_mask (&vma->flags, flags); } -#define vma_clear_flags (vmag, ...) \ - vma_clear_flags_mask(vmag, mk_vma_flags(__VA_ARGS__)) +#define vma_clear_flags (vma, ...) \ + vma_clear_flags_mask(vma, mk_vma_flags(__VA_ARGS__)) static __always_inline bool vma_desc_test(const struct vm_area_desc *desc,vma_flag_t bit)
tools/testing/vma/tests/vma.c renamed from tools/testing/vma/tests/vma.c~b +1 -2
@@ -356,10 +356,9 @@ static bool test_vma_flags_clear(void)/* Cursory check of _mask() variant, as the helper macros imply. */ vma_flags_clear_mask (&flags, mask); -vma_flags_clear_mask (&vma.flags, mask); +vma_clear_flags_mask (&vma, mask);vma_desc_clear_flags_mask (&desc, mask);#if NUM_VMA_FLAG_BITS >64 -vma_clear_flags_mask (&vma, mask);ASSERT_FALSE (vma_flags_test_any (&flags,VMA_EXEC_BIT ,64 ));ASSERT_FALSE (vma_test_any (&vma,VMA_EXEC_BIT ,64 ));ASSERT_FALSE (vma_desc_test_any (&desc,VMA_EXEC_BIT ,64 ));
_