Create 'old' ptes for faultaround mappings on arm64 with hardware access flag
Subsystems (90)
Will Deacon
maintainerv3 cover Create 'old' ptes for faultaround mappings on arm64 with hardware access flag
Hi again folks,
This is the third version of the patches I previously posted here:
v1: https://lore.kernel.org/r/[email protected]
v2: https://lore.kernel.org/r/[email protected]
The patches allow architectures to opt-in at runtime for faultaround mappings to be created as 'old' instead of 'young'. Although there have been previous attempts at this, they failed either because the decision was deferred to userspace [1] or because it was done unconditionally and shown to regress benchmarks for particular architectures [2].
Minor changes since v2 include:
* Update commit messages
* Remove repeated word 'from from' in a comment
* Restore 'vmf->flags' in filemap_map_pages()
The major additions are in the five RFC patches at the end of the series, which attempt to implement a suggestion from Linus to split up 'struct vm_fault', clearly separating the mutable and immutable fields in the data structure. I used Coccinelle to do most of the mechanical work, but I also ran into some tricky problems along the way:
1. 'vmf->flags' is modified on the '->page_mkwrite()' path so I couldn't find a satisfactory way to move it to the new const structure. I toyed with getting rid of FAULT_FLAG_[MK]WRITE completely and just tracking these as bools, but there's also a weird piece of code in vmw_bo_vm_mkwrite() which modifies FAULT_FLAG_ALLOW_RETRY, so I gave up and left the 'flags' field alone.
2. I had to perform terrifying surgery on __collapse_huge_page_swapin() and, in doing so, I'm a bit wary about the initialisation of 'pgoff', as it isn't updated along with the address (this matches the old code).
3. vmf_insert_pfn_pmd() and friends take both a 'struct vm_fault' _and_ a 'bool write'. I have left them alone, but that FAULT_FLAG_WRITE is causing trouble again.
4. Turns out 'struct vm_fault' is popular, so the diffstat is bloody massive.
Anyway, be good to hear any thoughts on this lot, particular with regards to my comments above. I've also pushed the series here:
https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=faultaround
Cheers
Will
[1] https://www.spinics.net/lists/linux-mm/msg143831.html
[2] 315d09bf30c2 ("Revert "mm: make faultaround produce old ptes"")
--->8
Kirill A. Shutemov (1):
mm: Cleanup faultaround and finish_fault() codepaths
Will Deacon (7):
mm: Allow architectures to request 'old' entries when prefaulting
arm64: mm: Implement arch_wants_old_prefaulted_pte()
mm: Separate fault info out of 'struct vm_fault'
mm: Pass 'address' to map to do_set_pte() and drop FAULT_FLAG_PREFAULT
mm: Avoid modifying vmf.info.address in __collapse_huge_page_swapin()
mm: Use static initialisers for 'info' field of 'struct vm_fault'
mm: Mark 'info' field of 'struct vm_fault' as 'const'
arch/arm64/include/asm/pgtable.h | 12 +- arch/arm64/kernel/vdso.c | 4 +- arch/powerpc/kvm/book3s_64_vio.c | 6 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 4 +- arch/powerpc/kvm/book3s_xive_native.c | 13 +- arch/powerpc/platforms/cell/spufs/file.c | 16 +- arch/s390/kernel/vdso.c | 4 +- arch/s390/kvm/kvm-s390.c | 2 +- arch/x86/entry/vdso/vma.c | 22 +- arch/x86/kernel/cpu/sgx/encl.c | 4 +- drivers/char/agp/alpha-agp.c | 2 +- drivers/char/mspec.c | 6 +- drivers/dax/device.c | 37 +- drivers/dma-buf/heaps/cma_heap.c | 6 +- drivers/dma-buf/udmabuf.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +- drivers/gpu/drm/armada/armada_gem.c | 6 +- drivers/gpu/drm/drm_gem_shmem_helper.c | 8 +- drivers/gpu/drm/drm_vm.c | 18 +- drivers/gpu/drm/etnaviv/etnaviv_gem.c | 10 +- drivers/gpu/drm/gma500/framebuffer.c | 4 +- drivers/gpu/drm/gma500/gem.c | 8 +- drivers/gpu/drm/i915/gem/i915_gem_mman.c | 8 +- drivers/gpu/drm/msm/msm_gem.c | 11 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 8 +- drivers/gpu/drm/nouveau/nouveau_ttm.c | 2 +- drivers/gpu/drm/omapdrm/omap_gem.c | 20 +- drivers/gpu/drm/radeon/radeon_ttm.c | 4 +- drivers/gpu/drm/tegra/gem.c | 6 +- drivers/gpu/drm/ttm/ttm_bo_vm.c | 10 +- drivers/gpu/drm/vc4/vc4_bo.c | 2 +- drivers/gpu/drm/vgem/vgem_drv.c | 6 +- drivers/gpu/drm/vmwgfx/vmwgfx_page_dirty.c | 12 +- drivers/hsi/clients/cmt_speech.c | 2 +- drivers/hwtracing/intel_th/msu.c | 8 +- drivers/infiniband/core/uverbs_main.c | 10 +- drivers/infiniband/hw/hfi1/file_ops.c | 2 +- drivers/infiniband/hw/qib/qib_file_ops.c | 2 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 6 +- drivers/misc/cxl/context.c | 9 +- drivers/misc/ocxl/context.c | 10 +- drivers/misc/ocxl/sysfs.c | 8 +- drivers/misc/sgi-gru/grumain.c | 4 +- drivers/scsi/cxlflash/ocxl_hw.c | 6 +- drivers/scsi/cxlflash/superpipe.c | 2 +- drivers/scsi/sg.c | 4 +- drivers/target/target_core_user.c | 6 +- drivers/uio/uio.c | 6 +- drivers/usb/mon/mon_bin.c | 4 +- drivers/vfio/pci/vfio_pci.c | 2 +- drivers/vfio/pci/vfio_pci_nvlink2.c | 8 +- drivers/vhost/vdpa.c | 6 +- drivers/video/fbdev/core/fb_defio.c | 14 +- drivers/xen/privcmd-buf.c | 5 +- drivers/xen/privcmd.c | 4 +- fs/9p/vfs_file.c | 2 +- fs/afs/write.c | 2 +- fs/btrfs/inode.c | 4 +- fs/ceph/addr.c | 6 +- fs/dax.c | 53 +-- fs/ext2/file.c | 6 +- fs/ext4/file.c | 6 +- fs/ext4/inode.c | 4 +- fs/f2fs/file.c | 8 +- fs/fuse/dax.c | 2 +- fs/fuse/file.c | 4 +- fs/gfs2/file.c | 8 +- fs/iomap/buffered-io.c | 2 +- fs/kernfs/file.c | 4 +- fs/nfs/file.c | 2 +- fs/nilfs2/file.c | 2 +- fs/ocfs2/mmap.c | 8 +- fs/orangefs/file.c | 2 +- fs/orangefs/inode.c | 4 +- fs/proc/vmcore.c | 4 +- fs/ubifs/file.c | 2 +- fs/userfaultfd.c | 17 +- fs/xfs/xfs_file.c | 18 +- fs/zonefs/super.c | 6 +- include/linux/huge_mm.h | 6 +- include/linux/mm.h | 21 +- include/linux/pgtable.h | 11 + include/trace/events/fs_dax.h | 28 +- ipc/shm.c | 2 +- kernel/events/core.c | 12 +- kernel/relay.c | 4 +- lib/test_hmm.c | 4 +- mm/filemap.c | 208 +++++++--- mm/huge_memory.c | 57 +-- mm/hugetlb.c | 6 +- mm/internal.h | 4 +- mm/khugepaged.c | 39 +- mm/memory.c | 452 +++++++++------------ mm/mmap.c | 6 +- mm/shmem.c | 16 +- mm/swap_state.c | 19 +- mm/swapfile.c | 13 +- samples/vfio-mdev/mbochs.c | 10 +- security/selinux/selinuxfs.c | 4 +- sound/core/pcm_native.c | 8 +- sound/usb/usx2y/us122l.c | 4 +- sound/usb/usx2y/usX2Yhwdep.c | 8 +- sound/usb/usx2y/usx2yhwdeppcm.c | 4 +- virt/kvm/kvm_main.c | 12 +- 104 files changed, 821 insertions(+), 730 deletions(-)
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
Will Deacon
maintainerPATCH v3 1/8 mm: Cleanup faultaround and finish_fault() codepaths
From: "Kirill A. Shutemov" <[email protected]>
alloc_set_pte() has two users with different requirements: in the faultaround code, it called from an atomic context and PTE page table has to be preallocated. finish_fault() can sleep and allocate page table as needed.
PTL locking rules are also strange, hard to follow and overkill for finish_fault().
Let's untangle the mess. alloc_set_pte() has gone now. All locking is explicit.
The price is some code duplication to handle huge pages in faultaround path, but it should be fine, having overall improvement in readability.
--- fs/xfs/xfs_file.c | 6 +- include/linux/mm.h | 12 ++- include/linux/pgtable.h | 11 +++ mm/filemap.c | 177 ++++++++++++++++++++++++++--------- mm/memory.c | 199 ++++++++++++---------------------------- 5 files changed, 213 insertions(+), 192 deletions(-)
fs/xfs/xfs_file.c +4 -2
@@ -1319,17 +1319,19 @@ xfs_filemap_pfn_mkwrite(return __xfs_filemap_fault (vmf,PE_SIZE_PTE , true); } -static void +static vm_fault_t xfs_filemap_map_pages (struct vm_fault *vmf,pgoff_t start_pgoff,pgoff_t end_pgoff) {struct inode *inode =file_inode (vmf->vma->vm_file); +vm_fault_t ret;xfs_ilock (XFS_I (inode),XFS_MMAPLOCK_SHARED ); -filemap_map_pages (vmf, start_pgoff, end_pgoff); + ret =filemap_map_pages (vmf, start_pgoff, end_pgoff);xfs_iunlock (XFS_I (inode),XFS_MMAPLOCK_SHARED ); +return ret; }static const struct vm_operations_struct xfs_file_vm_ops = {
include/linux/mm.h +7 -5
@@ -542,8 +542,8 @@ struct vm_fault {* is not NULL, otherwise pmd. */ pgtable_t prealloc_pte;/* Pre-allocated pte page table. -* vm_ops->map_pages() calls -* alloc_set_pte() from atomic context. +* vm_ops->map_pages() sets up a page +* table from atomic context. * do_fault_around() pre-allocates * page table to avoid allocation from * atomic context.
@@ -578,7 +578,7 @@ struct vm_operations_struct {vm_fault_t (*fault)(struct vm_fault *vmf);vm_fault_t (*huge_fault)(struct vm_fault *vmf,enum page_entry_size pe_size); -void (*map_pages)(struct vm_fault *vmf, +vm_fault_t (*map_pages)(struct vm_fault *vmf,pgoff_t start_pgoff,pgoff_t end_pgoff);unsigned long (*pagesize)(struct vm_area_struct * area);
@@ -988,7 +988,9 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) return pte; } -vm_fault_t alloc_set_pte (struct vm_fault *vmf,struct page *page); +vm_fault_t do_set_pmd (struct vm_fault *vmf,struct page *page); +void do_set_pte (struct vm_fault *vmf,struct page *page); +vm_fault_t finish_fault (struct vm_fault *vmf);vm_fault_t finish_mkwrite_fault (struct vm_fault *vmf);#endif
@@ -2622,7 +2624,7 @@ extern void truncate_inode_pages_final(struct address_space *);/* generic vm_area_ops exported for stackable file systems */ extern vm_fault_t filemap_fault (struct vm_fault *vmf); -extern void filemap_map_pages (struct vm_fault *vmf, +extern vm_fault_t filemap_map_pages (struct vm_fault *vmf,pgoff_t start_pgoff,pgoff_t end_pgoff);extern vm_fault_t filemap_page_mkwrite (struct vm_fault *vmf);
include/linux/pgtable.h +11 -0
@@ -1314,6 +1314,17 @@ static inline int pmd_trans_unstable(pmd_t *pmd)#endif } +/* +* the ordering of these checks is important for pmds with _page_devmap set. +* if we check pmd_trans_unstable() first we will trip the bad_pmd() check +* inside of pmd_none_or_trans_huge_or_clear_bad(). this will end up correctly +* returning 1 but not before it spams dmesg with the pmd_clear_bad() output. +*/ +static inline int pmd_devmap_trans_unstable (pmd_t *pmd) +{ +return pmd_devmap (*pmd) ||pmd_trans_unstable (pmd); +} +#ifndef CONFIG_NUMA_BALANCING /* * Technically a PTE can be PROTNONE even when not doing NUMA balancing but
mm/filemap.c +134 -43
214 lines not on this page · raw message
mm/memory.c +57 -142
285 lines not on this page · raw message
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
Re: [PATCH v3 1/8] mm: Cleanup faultaround and finish_fault() codepaths
Quoted 20 lines · Will Deacon
From: "Kirill A. Shutemov" <[email protected]>
alloc_set_pte()has two users with different requirements: in the faultaround code, it called from an atomic context and PTE page table has to be preallocated.finish_fault()can sleep and allocate page table as needed.PTL locking rules are also strange, hard to follow and overkill for
finish_fault().Let's untangle the mess.
alloc_set_pte()has gone now. All locking is explicit.The price is some code duplication to handle huge pages in faultaround path, but it should be fine, having overall improvement in readability.
Signed-off-by Kirill A. Shutemov <[email protected]>[will: s/from from/from/ in comment; spotted by willy]Signed-off-by Will Deacon <[email protected]>
Quoted 8 lines
--- fs/xfs/xfs_file.c | 6 +- include/linux/mm.h | 12 ++- include/linux/pgtable.h | 11 +++ mm/filemap.c | 177 ++++++++++++++++++++++++++--------- mm/memory.c | 199 ++++++++++++---------------------------- 5 files changed, 213 insertions(+), 192 deletions(-)
When building microblaze:mmu_defconfig:
mm/filemap.c: In function 'filemap_map_pages':mm/filemap.c:3153:3: error: implicit declaration of function 'update_mmu_cache'; did you mean 'update_mmu_tlb'?
Bisect log attached.
Guenter
---
# bad: [a4bfd8d46ac357c12529e4eebb6c89502b03ecc9] Add linux-next specific files for 20210209
# good: [92bf22614b21a2706f4993b278017e437f7785b3] Linux 5.11-rc7
git bisect start 'HEAD' 'v5.11-rc7'
# bad: [a8eb921ba7e8e77d994a1c6c69c8ef08456ecf53] Merge remote-tracking branch 'crypto/master'
git bisect bad a8eb921ba7e8e77d994a1c6c69c8ef08456ecf53
# bad: [b68df186dae8ae890df08059bb068b78252b053a] Merge remote-tracking branch 'hid/for-next'
git bisect bad b68df186dae8ae890df08059bb068b78252b053a
# bad: [323c9f6fb99b033883b404ecbc811e7b283a60b3] Merge remote-tracking branch 'sunxi/sunxi/for-next'
git bisect bad 323c9f6fb99b033883b404ecbc811e7b283a60b3
# bad: [4053c8a4d3b53205272aa16b65a5b7ed1a3a5b3e] Merge remote-tracking branch 'arm-soc/for-next'
git bisect bad 4053c8a4d3b53205272aa16b65a5b7ed1a3a5b3e
# good: [dfb8870aed6ad71fb7e378274521bf68a7d465cb] Merge branch 'arm/dt' into for-next
git bisect good dfb8870aed6ad71fb7e378274521bf68a7d465cb
# good: [9ad7e5a35f96b8bf8aebedfd8f397a64eecb21bd] Merge remote-tracking branch 'arm/for-next'
git bisect good 9ad7e5a35f96b8bf8aebedfd8f397a64eecb21bd
# good: [edc55d8409542cd05d5c17203615a162cddbcb4c] Merge branch 'arm/drivers' into for-next
git bisect good edc55d8409542cd05d5c17203615a162cddbcb4c
# good: [d1bbc35fcab28668c8992c4d5777234b794d7306] arm64: hibernate: add __force attribute to gfp_t casting
git bisect good d1bbc35fcab28668c8992c4d5777234b794d7306
# bad: [fb01b86f47a44f0c03278a7cc78ece8415898ed0] Merge branches 'for-next/cosmetic', 'for-next/crypto', 'for-next/faultaround', 'for-next/from-tip/irq/urgent', 'for-next/kexec', 'for-next/misc', 'for-next/mm', 'for-next/perf', 'for-next/random', 'for-next/rng', 'for-next/selftests', 'for-next/stacktrace', 'for-next/topology' and 'for-next/vdso' into for-next/core git bisect bad fb01b86f47a44f0c03278a7cc78ece8415898ed0
# good: [750d43b4a79e5f3767ee1db933b42abdf967ce1e] dt-bindings: arm: add Cortex-A78 binding
git bisect good 750d43b4a79e5f3767ee1db933b42abdf967ce1e
# good: [0188a894c390e51475274ece76b4d601782d709e] arm64: vmlinux.ld.S: add assertion for tramp_pg_dir offset
git bisect good 0188a894c390e51475274ece76b4d601782d709e
# bad: [3f98a28cc37253269b4104cf95a51f7716a2eb97] mm/nommu: Fix return type of filemap_map_pages()
git bisect bad 3f98a28cc37253269b4104cf95a51f7716a2eb97
# bad: [742d33729a0df11c9d8d4625dbf21dd20cdefd44] mm: Move immutable fields of 'struct vm_fault' into anonymous struct
git bisect bad 742d33729a0df11c9d8d4625dbf21dd20cdefd44
# bad: [46bdb4277f98e70d0c91f4289897ade533fe9e80] mm: Allow architectures to request 'old' entries when prefaulting
git bisect bad 46bdb4277f98e70d0c91f4289897ade533fe9e80
# bad: [f9ce0be71d1fbb038ada15ced83474b0e63f264d] mm: Cleanup faultaround and finish_fault() codepaths
git bisect bad f9ce0be71d1fbb038ada15ced83474b0e63f264d
# first bad commit: [f9ce0be71d1fbb038ada15ced83474b0e63f264d] mm: Cleanup faultaround and finish_fault() codepaths
Will Deacon
maintainerRe: [PATCH v3 1/8] mm: Cleanup faultaround and finish_fault() codepaths
Quoted 30 lines · Guenter Roeck
Quoted 20 lines · Will Deacon
From: "Kirill A. Shutemov" <[email protected]>
alloc_set_pte()has two users with different requirements: in the faultaround code, it called from an atomic context and PTE page table has to be preallocated.finish_fault()can sleep and allocate page table as needed.PTL locking rules are also strange, hard to follow and overkill for
finish_fault().Let's untangle the mess.
alloc_set_pte()has gone now. All locking is explicit.The price is some code duplication to handle huge pages in faultaround path, but it should be fine, having overall improvement in readability.
Signed-off-by Kirill A. Shutemov <[email protected]>[will: s/from from/from/ in comment; spotted by willy]Signed-off-by Will Deacon <[email protected]>Quoted 8 lines
--- fs/xfs/xfs_file.c | 6 +- include/linux/mm.h | 12 ++- include/linux/pgtable.h | 11 +++ mm/filemap.c | 177 ++++++++++++++++++++++++++--------- mm/memory.c | 199 ++++++++++++---------------------------- 5 files changed, 213 insertions(+), 192 deletions(-)
Quoted 6 lines
When building microblaze:mmu_defconfig:
mm/filemap.c: In function 'filemap_map_pages':mm/filemap.c:3153:3: error: implicit declaration of function 'update_mmu_cache'; did you mean 'update_mmu_tlb'?Bisect log attached.
Looks like a missing include.
Will
--->8
From 076f93117c067d5b6caab4773c6d6da130859cc4 Mon Sep 17 00:00:00 2001
From: Will Deacon <[email protected]>
Date: Wed, 10 Feb 2021 11:15:11 +0000
Subject: [PATCH] mm: filemap: Fix microblaze build failure with
'mmu_defconfig'
Commit f9ce0be71d1f ("mm: Cleanup faultaround and finish_fault() codepaths") added a call to 'update_mmu_cache()' in mm/filemap.c, which breaks the build for microblaze:
| mm/filemap.c: In function 'filemap_map_pages':
| mm/filemap.c:3153:3: error: implicit declaration of function 'update_mmu_cache'; did you mean 'update_mmu_tlb'?
Include asm/tlbflush.h in mm/filemap.c to make sure that the function (or indeed, macro) is available.
--- mm/filemap.c | 1 + 1 file changed, 1 insertion(+)
mm/filemap.c +1 -0
@@ -43,6 +43,7 @@#include <linux/ramfs.h> #include <linux/page_idle.h> #include <asm/pgalloc.h> +#include <asm/tlbflush.h> #include "internal.h" #define CREATE_TRACE_POINTS
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Guenter Roeck
Tested-byRe: [PATCH v3 1/8] mm: Cleanup faultaround and finish_fault() codepaths
Quoted 31 lines · Will Deacon
Quoted 30 lines · Guenter Roeck
Quoted 28 lines · Will Deacon
From: "Kirill A. Shutemov" <[email protected]>
alloc_set_pte()has two users with different requirements: in the faultaround code, it called from an atomic context and PTE page table has to be preallocated.finish_fault()can sleep and allocate page table as needed.PTL locking rules are also strange, hard to follow and overkill for
finish_fault().Let's untangle the mess.
alloc_set_pte()has gone now. All locking is explicit.The price is some code duplication to handle huge pages in faultaround path, but it should be fine, having overall improvement in readability.
Signed-off-by Kirill A. Shutemov <[email protected]>[will: s/from from/from/ in comment; spotted by willy]Signed-off-by Will Deacon <[email protected]>--- fs/xfs/xfs_file.c | 6 +- include/linux/mm.h | 12 ++- include/linux/pgtable.h | 11 +++ mm/filemap.c | 177 ++++++++++++++++++++++++++--------- mm/memory.c | 199 ++++++++++++---------------------------- 5 files changed, 213 insertions(+), 192 deletions(-)
Quoted 9 lines
Quoted 6 lines
When building microblaze:mmu_defconfig:
mm/filemap.c: In function 'filemap_map_pages':mm/filemap.c:3153:3: error: implicit declaration of function 'update_mmu_cache'; did you mean 'update_mmu_tlb'?Bisect log attached.
Looks like a missing include.
Indeed.
Quoted 17 lines
Will
--->8
Quoted 1 line
From
076f93117c067d5b6caab4773c6d6da130859cc4Mon Sep 17 00:00:00 2001From: Will Deacon <[email protected]>
Date: Wed, 10 Feb 2021 11:15:11 +0000
Subject: [PATCH] mm: filemap: Fix microblaze build failure with
'mmu_defconfig'Commit
f9ce0be71d1f("mm: Cleanup faultaround andfinish_fault()codepaths") added a call to 'update_mmu_cache()' inmm/filemap.c, which breaks the build for microblaze:|
mm/filemap.c: In function 'filemap_map_pages':
|mm/filemap.c:3153:3: error: implicit declaration of function 'update_mmu_cache'; did you mean 'update_mmu_tlb'?
Quoted 6 lines
Include
asm/tlbflush.hinmm/filemap.cto make sure that the function (or indeed, macro) is available.Reported-by Guenter Roeck <[email protected]>Signed-off-by Will Deacon <[email protected]>
Quoted 17 lines
--- mm/filemap.c | 1 + 1 file changed, 1 insertion(+)mm/filemap.c +1
-0@@ -43,6 +43,7 @@#include <linux/ramfs.h> #include <linux/page_idle.h> #include <asm/pgalloc.h> +#include <asm/tlbflush.h> #include "internal.h" #define CREATE_TRACE_POINTS
Will Deacon
maintainerPATCH v3 2/8 mm: Allow architectures to request 'old' entries when prefaulting
In reply to Will DeaconCommit 5c0a85fad949 ("mm: make faultaround produce old ptes") changed the "faultaround" behaviour to initialise prefaulted PTEs as 'old', since this avoids vmscan wrongly assuming that they are hot, despite having never been explicitly accessed by userspace. The change has been shown to benefit numerous arm64 micro-architectures (with hardware access flag) running Android, where both application launch latency and direct reclaim time are significantly reduced (by 10%+ and ~80% respectively).
Unfortunately, commit 315d09bf30c2 ("Revert "mm: make faultaround produce old ptes"") reverted the change due to it being identified as the cause of a ~6% regression in unixbench on x86. Experiments on a variety of recent arm64 micro-architectures indicate that unixbench is not affected by the original commit, which appears to yield a 0-1% performance improvement.
Since one size does not fit all for the initial state of prefaulted PTEs, introduce arch_wants_old_prefaulted_pte(), which allows an architecture to opt-in to 'old' prefaulted PTEs at runtime based on whatever criteria it may have.
--- include/linux/mm.h | 5 ++++- mm/filemap.c | 14 ++++++++++---- mm/memory.c | 20 +++++++++++++++++++- 3 files changed, 33 insertions(+), 6 deletions(-)
include/linux/mm.h +4 -1
@@ -434,6 +434,7 @@ extern pgprot_t protection_map[16];* @FAULT_FLAG_REMOTE: The fault is not for current task/mm. * @FAULT_FLAG_INSTRUCTION: The fault was during an instruction fetch. * @FAULT_FLAG_INTERRUPTIBLE: The fault can be interrupted by non-fatal signals. +* @FAULT_FLAG_PREFAULT: Fault was a prefault. * * About @FAULT_FLAG_ALLOW_RETRY and @FAULT_FLAG_TRIED: we can specify * whether we would allow page faults to retry by specifying these two
@@ -464,6 +465,7 @@ extern pgprot_t protection_map[16];#define FAULT_FLAG_REMOTE 0x80#define FAULT_FLAG_INSTRUCTION 0x100#define FAULT_FLAG_INTERRUPTIBLE 0x200 +#define FAULT_FLAG_PREFAULT 0x400/* * The default fault flags that should be used by most of the
@@ -501,7 +503,8 @@ static inline bool fault_flag_allow_retry_first(unsigned int flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_PREFAULT, "PREFAULT" }/* * vm_fault is filled by the pagefault handler and passed to the vma's
mm/filemap.c +10 -4
@@ -3019,6 +3019,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,struct address_space *mapping = file->f_mapping;pgoff_t last_pgoff = start_pgoff;unsigned long address = vmf->address; +unsigned long flags = vmf->flags;XA_STATE (xas, &mapping->i_pages, start_pgoff);struct page *head, *page;unsigned int mmap_miss =READ_ONCE (file->f_ra.mmap_miss);
@@ -3051,14 +3052,18 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,if (!pte_none (*vmf->pte)) goto unlock; +/* We're about to handle the fault */ +if (vmf->address == address) { + vmf->flags &= ~FAULT_FLAG_PREFAULT ; + ret =VM_FAULT_NOPAGE ; + }else { + vmf->flags |=FAULT_FLAG_PREFAULT ; + } +do_set_pte (vmf, page);/* no need to invalidate: a not-present page won't be cached */ update_mmu_cache (vma, vmf->address, vmf->pte);unlock_page (head); - -/* The fault is handled */ -if (vmf->address == address) - ret =VM_FAULT_NOPAGE ;continue ; unlock:unlock_page (head);
@@ -3067,6 +3072,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,pte_unmap_unlock (vmf->pte, vmf->ptl); out:rcu_read_unlock (); + vmf->flags = flags; vmf->address = address;WRITE_ONCE (file->f_ra.mmap_miss, mmap_miss);return ret;
mm/memory.c +19 -1
@@ -134,6 +134,18 @@ static inline bool arch_faults_on_old_pte(void) }#endif +#ifndef arch_wants_old_prefaulted_pte +static inline bool arch_wants_old_prefaulted_pte (void ) +{ +/* +* Transitioning a PTE from 'old' to 'young' can be expensive on +* some architectures, even if it's performed in hardware. By +* default, "false" means prefaulted entries will be 'young'. +*/ +return false; +} +#endif +static int __init disable_randmaps (char *s) { randomize_va_space =0 ;
@@ -3725,11 +3737,17 @@ void do_set_pte(struct vm_fault *vmf, struct page *page) {struct vm_area_struct *vma = vmf->vma;bool write = vmf->flags &FAULT_FLAG_WRITE ; +bool prefault = vmf->flags &FAULT_FLAG_PREFAULT ;pte_t entry;flush_icache_page (vma, page); entry =mk_pte (page, vma->vm_page_prot); - entry =pte_sw_mkyoung (entry); + +if (prefault &&arch_wants_old_prefaulted_pte ()) + entry =pte_mkold (entry); +else + entry =pte_sw_mkyoung (entry); +if (write) entry =maybe_mkwrite (pte_mkdirty (entry), vma);/* copy-on-write page */
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
Will Deacon
maintainerPATCH v3 3/8 arm64: mm: Implement arch_wants_old_prefaulted_pte()
In reply to Will DeaconOn CPUs with hardware AF/DBM, initialising prefaulted PTEs as 'old' improves vmscan behaviour and does not appear to introduce any overhead elsewhere.
Implement arch_wants_old_prefaulted_pte() to return 'true' if we detect hardware access flag support at runtime. This can be extended in future based on MIDR matching if necessary.
--- arch/arm64/include/asm/pgtable.h | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
arch/arm64/include/asm/pgtable.h +11 -1
@@ -980,7 +980,17 @@ static inline bool arch_faults_on_old_pte(void) return !cpu_has_hw_af(); } -#define arch_faults_on_old_pte arch_faults_on_old_pte +#define arch_faults_on_old_pte arch_faults_on_old_pte + +/* +* Experimentally, it's cheap to set the access flag in hardware and we +* benefit from prefaulting mappings as 'old' to start with. +*/ +static inline bool arch_wants_old_prefaulted_pte (void ) +{ +return !arch_faults_on_old_pte (); +} +#define arch_wants_old_prefaulted_pte arch_wants_old_prefaulted_pte#endif /* !__ASSEMBLY__ */
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
Will Deacon
maintainerRFC PATCH v3 4/8 mm: Separate fault info out of 'struct vm_fault'
In reply to Will Deacon'struct vm_fault' contains both information about the fault being serviced alongside mutable fields contributing to the state of the fault-handling logic. Unfortunately, the distinction between the two is not clear-cut, and a number of callers end up manipulating the structure temporarily before restoring it when returning.
Try to clean this up by splitting the immutable fault information out into a new 'struct vm_fault_info' which is embedded in 'struct vm_fault' and will later be made 'const'. The vast majority of this change was performed with a coccinelle patch:
// SPDX-License-Identifier: GPL-2.0
virtual context
virtual patch
virtual org
virtual report
@r_patch depends on !context && patch && !org && !report@ struct vm_fault *vmf;
@@
(
- vmf->vma
+ vmf->info.vma
|
- vmf->gfp_mask
+ vmf->info.gfp_mask
|
- vmf->pgoff
+ vmf->info.pgoff
|
- vmf->address
+ vmf->info.address
)
@r2_patch depends on !context && patch && !org && !report@ struct vm_fault vmf;
@@
(
- vmf.vma
+ vmf.info.vma
|
- vmf.gfp_mask
+ vmf.info.gfp_mask
|
- vmf.pgoff
+ vmf.info.pgoff
|
- vmf.address
+ vmf.info.address
)
but I ended up having to fix up some stragglers by hand.
Ideally, the 'flags' field would be part of the new structure too, but it seems as though the ->page_mkwrite() path is not ready for this yet.
--- arch/arm64/kernel/vdso.c | 4 +- arch/powerpc/kvm/book3s_64_vio.c | 6 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 4 +- arch/powerpc/kvm/book3s_xive_native.c | 13 +- arch/powerpc/platforms/cell/spufs/file.c | 16 +- arch/s390/kernel/vdso.c | 4 +- arch/s390/kvm/kvm-s390.c | 2 +- arch/x86/entry/vdso/vma.c | 22 +- arch/x86/kernel/cpu/sgx/encl.c | 4 +- drivers/char/agp/alpha-agp.c | 2 +- drivers/char/mspec.c | 6 +- drivers/dax/device.c | 37 ++-- drivers/dma-buf/heaps/cma_heap.c | 6 +- drivers/dma-buf/udmabuf.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +- drivers/gpu/drm/armada/armada_gem.c | 6 +- drivers/gpu/drm/drm_gem_shmem_helper.c | 8 +- drivers/gpu/drm/drm_vm.c | 18 +- drivers/gpu/drm/etnaviv/etnaviv_gem.c | 10 +- drivers/gpu/drm/gma500/framebuffer.c | 4 +- drivers/gpu/drm/gma500/gem.c | 8 +- drivers/gpu/drm/i915/gem/i915_gem_mman.c | 8 +- drivers/gpu/drm/msm/msm_gem.c | 11 +- drivers/gpu/drm/nouveau/nouveau_dmem.c | 8 +- drivers/gpu/drm/nouveau/nouveau_ttm.c | 2 +- drivers/gpu/drm/omapdrm/omap_gem.c | 20 +- drivers/gpu/drm/radeon/radeon_ttm.c | 4 +- drivers/gpu/drm/tegra/gem.c | 6 +- drivers/gpu/drm/ttm/ttm_bo_vm.c | 10 +- drivers/gpu/drm/vc4/vc4_bo.c | 2 +- drivers/gpu/drm/vgem/vgem_drv.c | 6 +- drivers/gpu/drm/vmwgfx/vmwgfx_page_dirty.c | 12 +- drivers/hsi/clients/cmt_speech.c | 2 +- drivers/hwtracing/intel_th/msu.c | 8 +- drivers/infiniband/core/uverbs_main.c | 10 +- drivers/infiniband/hw/hfi1/file_ops.c | 2 +- drivers/infiniband/hw/qib/qib_file_ops.c | 2 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 6 +- drivers/misc/cxl/context.c | 9 +- drivers/misc/ocxl/context.c | 10 +- drivers/misc/ocxl/sysfs.c | 8 +- drivers/misc/sgi-gru/grumain.c | 4 +- drivers/scsi/cxlflash/ocxl_hw.c | 6 +- drivers/scsi/cxlflash/superpipe.c | 2 +- drivers/scsi/sg.c | 4 +- drivers/target/target_core_user.c | 6 +- drivers/uio/uio.c | 6 +- drivers/usb/mon/mon_bin.c | 4 +- drivers/vfio/pci/vfio_pci.c | 2 +- drivers/vfio/pci/vfio_pci_nvlink2.c | 8 +- drivers/vhost/vdpa.c | 6 +- drivers/video/fbdev/core/fb_defio.c | 14 +- drivers/xen/privcmd-buf.c | 5 +- drivers/xen/privcmd.c | 4 +- fs/9p/vfs_file.c | 2 +- fs/afs/write.c | 2 +- fs/btrfs/inode.c | 4 +- fs/ceph/addr.c | 6 +- fs/dax.c | 53 ++--- fs/ext2/file.c | 6 +- fs/ext4/file.c | 6 +- fs/ext4/inode.c | 4 +- fs/f2fs/file.c | 8 +- fs/fuse/dax.c | 2 +- fs/fuse/file.c | 4 +- fs/gfs2/file.c | 8 +- fs/iomap/buffered-io.c | 2 +- fs/kernfs/file.c | 4 +- fs/nfs/file.c | 2 +- fs/nilfs2/file.c | 2 +- fs/ocfs2/mmap.c | 8 +- fs/orangefs/file.c | 2 +- fs/orangefs/inode.c | 4 +- fs/proc/vmcore.c | 4 +- fs/ubifs/file.c | 2 +- fs/userfaultfd.c | 17 +- fs/xfs/xfs_file.c | 12 +- fs/zonefs/super.c | 6 +- include/linux/huge_mm.h | 6 +- include/linux/mm.h | 9 +- include/trace/events/fs_dax.h | 28 +-- ipc/shm.c | 2 +- kernel/events/core.c | 12 +- kernel/relay.c | 4 +- lib/test_hmm.c | 4 +- mm/filemap.c | 49 +++-- mm/huge_memory.c | 57 ++--- mm/hugetlb.c | 6 +- mm/internal.h | 4 +- mm/khugepaged.c | 16 +- mm/memory.c | 245 +++++++++++---------- mm/mmap.c | 6 +- mm/shmem.c | 12 +- mm/swap_state.c | 19 +- mm/swapfile.c | 4 +- samples/vfio-mdev/mbochs.c | 10 +- security/selinux/selinuxfs.c | 4 +- sound/core/pcm_native.c | 8 +- sound/usb/usx2y/us122l.c | 4 +- sound/usb/usx2y/usX2Yhwdep.c | 8 +- sound/usb/usx2y/usx2yhwdeppcm.c | 4 +- virt/kvm/kvm_main.c | 12 +- 102 files changed, 574 insertions(+), 534 deletions(-)
arch/arm64/kernel/vdso.c +2 -2
@@ -183,7 +183,7 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,struct page *timens_page =find_timens_vvar_page (vma);unsigned long pfn; -switch (vmf->pgoff) { +switch (vmf->info.pgoff) {case VVAR_DATA_PAGE_OFFSET :if (timens_page) pfn =page_to_pfn (timens_page);
@@ -208,7 +208,7 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,return VM_FAULT_SIGBUS ; } -return vmf_insert_pfn (vma, vmf->address, pfn); +return vmf_insert_pfn (vma, vmf->info.address, pfn); } static int __setup_additional_pages(enum vdso_abi abi,
arch/powerpc/kvm/book3s_64_vio.c +3 -3
@@ -218,13 +218,13 @@ static struct page *kvm_spapr_get_tce_page(struct kvmppc_spapr_tce_table *stt,static vm_fault_t kvm_spapr_tce_fault (struct vm_fault *vmf) { -struct kvmppc_spapr_tce_table *stt = vmf->vma->vm_file->private_data; +struct kvmppc_spapr_tce_table *stt = vmf->info.vma->vm_file->private_data;struct page *page; -if (vmf->pgoff >=kvmppc_tce_pages (stt->size)) +if (vmf->info.pgoff >=kvmppc_tce_pages (stt->size))return VM_FAULT_SIGBUS ; - page =kvm_spapr_get_tce_page (stt, vmf->pgoff); + page =kvm_spapr_get_tce_page (stt, vmf->info.pgoff);if (!page)return VM_FAULT_OOM ;
arch/powerpc/kvm/book3s_hv_uvmem.c +2 -2
@@ -989,8 +989,8 @@ static vm_fault_t kvmppc_uvmem_migrate_to_ram(struct vm_fault *vmf) {struct kvmppc_uvmem_page_pvt *pvt = vmf->page->zone_device_data; -if (kvmppc_svm_page_out (vmf->vma, vmf->address, - vmf->address +PAGE_SIZE ,PAGE_SHIFT , +if (kvmppc_svm_page_out (vmf->info.vma, vmf->info.address, + vmf->info.address +PAGE_SIZE ,PAGE_SHIFT , pvt->kvm, pvt->gpa))return VM_FAULT_SIGBUS ;else
arch/powerpc/kvm/book3s_xive_native.c +7 -6
@@ -222,7 +222,7 @@ static struct kvmppc_xive_ops kvmppc_xive_native_ops = {static vm_fault_t xive_native_esb_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct kvm_device *dev = vma->vm_file->private_data;struct kvmppc_xive *xive = dev->private;struct kvmppc_xive_src_block *sb;
@@ -238,7 +238,7 @@ static vm_fault_t xive_native_esb_fault(struct vm_fault *vmf)* Linux/KVM uses a two pages ESB setting, one for trigger and * one for EOI */ - page_offset = vmf->pgoff - vma->vm_pgoff; + page_offset = vmf->info.pgoff - vma->vm_pgoff; irq = page_offset /2 ; sb =kvmppc_xive_find_source (xive, irq, &src);
@@ -272,7 +272,7 @@ static vm_fault_t xive_native_esb_fault(struct vm_fault *vmf)return VM_FAULT_SIGBUS ; } -vmf_insert_pfn (vma, vmf->address, page >>PAGE_SHIFT ); +vmf_insert_pfn (vma, vmf->info.address, page >>PAGE_SHIFT );return VM_FAULT_NOPAGE ; }
@@ -282,14 +282,15 @@ static const struct vm_operations_struct xive_native_esb_vmops = {static vm_fault_t xive_native_tima_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma; -switch (vmf->pgoff - vma->vm_pgoff) { +switch (vmf->info.pgoff - vma->vm_pgoff) {case 0 :/* HW - forbid access */ case 1 :/* HV - forbid access */ return VM_FAULT_SIGBUS ;case 2 :/* OS */ -vmf_insert_pfn (vma, vmf->address, xive_tima_os >>PAGE_SHIFT ); +vmf_insert_pfn (vma, vmf->info.address, + xive_tima_os >>PAGE_SHIFT );return VM_FAULT_NOPAGE ;case 3 :/* USER - TODO */ default :
arch/powerpc/platforms/cell/spufs/file.c +8 -8
@@ -229,17 +229,17 @@ spufs_mem_write(struct file *file, const char __user *buffer,static vm_fault_t spufs_mem_mmap_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct spu_context *ctx = vma->vm_file->private_data;unsigned long pfn, offset;vm_fault_t ret; - offset = vmf->pgoff <<PAGE_SHIFT ; + offset = vmf->info.pgoff <<PAGE_SHIFT ;if (offset >=LS_SIZE )return VM_FAULT_SIGBUS ;pr_debug ("spufs_mem_mmap_fault address=0x%lx, offset=0x%lx\n" , - vmf->address, offset); + vmf->info.address, offset);if (spu_acquire (ctx))return VM_FAULT_NOPAGE ;
@@ -251,7 +251,7 @@ spufs_mem_mmap_fault(struct vm_fault *vmf) vma->vm_page_prot =pgprot_noncached_wc (vma->vm_page_prot); pfn = (ctx->spu->local_store_phys + offset) >>PAGE_SHIFT ; } - ret =vmf_insert_pfn (vma, vmf->address, pfn); + ret =vmf_insert_pfn (vma, vmf->info.address, pfn);spu_release (ctx);
@@ -311,8 +311,8 @@ static vm_fault_t spufs_ps_fault(struct vm_fault *vmf, unsigned long ps_offs, unsigned long ps_size) { -struct spu_context *ctx = vmf->vma->vm_file->private_data; -unsigned long area, offset = vmf->pgoff <<PAGE_SHIFT ; +struct spu_context *ctx = vmf->info.vma->vm_file->private_data; +unsigned long area, offset = vmf->info.pgoff <<PAGE_SHIFT ;int err =0 ;vm_fault_t ret =VM_FAULT_NOPAGE ;
@@ -350,8 +350,8 @@ static vm_fault_t spufs_ps_fault(struct vm_fault *vmf,mmap_read_lock (current->mm); }else { area = ctx->spu->problem_phys + ps_offs; - ret =vmf_insert_pfn (vmf->vma, vmf->address, - (area + offset) >>PAGE_SHIFT ); + ret =vmf_insert_pfn (vmf->info.vma, vmf->info.address, + (area + offset) >>PAGE_SHIFT );spu_context_trace (spufs_ps_fault__insert, ctx, ctx->spu); }
arch/s390/kernel/vdso.c +2 -2
@@ -51,10 +51,10 @@ static vm_fault_t vdso_fault(const struct vm_special_mapping *sm, vdso_pagelist = vdso64_pagelist; vdso_pages = vdso64_pages; -if (vmf->pgoff >= vdso_pages) +if (vmf->info.pgoff >= vdso_pages)return VM_FAULT_SIGBUS ; - vmf->page = vdso_pagelist[vmf->pgoff]; + vmf->page = vdso_pagelist[vmf->info.pgoff];get_page (vmf->page);return 0 ; }
arch/s390/kvm/kvm-s390.c +1 -1
@@ -4952,7 +4952,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,vm_fault_t kvm_arch_vcpu_fault (struct kvm_vcpu *vcpu,struct vm_fault *vmf) {#ifdef CONFIG_KVM_S390_UCONTROL -if ((vmf->pgoff ==KVM_S390_SIE_PAGE_OFFSET ) +if ((vmf->info.pgoff ==KVM_S390_SIE_PAGE_OFFSET ) && (kvm_is_ucontrol (vcpu->kvm))) { vmf->page =virt_to_page (vcpu->arch.sie_block);get_page (vmf->page);
arch/x86/entry/vdso/vma.c +11 -11
@@ -61,10 +61,10 @@ static vm_fault_t vdso_fault(const struct vm_special_mapping *sm, {const struct vdso_image *image = vma->vm_mm->context.vdso_image; -if (!image || (vmf->pgoff <<PAGE_SHIFT ) >= image->size) +if (!image || (vmf->info.pgoff <<PAGE_SHIFT ) >= image->size)return VM_FAULT_SIGBUS ; - vmf->page =virt_to_page (image->data + (vmf->pgoff <<PAGE_SHIFT )); + vmf->page =virt_to_page (image->data + (vmf->info.pgoff <<PAGE_SHIFT ));get_page (vmf->page);return 0 ; }
@@ -157,7 +157,7 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,if (!image)return VM_FAULT_SIGBUS ; - sym_offset = (long )(vmf->pgoff <<PAGE_SHIFT ) + + sym_offset = (long )(vmf->info.pgoff <<PAGE_SHIFT ) + image->sym_vvar_start;/*
@@ -192,7 +192,7 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,* offsets for clocks on VVAR, it'll be faulted * shortly by VDSO code. */ - addr = vmf->address + (image->sym_timens_page - sym_offset); + addr = vmf->info.address + (image->sym_timens_page - sym_offset); err =vmf_insert_pfn (vma, addr, pfn);if (unlikely (err &VM_FAULT_ERROR ))return err;
@@ -200,21 +200,21 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm, pfn =page_to_pfn (timens_page); } -return vmf_insert_pfn (vma, vmf->address, pfn); +return vmf_insert_pfn (vma, vmf->info.address, pfn); }else if (sym_offset == image->sym_pvclock_page) {struct pvclock_vsyscall_time_info *pvti =pvclock_get_pvti_cpu0_va ();if (pvti &&vclock_was_used (VDSO_CLOCKMODE_PVCLOCK )) { -return vmf_insert_pfn_prot (vma, vmf->address, -__pa (pvti) >>PAGE_SHIFT , -pgprot_decrypted (vma->vm_page_prot)); +return vmf_insert_pfn_prot (vma, vmf->info.address, +__pa (pvti) >>PAGE_SHIFT , +pgprot_decrypted (vma->vm_page_prot)); } }else if (sym_offset == image->sym_hvclock_page) {struct ms_hyperv_tsc_page *tsc_pg =hv_get_tsc_page ();if (tsc_pg &&vclock_was_used (VDSO_CLOCKMODE_HVCLOCK )) -return vmf_insert_pfn (vma, vmf->address, -virt_to_phys (tsc_pg) >>PAGE_SHIFT ); +return vmf_insert_pfn (vma, vmf->info.address, +virt_to_phys (tsc_pg) >>PAGE_SHIFT ); }else if (sym_offset == image->sym_timens_page) {struct page *timens_page =find_timens_vvar_page (vma);
@@ -222,7 +222,7 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,return VM_FAULT_SIGBUS ; pfn =__pa_symbol (&__vvar_page) >>PAGE_SHIFT ; -return vmf_insert_pfn (vma, vmf->address, pfn); +return vmf_insert_pfn (vma, vmf->info.address, pfn); }return VM_FAULT_SIGBUS ;
arch/x86/kernel/cpu/sgx/encl.c +2 -2
@@ -136,8 +136,8 @@ static struct sgx_encl_page *sgx_encl_load_page(struct sgx_encl *encl,static vm_fault_t sgx_vma_fault (struct vm_fault *vmf) { -unsigned long addr = (unsigned long )vmf->address; -struct vm_area_struct *vma = vmf->vma; +unsigned long addr = (unsigned long ) vmf->info.address; +struct vm_area_struct *vma = vmf->info.vma;struct sgx_encl_page *entry;unsigned long phys_addr;struct sgx_encl *encl;
drivers/char/agp/alpha-agp.c +1 -1
@@ -18,7 +18,7 @@ static vm_fault_t alpha_core_agp_vm_fault(struct vm_fault *vmf)unsigned long pa;struct page *page; - dma_addr = vmf->address - vmf->vma->vm_start + agp->aperture.bus_base; + dma_addr = vmf->info.address - vmf->info.vma->vm_start + agp->aperture.bus_base; pa = agp->ops->translate (agp, dma_addr);if (pa == (unsigned long )-EINVAL )
drivers/char/mspec.c +3 -3
@@ -138,8 +138,8 @@ mspec_fault(struct vm_fault *vmf) {unsigned long paddr, maddr;unsigned long pfn; -pgoff_t index = vmf->pgoff; -struct vma_data *vdata = vmf->vma->vm_private_data; +pgoff_t index = vmf->info.pgoff; +struct vma_data *vdata = vmf->info.vma->vm_private_data; maddr = (volatile unsigned long ) vdata->maddr[index];if (maddr ==0 ) {
@@ -161,7 +161,7 @@ mspec_fault(struct vm_fault *vmf) paddr = maddr & ~__IA64_UNCACHED_OFFSET; pfn = paddr >>PAGE_SHIFT ; -return vmf_insert_pfn (vmf->vma, vmf->address, pfn); +return vmf_insert_pfn (vmf->info.vma, vmf->info.address, pfn); }static const struct vm_operations_struct mspec_vm_ops = {
drivers/dax/device.c +19 -18
@@ -80,7 +80,7 @@ static vm_fault_t __dev_dax_pte_fault(struct dev_dax *dev_dax,phys_addr_t phys;unsigned int fault_size =PAGE_SIZE ; -if (check_vma (dev_dax, vmf->vma, __func__)) +if (check_vma (dev_dax, vmf->info.vma, __func__))return VM_FAULT_SIGBUS ;if (dev_dax->align >PAGE_SIZE ) {
@@ -92,27 +92,27 @@ static vm_fault_t __dev_dax_pte_fault(struct dev_dax *dev_dax,if (fault_size != dev_dax->align)return VM_FAULT_SIGBUS ; - phys =dax_pgoff_to_phys (dev_dax, vmf->pgoff,PAGE_SIZE ); + phys =dax_pgoff_to_phys (dev_dax, vmf->info.pgoff,PAGE_SIZE );if (phys ==-1 ) { -dev_dbg (dev,"pgoff_to_phys(%#lx) failed\n" , vmf->pgoff); +dev_dbg (dev,"pgoff_to_phys(%#lx) failed\n" , vmf->info.pgoff);return VM_FAULT_SIGBUS ; } *pfn =phys_to_pfn_t (phys,PFN_DEV |PFN_MAP ); -return vmf_insert_mixed (vmf->vma, vmf->address, *pfn); +return vmf_insert_mixed (vmf->info.vma, vmf->info.address, *pfn); }static vm_fault_t __dev_dax_pmd_fault (struct dev_dax *dev_dax,struct vm_fault *vmf,pfn_t *pfn) { -unsigned long pmd_addr = vmf->address &PMD_MASK ; +unsigned long pmd_addr = vmf->info.address &PMD_MASK ;struct device *dev = &dev_dax->dev;phys_addr_t phys;pgoff_t pgoff;unsigned int fault_size =PMD_SIZE ; -if (check_vma (dev_dax, vmf->vma, __func__)) +if (check_vma (dev_dax, vmf->info.vma, __func__))return VM_FAULT_SIGBUS ;if (dev_dax->align >PMD_SIZE ) {
@@ -127,11 +127,11 @@ static vm_fault_t __dev_dax_pmd_fault(struct dev_dax *dev_dax,return VM_FAULT_FALLBACK ;/* if we are outside of the VMA */ -if (pmd_addr < vmf->vma->vm_start || - (pmd_addr +PMD_SIZE ) > vmf->vma->vm_end) +if (pmd_addr < vmf->info.vma->vm_start || + (pmd_addr +PMD_SIZE ) > vmf->info.vma->vm_end)return VM_FAULT_SIGBUS ; - pgoff =linear_page_index (vmf->vma, pmd_addr); + pgoff =linear_page_index (vmf->info.vma, pmd_addr); phys =dax_pgoff_to_phys (dev_dax, pgoff,PMD_SIZE );if (phys ==-1 ) {dev_dbg (dev,"pgoff_to_phys(%#lx) failed\n" , pgoff);
@@ -147,14 +147,14 @@ static vm_fault_t __dev_dax_pmd_fault(struct dev_dax *dev_dax,static vm_fault_t __dev_dax_pud_fault (struct dev_dax *dev_dax,struct vm_fault *vmf,pfn_t *pfn) { -unsigned long pud_addr = vmf->address &PUD_MASK ; +unsigned long pud_addr = vmf->info.address &PUD_MASK ;struct device *dev = &dev_dax->dev;phys_addr_t phys;pgoff_t pgoff;unsigned int fault_size =PUD_SIZE ; -if (check_vma (dev_dax, vmf->vma, __func__)) +if (check_vma (dev_dax, vmf->info.vma, __func__))return VM_FAULT_SIGBUS ;if (dev_dax->align >PUD_SIZE ) {
@@ -169,11 +169,11 @@ static vm_fault_t __dev_dax_pud_fault(struct dev_dax *dev_dax,return VM_FAULT_FALLBACK ;/* if we are outside of the VMA */ -if (pud_addr < vmf->vma->vm_start || - (pud_addr +PUD_SIZE ) > vmf->vma->vm_end) +if (pud_addr < vmf->info.vma->vm_start || + (pud_addr +PUD_SIZE ) > vmf->info.vma->vm_end)return VM_FAULT_SIGBUS ; - pgoff =linear_page_index (vmf->vma, pud_addr); + pgoff =linear_page_index (vmf->info.vma, pud_addr); phys =dax_pgoff_to_phys (dev_dax, pgoff,PUD_SIZE );if (phys ==-1 ) {dev_dbg (dev,"pgoff_to_phys(%#lx) failed\n" , pgoff);
@@ -195,7 +195,7 @@ static vm_fault_t __dev_dax_pud_fault(struct dev_dax *dev_dax,static vm_fault_t dev_dax_huge_fault (struct vm_fault *vmf,enum page_entry_size pe_size) { -struct file *filp = vmf->vma->vm_file; +struct file *filp = vmf->info.vma->vm_file;unsigned long fault_size;vm_fault_t rc =VM_FAULT_SIGBUS ;int id;
@@ -204,7 +204,8 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf,dev_dbg (&dev_dax->dev,"%s: %s (%#lx - %#lx) size = %d\n" , current->comm, (vmf->flags &FAULT_FLAG_WRITE ) ?"write" :"read" , - vmf->vma->vm_start, vmf->vma->vm_end, pe_size); + vmf->info.vma->vm_start, vmf->info.vma->vm_end, + pe_size); id =dax_read_lock ();switch (pe_size) {
@@ -234,8 +235,8 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf,* mapped. No need to consider the zero page, or racing * conflicting mappings. */ - pgoff =linear_page_index (vmf->vma, vmf->address - & ~(fault_size -1 )); + pgoff =linear_page_index (vmf->info.vma, vmf->info.address + & ~(fault_size -1 ));for (i =0 ; i < fault_size /PAGE_SIZE ; i++) {struct page *page;
drivers/dma-buf/heaps/cma_heap.c +3 -3
@@ -160,13 +160,13 @@ static int cma_heap_dma_buf_end_cpu_access(struct dma_buf *dmabuf,static vm_fault_t cma_heap_vm_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct cma_heap_buffer *buffer = vma->vm_private_data; -if (vmf->pgoff > buffer->pagecount) +if (vmf->info.pgoff > buffer->pagecount)return VM_FAULT_SIGBUS ; - vmf->page = buffer->pages[vmf->pgoff]; + vmf->page = buffer->pages[vmf->info.pgoff];get_page (vmf->page);return 0 ;
drivers/dma-buf/udmabuf.c +2 -2
@@ -24,10 +24,10 @@ struct udmabuf {static vm_fault_t udmabuf_vm_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct udmabuf *ubuf = vma->vm_private_data; - vmf->page = ubuf->pages[vmf->pgoff]; + vmf->page = ubuf->pages[vmf->info.pgoff];get_page (vmf->page);return 0 ; }
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2 -2
@@ -1983,7 +1983,7 @@ void amdgpu_ttm_set_buffer_funcs_status(struct amdgpu_device *adev, bool enable)static vm_fault_t amdgpu_ttm_fault (struct vm_fault *vmf) { -struct ttm_buffer_object *bo = vmf->vma->vm_private_data; +struct ttm_buffer_object *bo = vmf->info.vma->vm_private_data;vm_fault_t ret; ret =ttm_bo_vm_reserve (bo, vmf);
@@ -1994,7 +1994,7 @@ static vm_fault_t amdgpu_ttm_fault(struct vm_fault *vmf)if (ret) goto unlock; - ret =ttm_bo_vm_fault_reserved (vmf, vmf->vma->vm_page_prot, + ret =ttm_bo_vm_fault_reserved (vmf, vmf->info.vma->vm_page_prot,TTM_BO_VM_NUM_PREFAULT ,1 );if (ret ==VM_FAULT_RETRY && !(vmf->flags &FAULT_FLAG_RETRY_NOWAIT ))return ret;
drivers/gpu/drm/armada/armada_gem.c +3 -3
@@ -17,12 +17,12 @@static vm_fault_t armada_gem_vm_fault (struct vm_fault *vmf) { -struct drm_gem_object *gobj = vmf->vma->vm_private_data; +struct drm_gem_object *gobj = vmf->info.vma->vm_private_data;struct armada_gem_object *obj =drm_to_armada_gem (gobj);unsigned long pfn = obj->phys_addr >>PAGE_SHIFT ; - pfn += (vmf->address - vmf->vma->vm_start) >>PAGE_SHIFT ; -return vmf_insert_pfn (vmf->vma, vmf->address, pfn); + pfn += (vmf->info.address - vmf->info.vma->vm_start) >>PAGE_SHIFT ; +return vmf_insert_pfn (vmf->info.vma, vmf->info.address, pfn); }static const struct vm_operations_struct armada_gem_vm_ops = {
drivers/gpu/drm/drm_gem_shmem_helper.c +4 -4
@@ -521,18 +521,18 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_dumb_create);static vm_fault_t drm_gem_shmem_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_gem_object *obj = vma->vm_private_data;struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj);loff_t num_pages = obj->size >>PAGE_SHIFT ;struct page *page; -if (vmf->pgoff >= num_pages ||WARN_ON_ONCE (!shmem->pages)) +if (vmf->info.pgoff >= num_pages ||WARN_ON_ONCE (!shmem->pages))return VM_FAULT_SIGBUS ; - page = shmem->pages[vmf->pgoff]; + page = shmem->pages[vmf->info.pgoff]; -return vmf_insert_page (vma, vmf->address, page); +return vmf_insert_page (vma, vmf->info.address, page); }static void drm_gem_shmem_vm_open (struct vm_area_struct *vma)
drivers/gpu/drm/drm_vm.c +9 -9
@@ -111,7 +111,7 @@ static pgprot_t drm_dma_prot(uint32_t map_type, struct vm_area_struct *vma)#if IS_ENABLED (CONFIG_AGP )static vm_fault_t drm_vm_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_file *priv = vma->vm_file->private_data;struct drm_device *dev = priv->minor->dev;struct drm_local_map *map =NULL ;
@@ -138,7 +138,7 @@ static vm_fault_t drm_vm_fault(struct vm_fault *vmf)* Using vm_pgoff as a selector forces us to use this unusual * addressing scheme. */ -resource_size_t offset = vmf->address - vma->vm_start; +resource_size_t offset = vmf->info.address - vma->vm_start;resource_size_t baddr = map->offset + offset;struct drm_agp_mem *agpmem;struct page *page;
@@ -200,7 +200,7 @@ static vm_fault_t drm_vm_fault(struct vm_fault *vmf)*/ static vm_fault_t drm_vm_shm_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_local_map *map = vma->vm_private_data;unsigned long offset;unsigned long i;
@@ -209,7 +209,7 @@ static vm_fault_t drm_vm_shm_fault(struct vm_fault *vmf)if (!map)return VM_FAULT_SIGBUS ;/* Nothing allocated */ - offset = vmf->address - vma->vm_start; + offset = vmf->info.address - vma->vm_start; i = (unsigned long )map->handle + offset; page =vmalloc_to_page ((void *)i);if (!page)
@@ -300,7 +300,7 @@ static void drm_vm_shm_close(struct vm_area_struct *vma)*/ static vm_fault_t drm_vm_dma_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_file *priv = vma->vm_file->private_data;struct drm_device *dev = priv->minor->dev;struct drm_device_dma *dma = dev->dma;
@@ -313,9 +313,9 @@ static vm_fault_t drm_vm_dma_fault(struct vm_fault *vmf)if (!dma->pagelist)return VM_FAULT_SIGBUS ;/* Nothing allocated */ - offset = vmf->address - vma->vm_start; + offset = vmf->info.address - vma->vm_start;/* vm_[pg]off[set] should be 0 */ - page_nr = offset >>PAGE_SHIFT ;/* page_nr could just be vmf->pgoff */ + page_nr = offset >>PAGE_SHIFT ;/* page_nr could just be vmf->info.pgoff */ page =virt_to_page ((void *)dma->pagelist[page_nr]);get_page (page);
@@ -335,7 +335,7 @@ static vm_fault_t drm_vm_dma_fault(struct vm_fault *vmf)*/ static vm_fault_t drm_vm_sg_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_local_map *map = vma->vm_private_data;struct drm_file *priv = vma->vm_file->private_data;struct drm_device *dev = priv->minor->dev;
@@ -350,7 +350,7 @@ static vm_fault_t drm_vm_sg_fault(struct vm_fault *vmf)if (!entry->pagelist)return VM_FAULT_SIGBUS ;/* Nothing allocated */ - offset = vmf->address - vma->vm_start; + offset = vmf->info.address - vma->vm_start; map_offset = map->offset - (unsigned long )dev->sg->virtual; page_offset = (offset >>PAGE_SHIFT ) + (map_offset >>PAGE_SHIFT ); page = entry->pagelist[page_offset];
drivers/gpu/drm/etnaviv/etnaviv_gem.c +5 -5
@@ -171,7 +171,7 @@ int etnaviv_gem_mmap(struct file *filp, struct vm_area_struct *vma)static vm_fault_t etnaviv_gem_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_gem_object *obj = vma->vm_private_data;struct etnaviv_gem_object *etnaviv_obj =to_etnaviv_bo (obj);struct page **pages, *page;
@@ -195,15 +195,15 @@ static vm_fault_t etnaviv_gem_fault(struct vm_fault *vmf)return vmf_error (err); } -/* We don't use vmf->pgoff since that has the fake offset: */ - pgoff = (vmf->address - vma->vm_start) >>PAGE_SHIFT ; +/* We don't use vmf->info.pgoff since that has the fake offset: */ + pgoff = (vmf->info.address - vma->vm_start) >>PAGE_SHIFT ; page = pages[pgoff]; -VERB ("Inserting %p pfn %lx, pa %lx" , (void *)vmf->address, +VERB ("Inserting %p pfn %lx, pa %lx" , (void *) vmf->info.address,page_to_pfn (page),page_to_pfn (page) <<PAGE_SHIFT ); -return vmf_insert_page (vma, vmf->address, page); +return vmf_insert_page (vma, vmf->info.address, page); }int etnaviv_gem_mmap_offset (struct drm_gem_object *obj,u64 *offset)
drivers/gpu/drm/gma500/framebuffer.c +2 -2
@@ -78,7 +78,7 @@ static int psbfb_setcolreg(unsigned regno, unsigned red, unsigned green,static vm_fault_t psbfb_vm_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_framebuffer *fb = vma->vm_private_data;struct drm_device *dev = fb->dev;struct drm_psb_private *dev_priv = dev->dev_private;
@@ -92,7 +92,7 @@ static vm_fault_t psbfb_vm_fault(struct vm_fault *vmf) gtt->offset; page_num =vma_pages (vma); - address = vmf->address - (vmf->pgoff <<PAGE_SHIFT ); + address = vmf->info.address - (vmf->info.pgoff <<PAGE_SHIFT ); vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot);
drivers/gpu/drm/gma500/gem.c +4 -4
@@ -133,7 +133,7 @@ int psb_gem_dumb_create(struct drm_file *file, struct drm_device *dev,*/ static vm_fault_t psb_gem_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_gem_object *obj;struct gtt_range *r;int err;
@@ -166,15 +166,15 @@ static vm_fault_t psb_gem_fault(struct vm_fault *vmf) }/* Page relative to the VMA start - we must calculate this ourselves -because vmf->pgoff is the fake GEM offset */ - page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ; +because vmf->info.pgoff is the fake GEM offset */ + page_offset = (vmf->info.address - vma->vm_start) >>PAGE_SHIFT ;/* CPU view of the page, don't go via the GART for CPU writes */ if (r->stolen) pfn = (dev_priv->stolen_base + r->offset) >>PAGE_SHIFT ;else pfn =page_to_pfn (r->pages[page_offset]); - ret =vmf_insert_pfn (vma, vmf->address, pfn); + ret =vmf_insert_pfn (vma, vmf->info.address, pfn); fail:mutex_unlock (&dev_priv->mmap_mutex);
drivers/gpu/drm/i915/gem/i915_gem_mman.c +4 -4
@@ -235,7 +235,7 @@ static vm_fault_t i915_error_to_vmf_fault(int err)static vm_fault_t vm_fault_cpu (struct vm_fault *vmf) { -struct vm_area_struct *area = vmf->vma; +struct vm_area_struct *area = vmf->info.vma;struct i915_mmap_offset *mmo = area->vm_private_data;struct drm_i915_gem_object *obj = mmo->obj;resource_size_t iomap;
@@ -275,7 +275,7 @@ static vm_fault_t vm_fault_cpu(struct vm_fault *vmf)static vm_fault_t vm_fault_gtt (struct vm_fault *vmf) {#define MIN_CHUNK_PAGES (SZ_1M >> PAGE_SHIFT) -struct vm_area_struct *area = vmf->vma; +struct vm_area_struct *area = vmf->info.vma;struct i915_mmap_offset *mmo = area->vm_private_data;struct drm_i915_gem_object *obj = mmo->obj;struct drm_device *dev = obj->base.dev;
@@ -290,8 +290,8 @@ static vm_fault_t vm_fault_gtt(struct vm_fault *vmf)int srcu;int ret; -/* We don't use vmf->pgoff since that has the fake offset */ - page_offset = (vmf->address - area->vm_start) >>PAGE_SHIFT ; +/* We don't use vmf->info.pgoff since that has the fake offset */ + page_offset = (vmf->info.address - area->vm_start) >>PAGE_SHIFT ;trace_i915_gem_object_fault (obj, page_offset, true, write);
drivers/gpu/drm/msm/msm_gem.c +6 -5
@@ -237,7 +237,7 @@ int msm_gem_mmap(struct file *filp, struct vm_area_struct *vma)static vm_fault_t msm_gem_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_gem_object *obj = vma->vm_private_data;struct msm_gem_object *msm_obj =to_msm_bo (obj);struct page **pages;
@@ -268,15 +268,16 @@ static vm_fault_t msm_gem_fault(struct vm_fault *vmf) goto out_unlock; } -/* We don't use vmf->pgoff since that has the fake offset: */ - pgoff = (vmf->address - vma->vm_start) >>PAGE_SHIFT ; +/* We don't use vmf->info.pgoff since that has the fake offset: */ + pgoff = (vmf->info.address - vma->vm_start) >>PAGE_SHIFT ; pfn =page_to_pfn (pages[pgoff]); -VERB ("Inserting %p pfn %lx, pa %lx" , (void *)vmf->address, +VERB ("Inserting %p pfn %lx, pa %lx" , (void *) vmf->info.address, pfn, pfn <<PAGE_SHIFT ); - ret =vmf_insert_mixed (vma, vmf->address,__pfn_to_pfn_t (pfn,PFN_DEV )); + ret =vmf_insert_mixed (vma, vmf->info.address, +__pfn_to_pfn_t (pfn,PFN_DEV )); out_unlock:msm_gem_unlock (obj); out:
drivers/gpu/drm/nouveau/nouveau_dmem.c +4 -4
@@ -149,7 +149,7 @@ static vm_fault_t nouveau_dmem_fault_copy_one(struct nouveau_drm *drm,if (!spage || !(args->src[0 ] &MIGRATE_PFN_MIGRATE ))return 0 ; - dpage =alloc_page_vma (GFP_HIGHUSER , vmf->vma, vmf->address); + dpage =alloc_page_vma (GFP_HIGHUSER , vmf->info.vma, vmf->info.address);if (!dpage)return VM_FAULT_SIGBUS ;lock_page (dpage);
@@ -186,9 +186,9 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)dma_addr_t dma_addr =0 ;vm_fault_t ret;struct migrate_vma args = { - .vma = vmf->vma, - .start = vmf->address, - .end = vmf->address +PAGE_SIZE , + .vma = vmf->info.vma, + .start = vmf->info.address, + .end = vmf->info.address +PAGE_SIZE , .src = &src, .dst = &dst, .pgmap_owner = drm->dev,
drivers/gpu/drm/nouveau/nouveau_ttm.c +1 -1
@@ -129,7 +129,7 @@ const struct ttm_resource_manager_func nv04_gart_manager = {static vm_fault_t nouveau_ttm_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct ttm_buffer_object *bo = vma->vm_private_data;pgprot_t prot;vm_fault_t ret;
drivers/gpu/drm/omapdrm/omap_gem.c +10 -10
@@ -350,8 +350,8 @@ static vm_fault_t omap_gem_fault_1d(struct drm_gem_object *obj,unsigned long pfn;pgoff_t pgoff; -/* We don't use vmf->pgoff since that has the fake offset: */ - pgoff = (vmf->address - vma->vm_start) >>PAGE_SHIFT ; +/* We don't use vmf->info.pgoff since that has the fake offset: */ + pgoff = (vmf->info.address - vma->vm_start) >>PAGE_SHIFT ;if (omap_obj->pages) {omap_gem_cpu_sync_page (obj, pgoff);
@@ -361,11 +361,11 @@ static vm_fault_t omap_gem_fault_1d(struct drm_gem_object *obj, pfn = (omap_obj->dma_addr >>PAGE_SHIFT ) + pgoff; } -VERB ("Inserting %p pfn %lx, pa %lx" , (void *)vmf->address, +VERB ("Inserting %p pfn %lx, pa %lx" , (void *) vmf->info.address, pfn, pfn <<PAGE_SHIFT ); -return vmf_insert_mixed (vma, vmf->address, -__pfn_to_pfn_t (pfn,PFN_DEV )); +return vmf_insert_mixed (vma, vmf->info.address, +__pfn_to_pfn_t (pfn,PFN_DEV )); }/* Special handling for the case of faulting in 2d tiled buffers */
@@ -399,8 +399,8 @@ static vm_fault_t omap_gem_fault_2d(struct drm_gem_object *obj,*/ const int m =DIV_ROUND_UP (omap_obj->width << fmt,PAGE_SIZE ); -/* We don't use vmf->pgoff since that has the fake offset: */ - pgoff = (vmf->address - vma->vm_start) >>PAGE_SHIFT ; +/* We don't use vmf->info.pgoff since that has the fake offset: */ + pgoff = (vmf->info.address - vma->vm_start) >>PAGE_SHIFT ;/* * Actual address we start mapping at is rounded down to previous slot
@@ -411,7 +411,7 @@ static vm_fault_t omap_gem_fault_2d(struct drm_gem_object *obj,/* figure out buffer width in slots */ slots = omap_obj->width >> priv->usergart[fmt].slot_shift; - vaddr = vmf->address - ((pgoff - base_pgoff) <<PAGE_SHIFT ); + vaddr = vmf->info.address - ((pgoff - base_pgoff) <<PAGE_SHIFT ); entry = &priv->usergart[fmt].entry[priv->usergart[fmt].last];
@@ -456,7 +456,7 @@ static vm_fault_t omap_gem_fault_2d(struct drm_gem_object *obj, pfn = entry->dma_addr >>PAGE_SHIFT ; -VERB ("Inserting %p pfn %lx, pa %lx" , (void *)vmf->address, +VERB ("Inserting %p pfn %lx, pa %lx" , (void *) vmf->info.address, pfn, pfn <<PAGE_SHIFT );for (i = n; i >0 ; i--) {
@@ -489,7 +489,7 @@ static vm_fault_t omap_gem_fault_2d(struct drm_gem_object *obj,*/ static vm_fault_t omap_gem_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_gem_object *obj = vma->vm_private_data;struct omap_gem_object *omap_obj =to_omap_bo (obj);int err;
drivers/gpu/drm/radeon/radeon_ttm.c +2 -2
@@ -811,7 +811,7 @@ void radeon_ttm_set_active_vram_size(struct radeon_device *rdev, u64 size)static vm_fault_t radeon_ttm_fault (struct vm_fault *vmf) { -struct ttm_buffer_object *bo = vmf->vma->vm_private_data; +struct ttm_buffer_object *bo = vmf->info.vma->vm_private_data;struct radeon_device *rdev =radeon_get_rdev (bo->bdev);vm_fault_t ret;
@@ -825,7 +825,7 @@ static vm_fault_t radeon_ttm_fault(struct vm_fault *vmf)if (ret) goto unlock_resv; - ret =ttm_bo_vm_fault_reserved (vmf, vmf->vma->vm_page_prot, + ret =ttm_bo_vm_fault_reserved (vmf, vmf->info.vma->vm_page_prot,TTM_BO_VM_NUM_PREFAULT ,1 );if (ret ==VM_FAULT_RETRY && !(vmf->flags &FAULT_FLAG_RETRY_NOWAIT )) goto unlock_mclk;
drivers/gpu/drm/tegra/gem.c +3 -3
@@ -486,7 +486,7 @@ int tegra_bo_dumb_create(struct drm_file *file, struct drm_device *drm,static vm_fault_t tegra_bo_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_gem_object *gem = vma->vm_private_data;struct tegra_bo *bo =to_tegra_bo (gem);struct page *page;
@@ -495,10 +495,10 @@ static vm_fault_t tegra_bo_fault(struct vm_fault *vmf)if (!bo->pages)return VM_FAULT_SIGBUS ; - offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ; + offset = (vmf->info.address - vma->vm_start) >>PAGE_SHIFT ; page = bo->pages[offset]; -return vmf_insert_page (vma, vmf->address, page); +return vmf_insert_page (vma, vmf->info.address, page); }const struct vm_operations_struct tegra_bo_vm_ops = {
drivers/gpu/drm/ttm/ttm_bo_vm.c +5 -5
@@ -68,7 +68,7 @@ static vm_fault_t ttm_bo_vm_fault_idle(struct ttm_buffer_object *bo, goto out_unlock;ttm_bo_get (bo); -mmap_read_unlock (vmf->vma->vm_mm); +mmap_read_unlock (vmf->info.vma->vm_mm); (void )dma_fence_wait (bo->moving, true);dma_resv_unlock (bo->base.resv);ttm_bo_put (bo);
@@ -143,7 +143,7 @@ vm_fault_t ttm_bo_vm_reserve(struct ttm_buffer_object *bo,if (fault_flag_allow_retry_first (vmf->flags)) {if (!(vmf->flags &FAULT_FLAG_RETRY_NOWAIT )) {ttm_bo_get (bo); -mmap_read_unlock (vmf->vma->vm_mm); +mmap_read_unlock (vmf->info.vma->vm_mm);if (!dma_resv_lock_interruptible (bo->base.resv,NULL ))dma_resv_unlock (bo->base.resv);
@@ -277,7 +277,7 @@ vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf, pgoff_t num_prefault, pgoff_t fault_page_size) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct ttm_buffer_object *bo = vma->vm_private_data;struct ttm_bo_device *bdev = bo->bdev;unsigned long page_offset;
@@ -288,7 +288,7 @@ vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf,int err;pgoff_t i;vm_fault_t ret =VM_FAULT_NOPAGE ; -unsigned long address = vmf->address; +unsigned long address = vmf->info.address;/* * Wait for buffer data in transit, due to a pipelined
@@ -383,7 +383,7 @@ EXPORT_SYMBOL(ttm_bo_vm_fault_reserved);vm_fault_t ttm_bo_vm_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;pgprot_t prot;struct ttm_buffer_object *bo = vma->vm_private_data;vm_fault_t ret;
drivers/gpu/drm/vc4/vc4_bo.c +1 -1
@@ -704,7 +704,7 @@ struct dma_buf * vc4_prime_export(struct drm_gem_object *obj, int flags)static vm_fault_t vc4_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_gem_object *obj = vma->vm_private_data;struct vc4_bo *bo =to_vc4_bo (obj);
drivers/gpu/drm/vgem/vgem_drv.c +3 -3
@@ -73,10 +73,10 @@ static void vgem_gem_free_object(struct drm_gem_object *obj)static vm_fault_t vgem_gem_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct drm_vgem_gem_object *obj = vma->vm_private_data; -/* We don't use vmf->pgoff since that has the fake offset */ -unsigned long vaddr = vmf->address; +/* We don't use vmf->info.pgoff since that has the fake offset */ +unsigned long vaddr = vmf->info.address;vm_fault_t ret =VM_FAULT_SIGBUS ;loff_t num_pages;pgoff_t page_offset;
drivers/gpu/drm/vmwgfx/vmwgfx_page_dirty.c +6 -6
@@ -392,7 +392,7 @@ void vmw_bo_dirty_clear_res(struct vmw_resource *res)vm_fault_t vmw_bo_vm_mkwrite (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct ttm_buffer_object *bo = (struct ttm_buffer_object *) vma->vm_private_data;vm_fault_t ret;
@@ -412,7 +412,7 @@ vm_fault_t vmw_bo_vm_mkwrite(struct vm_fault *vmf)if (ret)return ret; - page_offset = vmf->pgoff -drm_vma_node_start (&bo->base.vma_node); + page_offset = vmf->info.pgoff -drm_vma_node_start (&bo->base.vma_node);if (unlikely (page_offset >= bo->num_pages)) { ret =VM_FAULT_SIGBUS ; goto out_unlock;
@@ -434,7 +434,7 @@ vm_fault_t vmw_bo_vm_mkwrite(struct vm_fault *vmf)vm_fault_t vmw_bo_vm_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct ttm_buffer_object *bo = (struct ttm_buffer_object *) vma->vm_private_data; struct vmw_buffer_object *vbo =
@@ -454,7 +454,7 @@ vm_fault_t vmw_bo_vm_fault(struct vm_fault *vmf)pgoff_t allowed_prefault;unsigned long page_offset; - page_offset = vmf->pgoff - + page_offset = vmf->info.pgoff -drm_vma_node_start (&bo->base.vma_node);if (page_offset >= bo->num_pages || vmw_resources_clean(vbo, page_offset,
@@ -491,7 +491,7 @@ vm_fault_t vmw_bo_vm_fault(struct vm_fault *vmf)vm_fault_t vmw_bo_vm_huge_fault (struct vm_fault *vmf,enum page_entry_size pe_size) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct ttm_buffer_object *bo = (struct ttm_buffer_object *) vma->vm_private_data; struct vmw_buffer_object *vbo =
@@ -529,7 +529,7 @@ vm_fault_t vmw_bo_vm_huge_fault(struct vm_fault *vmf,pgoff_t allowed_prefault;unsigned long page_offset; - page_offset = vmf->pgoff - + page_offset = vmf->info.pgoff -drm_vma_node_start (&bo->base.vma_node);if (page_offset >= bo->num_pages || vmw_resources_clean(vbo, page_offset,
drivers/hsi/clients/cmt_speech.c +1 -1
@@ -1086,7 +1086,7 @@ static void cs_hsi_stop(struct cs_hsi_iface *hi)static vm_fault_t cs_char_vma_fault (struct vm_fault *vmf) { -struct cs_char *csdata = vmf->vma->vm_private_data; +struct cs_char *csdata = vmf->info.vma->vm_private_data;struct page *page; page =virt_to_page (csdata->mmap_base);
drivers/hwtracing/intel_th/msu.c +4 -4
@@ -1583,16 +1583,16 @@ static void msc_mmap_close(struct vm_area_struct *vma)static vm_fault_t msc_mmap_fault (struct vm_fault *vmf) { -struct msc_iter *iter = vmf->vma->vm_file->private_data; +struct msc_iter *iter = vmf->info.vma->vm_file->private_data;struct msc *msc = iter->msc; - vmf->page =msc_buffer_get_page (msc, vmf->pgoff); + vmf->page =msc_buffer_get_page (msc, vmf->info.pgoff);if (!vmf->page)return VM_FAULT_SIGBUS ;get_page (vmf->page); - vmf->page->mapping = vmf->vma->vm_file->f_mapping; - vmf->page->index = vmf->pgoff; + vmf->page->mapping = vmf->info.vma->vm_file->f_mapping; + vmf->page->index = vmf->info.pgoff;return 0 ; }
drivers/infiniband/core/uverbs_main.c +5 -5
@@ -768,16 +768,16 @@ static void rdma_umap_close(struct vm_area_struct *vma)*/ static vm_fault_t rdma_umap_fault (struct vm_fault *vmf) { -struct ib_uverbs_file *ufile = vmf->vma->vm_file->private_data; -struct rdma_umap_priv *priv = vmf->vma->vm_private_data; +struct ib_uverbs_file *ufile = vmf->info.vma->vm_file->private_data; +struct rdma_umap_priv *priv = vmf->info.vma->vm_private_data;vm_fault_t ret =0 ;if (!priv)return VM_FAULT_SIGBUS ;/* Read only pages can just use the system zero page. */ -if (!(vmf->vma->vm_flags & (VM_WRITE |VM_MAYWRITE ))) { - vmf->page =ZERO_PAGE (vmf->address); +if (!(vmf->info.vma->vm_flags & (VM_WRITE |VM_MAYWRITE ))) { + vmf->page =ZERO_PAGE (vmf->info.address);get_page (vmf->page);return 0 ; }
@@ -785,7 +785,7 @@ static vm_fault_t rdma_umap_fault(struct vm_fault *vmf)mutex_lock (&ufile->umap_lock);if (!ufile->disassociate_page) ufile->disassociate_page = -alloc_pages (vmf->gfp_mask | __GFP_ZERO,0 ); +alloc_pages (vmf->info.gfp_mask | __GFP_ZERO,0 );if (ufile->disassociate_page) {/*
drivers/infiniband/hw/hfi1/file_ops.c +1 -1
@@ -604,7 +604,7 @@ static vm_fault_t vma_fault(struct vm_fault *vmf) {struct page *page; - page =vmalloc_to_page ((void *)(vmf->pgoff <<PAGE_SHIFT )); + page =vmalloc_to_page ((void *)(vmf->info.pgoff <<PAGE_SHIFT ));if (!page)return VM_FAULT_SIGBUS ;
drivers/infiniband/hw/qib/qib_file_ops.c +1 -1
@@ -876,7 +876,7 @@ static vm_fault_t qib_file_vma_fault(struct vm_fault *vmf) {struct page *page; - page =vmalloc_to_page ((void *)(vmf->pgoff <<PAGE_SHIFT )); + page =vmalloc_to_page ((void *)(vmf->info.pgoff <<PAGE_SHIFT ));if (!page)return VM_FAULT_SIGBUS ;
drivers/media/v4l2-core/videobuf-dma-sg.c +3 -3
@@ -434,16 +434,16 @@ static void videobuf_vm_close(struct vm_area_struct *vma)*/ static vm_fault_t videobuf_vm_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct page *page;dprintk (3 ,"fault: fault @ %08lx [vma %08lx-%08lx]\n" , - vmf->address, vma->vm_start, vma->vm_end); + vmf->info.address, vma->vm_start, vma->vm_end); page =alloc_page (GFP_USER | __GFP_DMA32);if (!page)return VM_FAULT_OOM ; -clear_user_highpage (page, vmf->address); +clear_user_highpage (page, vmf->info.address); vmf->page = page;return 0 ;
drivers/misc/cxl/context.c +5 -4
@@ -125,15 +125,15 @@ void cxl_context_set_mapping(struct cxl_context *ctx,static vm_fault_t cxl_mmap_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct cxl_context *ctx = vma->vm_file->private_data;u64 area, offset;vm_fault_t ret; - offset = vmf->pgoff <<PAGE_SHIFT ; + offset = vmf->info.pgoff <<PAGE_SHIFT ;pr_devel ("%s: pe: %i address: 0x%lx offset: 0x%llx\n" , - __func__, ctx->pe, vmf->address, offset); + __func__, ctx->pe, vmf->info.address, offset);if (ctx->afu->current_mode ==CXL_MODE_DEDICATED ) { area = ctx->afu->psn_phys;
@@ -165,7 +165,8 @@ static vm_fault_t cxl_mmap_fault(struct vm_fault *vmf)return VM_FAULT_SIGBUS ; } - ret =vmf_insert_pfn (vma, vmf->address, (area + offset) >>PAGE_SHIFT ); + ret =vmf_insert_pfn (vma, vmf->info.address, + (area + offset) >>PAGE_SHIFT );mutex_unlock (&ctx->status_mutex);
drivers/misc/ocxl/context.c +5 -5
@@ -138,19 +138,19 @@ static vm_fault_t map_pp_mmio(struct vm_area_struct *vma, unsigned long address,static vm_fault_t ocxl_mmap_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct ocxl_context *ctx = vma->vm_file->private_data;u64 offset;vm_fault_t ret; - offset = vmf->pgoff <<PAGE_SHIFT ; + offset = vmf->info.pgoff <<PAGE_SHIFT ;pr_debug ("%s: pasid %d address 0x%lx offset 0x%llx\n" , __func__, - ctx->pasid, vmf->address, offset); + ctx->pasid, vmf->info.address, offset);if (offset < ctx->afu->irq_base_offset) - ret =map_pp_mmio (vma, vmf->address, offset, ctx); + ret =map_pp_mmio (vma, vmf->info.address, offset, ctx);else - ret =map_afu_irq (vma, vmf->address, offset, ctx); + ret =map_afu_irq (vma, vmf->info.address, offset, ctx);return ret; }
drivers/misc/ocxl/sysfs.c +4 -4
@@ -108,16 +108,16 @@ static ssize_t global_mmio_read(struct file *filp, struct kobject *kobj,static vm_fault_t global_mmio_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct ocxl_afu *afu = vma->vm_private_data;unsigned long offset; -if (vmf->pgoff >= (afu->config.global_mmio_size >>PAGE_SHIFT )) +if (vmf->info.pgoff >= (afu->config.global_mmio_size >>PAGE_SHIFT ))return VM_FAULT_SIGBUS ; - offset = vmf->pgoff; + offset = vmf->info.pgoff; offset += (afu->global_mmio_start >>PAGE_SHIFT ); -return vmf_insert_pfn (vma, vmf->address, offset); +return vmf_insert_pfn (vma, vmf->info.address, offset); }static const struct vm_operations_struct global_mmio_vmops = {
drivers/misc/sgi-gru/grumain.c +2 -2
@@ -915,12 +915,12 @@ struct gru_state *gru_assign_gru_context(struct gru_thread_state *gts)*/ vm_fault_t gru_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct gru_thread_state *gts;unsigned long paddr, vaddr;unsigned long expires; - vaddr = vmf->address; + vaddr = vmf->info.address;gru_dbg (grudev,"vma %p, vaddr 0x%lx (0x%lx)\n" , vma, vaddr,GSEG_BASE (vaddr));STAT (nopfn);
drivers/scsi/cxlflash/ocxl_hw.c +3 -3
@@ -1123,12 +1123,12 @@ static int afu_release(struct inode *inode, struct file *file)*/ static vm_fault_t ocxlflash_mmap_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct ocxlflash_context *ctx = vma->vm_file->private_data;struct device *dev = ctx->hw_afu->dev;u64 mmio_area, offset; - offset = vmf->pgoff <<PAGE_SHIFT ; + offset = vmf->info.pgoff <<PAGE_SHIFT ;if (offset >= ctx->psn_size)return VM_FAULT_SIGBUS ;
@@ -1144,7 +1144,7 @@ static vm_fault_t ocxlflash_mmap_fault(struct vm_fault *vmf) mmio_area = ctx->psn_phys; mmio_area += offset; -return vmf_insert_pfn (vma, vmf->address, mmio_area >>PAGE_SHIFT ); +return vmf_insert_pfn (vma, vmf->info.address, mmio_area >>PAGE_SHIFT ); }static const struct vm_operations_struct ocxlflash_vmops = {
drivers/scsi/cxlflash/superpipe.c +1 -1
@@ -1101,7 +1101,7 @@ static struct page *get_err_page(struct cxlflash_cfg *cfg)*/ static vm_fault_t cxlflash_mmap_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct file *file = vma->vm_file; struct cxlflash_cfg *cfg = container_of(file->f_op, struct cxlflash_cfg, cxl_fops);
drivers/scsi/sg.c +2 -2
@@ -1241,7 +1241,7 @@ sg_fasync(int fd, struct file *filp, int mode)static vm_fault_t sg_vma_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;Sg_fd *sfp;unsigned long offset, len, sa;Sg_scatter_hold *rsv_schp;
@@ -1250,7 +1250,7 @@ sg_vma_fault(struct vm_fault *vmf)if ((NULL == vma) || (!(sfp = (Sg_fd *) vma->vm_private_data)))return VM_FAULT_SIGBUS ; rsv_schp = &sfp->reserve; - offset = vmf->pgoff <<PAGE_SHIFT ; + offset = vmf->info.pgoff <<PAGE_SHIFT ;if (offset >= rsv_schp->bufflen)return VM_FAULT_SIGBUS ; SCSI_LOG_TIMEOUT(3, sg_printk(KERN_INFO, sfp->parentdp,
drivers/target/target_core_user.c +3 -3
@@ -1676,13 +1676,13 @@ static struct page *tcmu_try_get_block_page(struct tcmu_dev *udev, uint32_t dbi)static vm_fault_t tcmu_vma_fault (struct vm_fault *vmf) { -struct tcmu_dev *udev = vmf->vma->vm_private_data; +struct tcmu_dev *udev = vmf->info.vma->vm_private_data;struct uio_info *info = &udev->uio_info;struct page *page;unsigned long offset;void *addr; -int mi =tcmu_find_mem_index (vmf->vma); +int mi =tcmu_find_mem_index (vmf->info.vma);if (mi <0 )return VM_FAULT_SIGBUS ;
@@ -1690,7 +1690,7 @@ static vm_fault_t tcmu_vma_fault(struct vm_fault *vmf)* We need to subtract mi because userspace uses offset = N*PAGE_SIZE * to use mem[N]. */ - offset = (vmf->pgoff - mi) <<PAGE_SHIFT ; + offset = (vmf->info.pgoff - mi) <<PAGE_SHIFT ;if (offset < udev->data_off) {/* For the vmalloc()ed cmd area pages */
drivers/uio/uio.c +3 -3
@@ -666,7 +666,7 @@ static int uio_find_mem_index(struct vm_area_struct *vma)static vm_fault_t uio_vma_fault (struct vm_fault *vmf) { -struct uio_device *idev = vmf->vma->vm_private_data; +struct uio_device *idev = vmf->info.vma->vm_private_data;struct page *page;unsigned long offset;void *addr;
@@ -679,7 +679,7 @@ static vm_fault_t uio_vma_fault(struct vm_fault *vmf) goto out; } - mi =uio_find_mem_index (vmf->vma); + mi =uio_find_mem_index (vmf->info.vma);if (mi <0 ) { ret =VM_FAULT_SIGBUS ; goto out;
@@ -689,7 +689,7 @@ static vm_fault_t uio_vma_fault(struct vm_fault *vmf)* We need to subtract mi because userspace uses offset = N*PAGE_SIZE * to use mem[N]. */ - offset = (vmf->pgoff - mi) <<PAGE_SHIFT ; + offset = (vmf->info.pgoff - mi) <<PAGE_SHIFT ; addr = (void *)(unsigned long )idev->info->mem[mi].addr + offset;if (idev->info->mem[mi].memtype ==UIO_MEM_LOGICAL )
drivers/usb/mon/mon_bin.c +2 -2
@@ -1244,11 +1244,11 @@ static void mon_bin_vma_close(struct vm_area_struct *vma)*/ static vm_fault_t mon_bin_vma_fault (struct vm_fault *vmf) { -struct mon_reader_bin *rp = vmf->vma->vm_private_data; +struct mon_reader_bin *rp = vmf->info.vma->vm_private_data;unsigned long offset, chunk_idx;struct page *pageptr; - offset = vmf->pgoff <<PAGE_SHIFT ; + offset = vmf->info.pgoff <<PAGE_SHIFT ;if (offset >= rp->b_size)return VM_FAULT_SIGBUS ; chunk_idx = offset /CHUNK_SIZE ;
drivers/vfio/pci/vfio_pci.c +1 -1
@@ -1612,7 +1612,7 @@ static void vfio_pci_mmap_close(struct vm_area_struct *vma)static vm_fault_t vfio_pci_mmap_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct vfio_pci_device *vdev = vma->vm_private_data;vm_fault_t ret =VM_FAULT_NOPAGE ;
drivers/vfio/pci/vfio_pci_nvlink2.c +4 -4
@@ -116,18 +116,18 @@ static void vfio_pci_nvgpu_release(struct vfio_pci_device *vdev,static vm_fault_t vfio_pci_nvgpu_mmap_fault (struct vm_fault *vmf) {vm_fault_t ret; -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct vfio_pci_region *region = vma->vm_private_data;struct vfio_pci_nvgpu_data *data = region->data; -unsigned long vmf_off = (vmf->address - vma->vm_start) >>PAGE_SHIFT ; +unsigned long vmf_off = (vmf->info.address - vma->vm_start) >>PAGE_SHIFT ;unsigned long nv2pg = data->gpu_hpa >>PAGE_SHIFT ;unsigned long vm_pgoff = vma->vm_pgoff & ((1U << (VFIO_PCI_OFFSET_SHIFT -PAGE_SHIFT )) -1 );unsigned long pfn = nv2pg + vm_pgoff + vmf_off; - ret =vmf_insert_pfn (vma, vmf->address, pfn); + ret =vmf_insert_pfn (vma, vmf->info.address, pfn);trace_vfio_pci_nvgpu_mmap_fault (data->gpdev, pfn <<PAGE_SHIFT , - vmf->address, ret); + vmf->info.address, ret);return ret; }
drivers/vhost/vdpa.c +3 -3
@@ -936,17 +936,17 @@ static int vhost_vdpa_release(struct inode *inode, struct file *filep)#ifdef CONFIG_MMU static vm_fault_t vhost_vdpa_fault (struct vm_fault *vmf) { -struct vhost_vdpa *v = vmf->vma->vm_file->private_data; +struct vhost_vdpa *v = vmf->info.vma->vm_file->private_data;struct vdpa_device *vdpa = v->vdpa;const struct vdpa_config_ops *ops = vdpa->config;struct vdpa_notification_area notify; -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;u16 index = vma->vm_pgoff; notify = ops->get_vq_notification (vdpa, index); vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); -if (remap_pfn_range (vma, vmf->address &PAGE_MASK , +if (remap_pfn_range (vma, vmf->info.address &PAGE_MASK , notify.addr >>PAGE_SHIFT ,PAGE_SIZE , vma->vm_page_prot))return VM_FAULT_SIGBUS ;
drivers/video/fbdev/core/fb_defio.c +7 -7
@@ -41,9 +41,9 @@ static vm_fault_t fb_deferred_io_fault(struct vm_fault *vmf) {unsigned long offset;struct page *page; -struct fb_info *info = vmf->vma->vm_private_data; +struct fb_info *info = vmf->info.vma->vm_private_data; - offset = vmf->pgoff <<PAGE_SHIFT ; + offset = vmf->info.pgoff <<PAGE_SHIFT ;if (offset >= info->fix.smem_len)return VM_FAULT_SIGBUS ;
@@ -53,13 +53,13 @@ static vm_fault_t fb_deferred_io_fault(struct vm_fault *vmf)get_page (page); -if (vmf->vma->vm_file) - page->mapping = vmf->vma->vm_file->f_mapping; +if (vmf->info.vma->vm_file) + page->mapping = vmf->info.vma->vm_file->f_mapping;else printk (KERN_ERR "no mapping available\n" );BUG_ON (!page->mapping); - page->index = vmf->pgoff; + page->index = vmf->info.pgoff; vmf->page = page;return 0 ;
@@ -93,7 +93,7 @@ EXPORT_SYMBOL_GPL(fb_deferred_io_fsync);static vm_fault_t fb_deferred_io_mkwrite (struct vm_fault *vmf) {struct page *page = vmf->page; -struct fb_info *info = vmf->vma->vm_private_data; +struct fb_info *info = vmf->info.vma->vm_private_data;struct fb_deferred_io *fbdefio = info->fbdefio;struct page *cur;
@@ -103,7 +103,7 @@ static vm_fault_t fb_deferred_io_mkwrite(struct vm_fault *vmf)deferred framebuffer IO. then if userspace touches a page again, we repeat the same scheme */ -file_update_time (vmf->vma->vm_file); +file_update_time (vmf->info.vma->vm_file);/* protect against the workqueue changing the page list */ mutex_lock (&fbdefio->lock);
drivers/xen/privcmd-buf.c +3 -2
@@ -117,8 +117,9 @@ static void privcmd_buf_vma_close(struct vm_area_struct *vma)static vm_fault_t privcmd_buf_vma_fault (struct vm_fault *vmf) {pr_debug ("fault: vma=%p %lx-%lx, pgoff=%lx, uv=%p\n" , - vmf->vma, vmf->vma->vm_start, vmf->vma->vm_end, - vmf->pgoff, (void *)vmf->address); + vmf->info.vma, vmf->info.vma->vm_start, + vmf->info.vma->vm_end, + vmf->info.pgoff, (void *) vmf->info.address);return VM_FAULT_SIGBUS ; }
drivers/xen/privcmd.c +2 -2
@@ -905,8 +905,8 @@ static void privcmd_close(struct vm_area_struct *vma)static vm_fault_t privcmd_fault (struct vm_fault *vmf) {printk (KERN_DEBUG "privcmd_fault: vma=%p %lx-%lx, pgoff=%lx, uv=%p\n" , - vmf->vma, vmf->vma->vm_start, vmf->vma->vm_end, - vmf->pgoff, (void *)vmf->address); + vmf->info.vma, vmf->info.vma->vm_start, vmf->info.vma->vm_end, + vmf->info.pgoff, (void *) vmf->info.address);return VM_FAULT_SIGBUS ; }
fs/9p/vfs_file.c +1 -1
@@ -548,7 +548,7 @@ v9fs_vm_page_mkwrite(struct vm_fault *vmf) {struct v9fs_inode *v9inode;struct page *page = vmf->page; -struct file *filp = vmf->vma->vm_file; +struct file *filp = vmf->info.vma->vm_file;struct inode *inode =file_inode (filp);
fs/afs/write.c +1 -1
@@ -834,7 +834,7 @@ int afs_fsync(struct file *file, loff_t start, loff_t end, int datasync)*/ vm_fault_t afs_page_mkwrite (struct vm_fault *vmf) { -struct file *file = vmf->vma->vm_file; +struct file *file = vmf->info.vma->vm_file;struct inode *inode =file_inode (file);struct afs_vnode *vnode =AFS_FS_I (inode);unsigned long priv;
fs/btrfs/inode.c +2 -2
@@ -8297,7 +8297,7 @@ static void btrfs_invalidatepage(struct page *page, unsigned int offset,vm_fault_t btrfs_page_mkwrite (struct vm_fault *vmf) {struct page *page = vmf->page; -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);struct extent_io_tree *io_tree = &BTRFS_I (inode)->io_tree;struct btrfs_ordered_extent *ordered;
@@ -8332,7 +8332,7 @@ vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf) ret2 =btrfs_delalloc_reserve_space (BTRFS_I (inode), &data_reserved, page_start, reserved_space);if (!ret2) { - ret2 =file_update_time (vmf->vma->vm_file); + ret2 =file_update_time (vmf->info.vma->vm_file); reserved =1 ; }if (ret2) {
fs/ceph/addr.c +3 -3
@@ -1466,12 +1466,12 @@ static void ceph_restore_sigs(sigset_t *oldset)*/ static vm_fault_t ceph_filemap_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct inode *inode =file_inode (vma->vm_file);struct ceph_inode_info *ci =ceph_inode (inode);struct ceph_file_info *fi = vma->vm_file->private_data;struct page *pinned_page =NULL ; -loff_t off = (loff_t )vmf->pgoff <<PAGE_SHIFT ; +loff_t off = (loff_t ) vmf->info.pgoff <<PAGE_SHIFT ;int want, got, err;sigset_t oldset;vm_fault_t ret =VM_FAULT_SIGBUS ;
@@ -1558,7 +1558,7 @@ static vm_fault_t ceph_filemap_fault(struct vm_fault *vmf)*/ static vm_fault_t ceph_page_mkwrite (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct inode *inode =file_inode (vma->vm_file);struct ceph_inode_info *ci =ceph_inode (inode);struct ceph_file_info *fi = vma->vm_file->private_data;
fs/dax.c +28 -25
144 lines not on this page · raw message
fs/ext2/file.c +3 -3
@@ -90,15 +90,15 @@ static ssize_t ext2_dax_write_iter(struct kiocb *iocb, struct iov_iter *from)*/ static vm_fault_t ext2_dax_fault (struct vm_fault *vmf) { -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);struct ext2_inode_info *ei =EXT2_I (inode);vm_fault_t ret;bool write = (vmf->flags &FAULT_FLAG_WRITE ) && - (vmf->vma->vm_flags &VM_SHARED ); + (vmf->info.vma->vm_flags &VM_SHARED );if (write) {sb_start_pagefault (inode->i_sb); -file_update_time (vmf->vma->vm_file); +file_update_time (vmf->info.vma->vm_file); }down_read (&ei->dax_sem);
fs/ext4/file.c +3 -3
@@ -672,7 +672,7 @@ static vm_fault_t ext4_dax_huge_fault(struct vm_fault *vmf,vm_fault_t result;int retries =0 ;handle_t *handle =NULL ; -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);struct super_block *sb = inode->i_sb;/*
@@ -687,12 +687,12 @@ static vm_fault_t ext4_dax_huge_fault(struct vm_fault *vmf,* we eventually come back with a COW page. */ bool write = (vmf->flags &FAULT_FLAG_WRITE ) && - (vmf->vma->vm_flags &VM_SHARED ); + (vmf->info.vma->vm_flags &VM_SHARED );pfn_t pfn;if (write) {sb_start_pagefault (sb); -file_update_time (vmf->vma->vm_file); +file_update_time (vmf->info.vma->vm_file);down_read (&EXT4_I (inode)->i_mmap_sem); retry: handle = ext4_journal_start_sb(sb, EXT4_HT_WRITE_PAGE,
fs/ext4/inode.c +2 -2
@@ -6051,7 +6051,7 @@ static int ext4_bh_unmapped(handle_t *handle, struct buffer_head *bh)vm_fault_t ext4_page_mkwrite (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct page *page = vmf->page;loff_t size;unsigned long len;
@@ -6194,7 +6194,7 @@ vm_fault_t ext4_page_mkwrite(struct vm_fault *vmf)vm_fault_t ext4_filemap_fault (struct vm_fault *vmf) { -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);vm_fault_t ret;down_read (&EXT4_I (inode)->i_mmap_sem);
fs/f2fs/file.c +4 -4
@@ -35,7 +35,7 @@static vm_fault_t f2fs_filemap_fault (struct vm_fault *vmf) { -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);vm_fault_t ret;down_read (&F2FS_I (inode)->i_mmap_sem);
@@ -46,7 +46,7 @@ static vm_fault_t f2fs_filemap_fault(struct vm_fault *vmf)f2fs_update_iostat (F2FS_I_SB (inode),APP_MAPPED_READ_IO ,F2FS_BLKSIZE ); -trace_f2fs_filemap_fault (inode, vmf->pgoff, (unsigned long )ret); +trace_f2fs_filemap_fault (inode, vmf->info.pgoff, (unsigned long )ret);return ret; }
@@ -54,7 +54,7 @@ static vm_fault_t f2fs_filemap_fault(struct vm_fault *vmf)static vm_fault_t f2fs_vm_page_mkwrite (struct vm_fault *vmf) {struct page *page = vmf->page; -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);struct f2fs_sb_info *sbi =F2FS_I_SB (inode);struct dnode_of_data dn;bool need_alloc = true;
@@ -94,7 +94,7 @@ static vm_fault_t f2fs_vm_page_mkwrite(struct vm_fault *vmf)f2fs_bug_on (sbi,f2fs_has_inline_data (inode)); -file_update_time (vmf->vma->vm_file); +file_update_time (vmf->info.vma->vm_file);down_read (&F2FS_I (inode)->i_mmap_sem);lock_page (page); if (unlikely(page->mapping != inode->i_mapping ||
fs/fuse/dax.c +1 -1
@@ -792,7 +792,7 @@ static vm_fault_t __fuse_dax_fault(struct vm_fault *vmf, enum page_entry_size pe_size, bool write) {vm_fault_t ret; -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);struct super_block *sb = inode->i_sb;pfn_t pfn;int error =0 ;
fs/fuse/file.c +2 -2
@@ -2336,9 +2336,9 @@ static void fuse_vma_close(struct vm_area_struct *vma)static vm_fault_t fuse_page_mkwrite (struct vm_fault *vmf) {struct page *page = vmf->page; -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file); -file_update_time (vmf->vma->vm_file); +file_update_time (vmf->info.vma->vm_file);lock_page (page);if (page->mapping != inode->i_mapping) {unlock_page (page);
fs/gfs2/file.c +4 -4
@@ -445,7 +445,7 @@ static int gfs2_allocate_page_backing(struct page *page, unsigned int length)static vm_fault_t gfs2_page_mkwrite (struct vm_fault *vmf) {struct page *page = vmf->page; -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);struct gfs2_inode *ip =GFS2_I (inode);struct gfs2_sbd *sdp =GFS2_SB (inode);struct gfs2_alloc_parms ap = { .aflags =0 , };
@@ -471,7 +471,7 @@ static vm_fault_t gfs2_page_mkwrite(struct vm_fault *vmf) }/* Update file times before taking page lock */ -file_update_time (vmf->vma->vm_file); +file_update_time (vmf->info.vma->vm_file);/* page is wholly or partially inside EOF */ if (offset > size -PAGE_SIZE )
@@ -479,7 +479,7 @@ static vm_fault_t gfs2_page_mkwrite(struct vm_fault *vmf)else length =PAGE_SIZE ; -gfs2_size_hint (vmf->vma->vm_file, offset, length); +gfs2_size_hint (vmf->info.vma->vm_file, offset, length);set_bit (GLF_DIRTY , &ip->i_gl->gl_flags);set_bit (GIF_SW_PAGED , &ip->i_flags);
@@ -560,7 +560,7 @@ static vm_fault_t gfs2_page_mkwrite(struct vm_fault *vmf)static vm_fault_t gfs2_fault (struct vm_fault *vmf) { -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);struct gfs2_inode *ip =GFS2_I (inode);struct gfs2_holder gh;vm_fault_t ret;
fs/iomap/buffered-io.c +1 -1
@@ -1007,7 +1007,7 @@ iomap_page_mkwrite_actor(struct inode *inode, loff_t pos, loff_t length,vm_fault_t iomap_page_mkwrite (struct vm_fault *vmf,const struct iomap_ops *ops) {struct page *page = vmf->page; -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);unsigned long length;loff_t offset;ssize_t ret;
fs/kernfs/file.c +2 -2
@@ -349,7 +349,7 @@ static void kernfs_vma_open(struct vm_area_struct *vma)static vm_fault_t kernfs_vma_fault (struct vm_fault *vmf) { -struct file *file = vmf->vma->vm_file; +struct file *file = vmf->info.vma->vm_file;struct kernfs_open_file *of =kernfs_of (file);vm_fault_t ret;
@@ -369,7 +369,7 @@ static vm_fault_t kernfs_vma_fault(struct vm_fault *vmf)static vm_fault_t kernfs_vma_page_mkwrite (struct vm_fault *vmf) { -struct file *file = vmf->vma->vm_file; +struct file *file = vmf->info.vma->vm_file;struct kernfs_open_file *of =kernfs_of (file);vm_fault_t ret;
fs/nfs/file.c +1 -1
@@ -542,7 +542,7 @@ const struct address_space_operations nfs_file_aops = {static vm_fault_t nfs_vm_page_mkwrite (struct vm_fault *vmf) {struct page *page = vmf->page; -struct file *filp = vmf->vma->vm_file; +struct file *filp = vmf->info.vma->vm_file;struct inode *inode =file_inode (filp);unsigned pagelen;vm_fault_t ret =VM_FAULT_NOPAGE ;
fs/nilfs2/file.c +1 -1
@@ -44,7 +44,7 @@ int nilfs_sync_file(struct file *file, loff_t start, loff_t end, int datasync)static vm_fault_t nilfs_page_mkwrite (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;struct page *page = vmf->page;struct inode *inode =file_inode (vma->vm_file);struct nilfs_transaction_info ti;
fs/ocfs2/mmap.c +4 -4
@@ -32,7 +32,7 @@static vm_fault_t ocfs2_fault (struct vm_fault *vmf) { -struct vm_area_struct *vma = vmf->vma; +struct vm_area_struct *vma = vmf->info.vma;sigset_t oldset;vm_fault_t ret;
@@ -41,7 +41,7 @@ static vm_fault_t ocfs2_fault(struct vm_fault *vmf)ocfs2_unblock_signals (&oldset);trace_ocfs2_fault (OCFS2_I (vma->vm_file->f_mapping->host)->ip_blkno, - vma, vmf->page, vmf->pgoff); + vma, vmf->page, vmf->info.pgoff);return ret; }
@@ -115,7 +115,7 @@ static vm_fault_t __ocfs2_page_mkwrite(struct file *file,static vm_fault_t ocfs2_page_mkwrite (struct vm_fault *vmf) {struct page *page = vmf->page; -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);struct buffer_head *di_bh =NULL ;sigset_t oldset;int err;
@@ -143,7 +143,7 @@ static vm_fault_t ocfs2_page_mkwrite(struct vm_fault *vmf)*/ down_write (&OCFS2_I (inode)->ip_alloc_sem); - ret =__ocfs2_page_mkwrite (vmf->vma->vm_file, di_bh, page); + ret =__ocfs2_page_mkwrite (vmf->info.vma->vm_file, di_bh, page);up_write (&OCFS2_I (inode)->ip_alloc_sem);
fs/orangefs/file.c +1 -1
@@ -455,7 +455,7 @@ static long orangefs_ioctl(struct file *file, unsigned int cmd, unsigned long arstatic vm_fault_t orangefs_fault (struct vm_fault *vmf) { -struct file *file = vmf->vma->vm_file; +struct file *file = vmf->info.vma->vm_file;int ret; ret =orangefs_inode_getattr (file->f_mapping->host,ORANGEFS_GETATTR_SIZE );
fs/orangefs/inode.c +2 -2
@@ -674,7 +674,7 @@ static const struct address_space_operations orangefs_address_operations = {vm_fault_t orangefs_page_mkwrite (struct vm_fault *vmf) {struct page *page = vmf->page; -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);struct orangefs_inode_s *orangefs_inode =ORANGEFS_I (inode);unsigned long *bitlock = &orangefs_inode->bitlock;vm_fault_t ret;
@@ -725,7 +725,7 @@ vm_fault_t orangefs_page_mkwrite(struct vm_fault *vmf)attach_page_private (page, wr); okay: -file_update_time (vmf->vma->vm_file); +file_update_time (vmf->info.vma->vm_file);if (page->mapping != inode->i_mapping) {unlock_page (page); ret =VM_FAULT_LOCKED |VM_FAULT_NOPAGE ;
fs/proc/vmcore.c +2 -2
@@ -411,8 +411,8 @@ static ssize_t read_vmcore(struct file *file, char __user *buffer,static vm_fault_t mmap_vmcore_fault (struct vm_fault *vmf) {#ifdef CONFIG_S390 -struct address_space *mapping = vmf->vma->vm_file->f_mapping; -pgoff_t index = vmf->pgoff; +struct address_space *mapping = vmf->info.vma->vm_file->f_mapping; +pgoff_t index = vmf->info.pgoff;struct page *page;loff_t offset;char *buf;
fs/ubifs/file.c +1 -1
@@ -1507,7 +1507,7 @@ static int ubifs_releasepage(struct page *page, gfp_t unused_gfp_flags)static vm_fault_t ubifs_vm_page_mkwrite (struct vm_fault *vmf) {struct page *page = vmf->page; -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);struct ubifs_info *c = inode->i_sb->s_fs_info;struct timespec64 now =current_time (inode);struct ubifs_budget_req req = { .new_page =1 };
fs/userfaultfd.c +9 -8
@@ -367,7 +367,7 @@ static inline long userfaultfd_get_blocking_state(unsigned int flags)*/ vm_fault_t handle_userfault (struct vm_fault *vmf,unsigned long reason) { -struct mm_struct *mm = vmf->vma->vm_mm; +struct mm_struct *mm = vmf->info.vma->vm_mm;struct userfaultfd_ctx *ctx;struct userfaultfd_wait_queue uwq;vm_fault_t ret =VM_FAULT_SIGBUS ;
@@ -394,7 +394,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)*/ mmap_assert_locked (mm); - ctx = vmf->vma->vm_userfaultfd_ctx.ctx; + ctx = vmf->info.vma->vm_userfaultfd_ctx.ctx;if (!ctx) goto out;
@@ -480,8 +480,8 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)init_waitqueue_func_entry (&uwq.wq, userfaultfd_wake_function); uwq.wq.private = current; - uwq.msg =userfault_msg (vmf->address, vmf->flags, reason, - ctx->features); + uwq.msg =userfault_msg (vmf->info.address, vmf->flags, reason, + ctx->features); uwq.ctx = ctx; uwq.waken = false;
@@ -501,12 +501,13 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)set_current_state (blocking_state);spin_unlock_irq (&ctx->fault_pending_wqh.lock); -if (!is_vm_hugetlb_page (vmf->vma)) - must_wait =userfaultfd_must_wait (ctx, vmf->address, vmf->flags, +if (!is_vm_hugetlb_page (vmf->info.vma)) + must_wait =userfaultfd_must_wait (ctx, vmf->info.address, + vmf->flags, reason);else - must_wait =userfaultfd_huge_must_wait (ctx, vmf->vma, - vmf->address, + must_wait =userfaultfd_huge_must_wait (ctx, vmf->info.vma, + vmf->info.address, vmf->flags, reason);mmap_read_unlock (mm);
fs/xfs/xfs_file.c +6 -6
@@ -1233,7 +1233,7 @@ __xfs_filemap_fault( enum page_entry_size pe_size, bool write_fault) { -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);struct xfs_inode *ip =XFS_I (inode);vm_fault_t ret;
@@ -1241,7 +1241,7 @@ __xfs_filemap_fault(if (write_fault) {sb_start_pagefault (inode->i_sb); -file_update_time (vmf->vma->vm_file); +file_update_time (vmf->info.vma->vm_file); }xfs_ilock (XFS_I (inode),XFS_MMAPLOCK_SHARED );
@@ -1273,7 +1273,7 @@ xfs_is_write_fault( struct vm_fault *vmf) {return (vmf->flags &FAULT_FLAG_WRITE ) && - (vmf->vma->vm_flags &VM_SHARED ); + (vmf->info.vma->vm_flags &VM_SHARED ); } static vm_fault_t
@@ -1282,7 +1282,7 @@ xfs_filemap_fault( {/* DAX can shortcut the normal fault path on write faults! */ return __xfs_filemap_fault (vmf,PE_SIZE_PTE , -IS_DAX (file_inode (vmf->vma->vm_file)) && +IS_DAX (file_inode (vmf->info.vma->vm_file)) &&xfs_is_write_fault (vmf)); }
@@ -1291,7 +1291,7 @@ xfs_filemap_huge_fault( struct vm_fault *vmf, enum page_entry_size pe_size) { -if (!IS_DAX (file_inode (vmf->vma->vm_file))) +if (!IS_DAX (file_inode (vmf->info.vma->vm_file)))return VM_FAULT_FALLBACK ;/* DAX can shortcut the normal fault path on write faults! */
@@ -1325,7 +1325,7 @@ xfs_filemap_map_pages( pgoff_t start_pgoff, pgoff_t end_pgoff) { -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);vm_fault_t ret;xfs_ilock (XFS_I (inode),XFS_MMAPLOCK_SHARED );
fs/zonefs/super.c +3 -3
@@ -551,7 +551,7 @@ static int zonefs_file_fsync(struct file *file, loff_t start, loff_t end,static vm_fault_t zonefs_filemap_fault (struct vm_fault *vmf) { -struct zonefs_inode_info *zi =ZONEFS_I (file_inode (vmf->vma->vm_file)); +struct zonefs_inode_info *zi =ZONEFS_I (file_inode (vmf->info.vma->vm_file));vm_fault_t ret;down_read (&zi->i_mmap_sem);
@@ -563,7 +563,7 @@ static vm_fault_t zonefs_filemap_fault(struct vm_fault *vmf)static vm_fault_t zonefs_filemap_page_mkwrite (struct vm_fault *vmf) { -struct inode *inode =file_inode (vmf->vma->vm_file); +struct inode *inode =file_inode (vmf->info.vma->vm_file);struct zonefs_inode_info *zi =ZONEFS_I (inode);vm_fault_t ret;
@@ -578,7 +578,7 @@ static vm_fault_t zonefs_filemap_page_mkwrite(struct vm_fault *vmf)return VM_FAULT_NOPAGE ;sb_start_pagefault (inode->i_sb); -file_update_time (vmf->vma->vm_file); +file_update_time (vmf->info.vma->vm_file);/* Serialize against truncates */ down_read (&zi->i_mmap_sem);
include/linux/huge_mm.h +4 -2
@@ -55,7 +55,8 @@ vm_fault_t vmf_insert_pfn_pmd_prot(struct vm_fault *vmf, pfn_t pfn,static inline vm_fault_t vmf_insert_pfn_pmd (struct vm_fault *vmf,pfn_t pfn,bool write) { -return vmf_insert_pfn_pmd_prot (vmf, pfn, vmf->vma->vm_page_prot, write); +struct vm_area_struct *vma = vmf->info.vma; +return vmf_insert_pfn_pmd_prot (vmf, pfn, vma->vm_page_prot, write); }vm_fault_t vmf_insert_pfn_pud_prot (struct vm_fault *vmf,pfn_t pfn,pgprot_t pgprot,bool write);
@@ -74,7 +75,8 @@ vm_fault_t vmf_insert_pfn_pud_prot(struct vm_fault *vmf, pfn_t pfn,static inline vm_fault_t vmf_insert_pfn_pud (struct vm_fault *vmf,pfn_t pfn,bool write) { -return vmf_insert_pfn_pud_prot (vmf, pfn, vmf->vma->vm_page_prot, write); +struct vm_area_struct *vma = vmf->info.vma; +return vmf_insert_pfn_pud_prot (vmf, pfn, vma->vm_page_prot, write); }enum transparent_hugepage_flag {
include/linux/mm.h +7 -2
@@ -516,12 +516,17 @@ static inline bool fault_flag_allow_retry_first(unsigned int flags)* * pgoff should be used in favour of virtual_address, if possible. */ -struct vm_fault { +struct vm_fault_info {struct vm_area_struct *vma;/* Target VMA */ -unsigned int flags;/* FAULT_FLAG_xxx flags */ gfp_t gfp_mask;/* gfp mask to be used for allocations */ pgoff_t pgoff;/* Logical page offset based on vma */ unsigned long address;/* Faulting virtual address */ +}; + +struct vm_fault { +struct vm_fault_info info; +unsigned int flags;/* FAULT_FLAG_xxx flags +* XXX: should be in vm_fault_info */ pmd_t *pmd;/* Pointer to pmd entry matching * the 'address' */ pud_t *pud;/* Pointer to pud entry matching
include/trace/events/fs_dax.h +14 -14
@@ -26,12 +26,12 @@ DECLARE_EVENT_CLASS(dax_pmd_fault_class,TP_fast_assign ( __entry->dev = inode->i_sb->s_dev; __entry->ino = inode->i_ino; - __entry->vm_start = vmf->vma->vm_start; - __entry->vm_end = vmf->vma->vm_end; - __entry->vm_flags = vmf->vma->vm_flags; - __entry->address = vmf->address; + __entry->vm_start = vmf->info.vma->vm_start; + __entry->vm_end = vmf->info.vma->vm_end; + __entry->vm_flags = vmf->info.vma->vm_flags; + __entry->address = vmf->info.address; __entry->flags = vmf->flags; - __entry->pgoff = vmf->pgoff; + __entry->pgoff = vmf->info.pgoff; __entry->max_pgoff = max_pgoff; __entry->result = result; ),
@@ -76,8 +76,8 @@ DECLARE_EVENT_CLASS(dax_pmd_load_hole_class,TP_fast_assign ( __entry->dev = inode->i_sb->s_dev; __entry->ino = inode->i_ino; - __entry->vm_flags = vmf->vma->vm_flags; - __entry->address = vmf->address; + __entry->vm_flags = vmf->info.vma->vm_flags; + __entry->address = vmf->info.address; __entry->zero_page = zero_page; __entry->radix_entry = radix_entry; ),
@@ -119,8 +119,8 @@ DECLARE_EVENT_CLASS(dax_pmd_insert_mapping_class,TP_fast_assign ( __entry->dev = inode->i_sb->s_dev; __entry->ino = inode->i_ino; - __entry->vm_flags = vmf->vma->vm_flags; - __entry->address = vmf->address; + __entry->vm_flags = vmf->info.vma->vm_flags; + __entry->address = vmf->info.address; __entry->write = vmf->flags &FAULT_FLAG_WRITE ; __entry->length = length; __entry->pfn_val = pfn.val;
@@ -165,10 +165,10 @@ DECLARE_EVENT_CLASS(dax_pte_fault_class,TP_fast_assign ( __entry->dev = inode->i_sb->s_dev; __entry->ino = inode->i_ino; - __entry->vm_flags = vmf->vma->vm_flags; - __entry->address = vmf->address; + __entry->vm_flags = vmf->info.vma->vm_flags; + __entry->address = vmf->info.address; __entry->flags = vmf->flags; - __entry->pgoff = vmf->pgoff; + __entry->pgoff = vmf->info.pgoff; __entry->result = result; ), TP_printk("dev %d:%d ino %#lx %s %s address %#lx pgoff %#lx %s",
@@ -208,8 +208,8 @@ TRACE_EVENT(dax_insert_mapping,TP_fast_assign ( __entry->dev = inode->i_sb->s_dev; __entry->ino = inode->i_ino; - __entry->vm_flags = vmf->vma->vm_flags; - __entry->address = vmf->address; + __entry->vm_flags = vmf->info.vma->vm_flags; + __entry->address = vmf->info.address; __entry->write = vmf->flags &FAULT_FLAG_WRITE ; __entry->radix_entry = radix_entry; ),
ipc/shm.c +1 -1
@@ -428,7 +428,7 @@ void exit_shm(struct task_struct *task)static vm_fault_t shm_fault (struct vm_fault *vmf) { -struct file *file = vmf->vma->vm_file; +struct file *file = vmf->info.vma->vm_file;struct shm_file_data *sfd =shm_file_data (file);return sfd->vm_ops->fault (vmf);
kernel/events/core.c +6 -6
@@ -5740,12 +5740,12 @@ EXPORT_SYMBOL_GPL(perf_event_update_userpage);static vm_fault_t perf_mmap_fault (struct vm_fault *vmf) { -struct perf_event *event = vmf->vma->vm_file->private_data; +struct perf_event *event = vmf->info.vma->vm_file->private_data;struct perf_buffer *rb;vm_fault_t ret =VM_FAULT_SIGBUS ;if (vmf->flags &FAULT_FLAG_MKWRITE ) { -if (vmf->pgoff ==0 ) +if (vmf->info.pgoff ==0 ) ret =0 ;return ret; }
@@ -5755,16 +5755,16 @@ static vm_fault_t perf_mmap_fault(struct vm_fault *vmf)if (!rb) goto unlock; -if (vmf->pgoff && (vmf->flags &FAULT_FLAG_WRITE )) +if (vmf->info.pgoff && (vmf->flags &FAULT_FLAG_WRITE )) goto unlock; - vmf->page =perf_mmap_to_page (rb, vmf->pgoff); + vmf->page =perf_mmap_to_page (rb, vmf->info.pgoff);if (!vmf->page) goto unlock;get_page (vmf->page); - vmf->page->mapping = vmf->vma->vm_file->f_mapping; - vmf->page->index = vmf->pgoff; + vmf->page->mapping = vmf->info.vma->vm_file->f_mapping; + vmf->page->index = vmf->info.pgoff; ret =0 ; unlock:
kernel/relay.c +2 -2
@@ -33,8 +33,8 @@ static LIST_HEAD(relay_channels);static vm_fault_t relay_buf_fault (struct vm_fault *vmf) {struct page *page; -struct rchan_buf *buf = vmf->vma->vm_private_data; -pgoff_t pgoff = vmf->pgoff; +struct rchan_buf *buf = vmf->info.vma->vm_private_data; +pgoff_t pgoff = vmf->info.pgoff;if (!buf)return VM_FAULT_OOM ;
lib/test_hmm.c +2 -2
@@ -1040,8 +1040,8 @@ static vm_fault_t dmirror_devmem_fault(struct vm_fault *vmf) dmirror = rpage->zone_device_data;/* FIXME demonstrate how we can adjust migrate range */ - args.vma = vmf->vma; - args.start = vmf->address; + args.vma = vmf->info.vma; + args.start = vmf->info.address; args.end = args.start +PAGE_SIZE ; args.src = &src_pfns; args.dst = &dst_pfns;
mm/filemap.c +25 -24
165 lines not on this page · raw message
mm/huge_memory.c +30 -27
165 lines not on this page · raw message
mm/hugetlb.c +4 -2
@@ -4294,8 +4294,10 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,if (userfaultfd_missing (vma)) {u32 hash;struct vm_fault vmf = { - .vma = vma, - .address = haddr, + .info = { + .vma = vma, + .address = haddr, + }, .flags = flags,/* * Hard to debug if it ends up being
mm/internal.h +2 -2
@@ -422,8 +422,8 @@ static inline struct file *maybe_unlock_mmap_for_io(struct vm_fault *vmf,*/ if (fault_flag_allow_retry_first (flags) && !(flags &FAULT_FLAG_RETRY_NOWAIT )) { - fpin =get_file (vmf->vma->vm_file); -mmap_read_unlock (vmf->vma->vm_mm); + fpin =get_file (vmf->info.vma->vm_file); +mmap_read_unlock (vmf->info.vma->vm_mm); }return fpin; }
mm/khugepaged.c +9 -7
@@ -997,16 +997,18 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,int swapped_in =0 ;vm_fault_t ret =0 ;struct vm_fault vmf = { - .vma = vma, - .address = address, + .info = { + .vma = vma, + .address = address, + .pgoff =linear_page_index (vma, address), + }, .flags =FAULT_FLAG_ALLOW_RETRY , .pmd = pmd, - .pgoff =linear_page_index (vma, address), }; vmf.pte =pte_offset_map (pmd, address); -for (; vmf.address < address +HPAGE_PMD_NR *PAGE_SIZE ; - vmf.pte++, vmf.address +=PAGE_SIZE ) { +for (; vmf.info.address < address +HPAGE_PMD_NR *PAGE_SIZE ; + vmf.pte++, vmf.info.address +=PAGE_SIZE ) { vmf.orig_pte = *vmf.pte;if (!is_swap_pte (vmf.orig_pte))continue ;
@@ -1016,7 +1018,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,/* do_swap_page returns VM_FAULT_RETRY with released mmap_lock */ if (ret &VM_FAULT_RETRY ) {mmap_read_lock (mm); -if (hugepage_vma_revalidate (mm, address, &vmf.vma)) { +if (hugepage_vma_revalidate (mm, address, &vmf.info.vma)) {/* vma is no longer available, don't continue to swapin */ trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced,0 );return false;
@@ -1032,7 +1034,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,return false; }/* pte is unmapped now, we need to map it */ - vmf.pte =pte_offset_map (pmd, vmf.address); + vmf.pte =pte_offset_map (pmd, vmf.info.address); } vmf.pte--;pte_unmap (vmf.pte);
mm/memory.c +130 -115
624 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
Part of the diff is not shown here. View the raw message.
Re: [RFC PATCH 4/8] mm: Separate fault info out of 'struct vm_fault'
Quoted 5 lines · Will Deacon
Try to clean this up by splitting the immutable fault information out into a new 'struct
vm_fault_info' which is embedded in 'structvm_fault' and will later be made 'const'. The vast majority of this change was performed with a coccinelle patch:
You may have a reason for doing it this way, but my reaction to this
was: "just make the new embedded struct unnamed".
Then you wouldn't need to do all the automated coccinelle changes.
Is there some reason you didn't do that, or just a "oh, I didn't think of it".
Linus
Will Deacon
maintainerRe: [RFC PATCH 4/8] mm: Separate fault info out of 'struct vm_fault'
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Will Deacon
Try to clean this up by splitting the immutable fault information out into a new 'struct
vm_fault_info' which is embedded in 'structvm_fault' and will later be made 'const'. The vast majority of this change was performed with a coccinelle patch:
Quoted 7 lines
You may have a reason for doing it this way, but my reaction to this
was: "just make the new embedded struct unnamed".Then you wouldn't need to do all the automated coccinelle changes.
Is there some reason you didn't do that, or just a "oh, I didn't think of it".
I tried that initially, e.g.
struct vm_fault {
const struct {
unsigned long address;
...
};
};but I found that I had to make all of the members const to get it to work, at which point the anonymous struct wasn't really adding anything. Did I just botch the syntax?
Will
Re: [RFC PATCH 4/8] mm: Separate fault info out of 'struct vm_fault'
Quoted 4 lines · Will Deacon
I tried that initially, but I found that I had to make all of the members const to get it to work, at which point the anonymous struct wasn't really adding anything. Did I just botch the syntax?
I'm not sure what you tried. But this stupid test-case sure works for me:
struct hello {
const struct {
unsigned long address;
};
unsigned int flags;
};
extern int fn(struct hello *);
int test(void)
{
struct hello a = {
.address = 1,
};
a.flags = 0;
return fn(&a);
}and because "address" is in that unnamed constant struct, you can only set it within that initializer, and cannot do
a.address = 0;
without an error (the way you _can_ do "a.flags = 0").
I don't see naming the struct making a difference - apart from forcing that big rename patch, of course.
But maybe we're talking about different issues?
Linus
Will Deacon
maintainerRe: [RFC PATCH 4/8] mm: Separate fault info out of 'struct vm_fault'
Quoted 29 lines · Linus Torvalds
Quoted 4 lines · Will Deacon
I tried that initially, but I found that I had to make all of the members const to get it to work, at which point the anonymous struct wasn't really adding anything. Did I just botch the syntax?
I'm not sure what you tried. But this stupid test-case sure works for me:
struct hello { const struct { unsigned long address; }; unsigned int flags; }; extern int fn(struct hello *); int test(void) { struct hello a = { .address = 1, }; a.flags = 0; return fn(&a); }and because "address" is in that unnamed constant struct, you can only set it within that initializer, and cannot do
Quoted 8 lines
a.address = 0;without an error (the way you _can_ do "a.flags = 0").
I don't see naming the struct making a difference - apart from forcing that big rename patch, of course.
But maybe we're talking about different issues?
Urgh...
We _are_ both on the same page, and your reply above had me thinking I've lost the plot, so I went back to the start. Check out v5.11-rc3 and apply this patch:
include/linux/mm.h +7 -4
@@ -514,11 +514,14 @@ static inline bool fault_flag_allow_retry_first(unsigned int flags)* pgoff should be used in favour of virtual_address, if possible. */ struct vm_fault { -struct vm_area_struct *vma;/* Target VMA */ +const struct { +struct vm_area_struct *vma;/* Target VMA */ +gfp_t gfp_mask;/* gfp mask to be used for allocations */ +pgoff_t pgoff;/* Logical page offset based on vma */ +unsigned long address;/* Faulting virtual address */ + }; +unsigned int flags;/* FAULT_FLAG_xxx flags */ -gfp_t gfp_mask;/* gfp mask to be used for allocations */ -pgoff_t pgoff;/* Logical page offset based on vma */ -unsigned long address;/* Faulting virtual address */ pmd_t *pmd;/* Pointer to pmd entry matching * the 'address' */ pud_t *pud;/* Pointer to pud entry matching
Sure enough, an arm64 defconfig builds perfectly alright with that change, but it really shouldn't. I'm using clang 11.0.5, so I had another go with GCC 9.2.1 and bang:
mm/filemap.c: In function ‘filemap_map_pages’:mm/filemap.c:2963:16: error: assignment of member ‘address’ in read-only object
2963 | vmf->address += (xas.xa_index - last_pgoff) << PAGE_SHIFT;
| ^~
make[1]: *** [scripts/Makefile.build:279: mm/filemap.o] Error 1 make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:1805: mm] Error 2
make: *** Waiting for unfinished jobs....
Nick -- any clue what's happening here? We would like that const anonymous struct to behave like a const struct member, as the alternative (naming the thing) results in a lot of refactoring churn.
Cheers,
Will
Re: [RFC PATCH 4/8] mm: Separate fault info out of 'struct vm_fault'
Quoted 85 lines · Will Deacon
Quoted 29 lines · Linus Torvalds
Quoted 4 lines · Will Deacon
I tried that initially, but I found that I had to make all of the members const to get it to work, at which point the anonymous struct wasn't really adding anything. Did I just botch the syntax?
I'm not sure what you tried. But this stupid test-case sure works for me:
struct hello { const struct { unsigned long address; }; unsigned int flags; }; extern int fn(struct hello *); int test(void) { struct hello a = { .address = 1, }; a.flags = 0; return fn(&a); }and because "address" is in that unnamed constant struct, you can only set it within that initializer, and cannot do
Quoted 8 lines
a.address = 0;without an error (the way you _can_ do "a.flags = 0").
I don't see naming the struct making a difference - apart from forcing that big rename patch, of course.
But maybe we're talking about different issues?
Urgh...
We _are_ both on the same page, and your reply above had me thinking I've lost the plot, so I went back to the start. Check out v5.11-rc3 and apply this patch:
include/linux/mm.h +7
-4@@ -514,11 +514,14 @@ static inline bool fault_flag_allow_retry_first(unsigned int flags)* pgoff should be used in favour of virtual_address, if possible. */ struct vm_fault { -struct vm_area_struct *vma;/* Target VMA */ +const struct { +struct vm_area_struct *vma;/* Target VMA */ +gfp_t gfp_mask;/* gfp mask to be used for allocations */ +pgoff_t pgoff;/* Logical page offset based on vma */ +unsigned long address;/* Faulting virtual address */ + }; +unsigned int flags;/* FAULT_FLAG_xxx flags */ -gfp_t gfp_mask;/* gfp mask to be used for allocations */ -pgoff_t pgoff;/* Logical page offset based on vma */ -unsigned long address;/* Faulting virtual address */ pmd_t *pmd;/* Pointer to pmd entry matching * the 'address' */ pud_t *pud;/* Pointer to pud entry matching Sure enough, an arm64 defconfig builds perfectly alright with that change, but it really shouldn't. I'm using clang 11.0.5, so I had another go with GCC 9.2.1 and bang:
mm/filemap.c: In function ‘filemap_map_pages’:mm/filemap.c:2963:16: error: assignment of member ‘address’ in read-only object
2963 |vmf->address+= (xas.xa_index -last_pgoff) <<PAGE_SHIFT;
| ^~
make[1]: *** [scripts/Makefile.build:279:mm/filemap.o] Error 1 make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:1805: mm] Error 2
make: *** Waiting for unfinished jobs....
Quoted 3 lines
Nick -- any clue what's happening here? We would like that const anonymous struct to behave like a const struct member, as the alternative (naming the thing) results in a lot of refactoring churn.
Weird, looks like a bug to me in Clang, filed
https://bugs.llvm.org/show_bug.cgi?id=48755.
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [RFC PATCH 4/8] mm: Separate fault info out of 'struct vm_fault'
In reply to Will DeaconQuoted 4 lines · Will Deacon
Sure enough, an arm64 defconfig builds perfectly alright with that change, but it really shouldn't. I'm using clang 11.0.5, so I had another go with GCC 9.2.1 and bang:
Ok, looks like a clang bug, but a reasonably benign one.
As long as we have sufficient coverage with gcc, we'll get error reporting in a timely manner for any new incorrect assignments, so I think we can do that constant anonymous struct even if it does mean that clang might let some bad cases through (I personally use gcc for build testing, and then clang for building my boot kernels, so I'd catch anything x86-64 allmodconfig in my build tests).
And keeping it unnamed it would avoid a lot of noisy churn..
Linus
Will Deacon
maintainerRe: [RFC PATCH 4/8] mm: Separate fault info out of 'struct vm_fault'
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Will Deacon
Sure enough, an arm64 defconfig builds perfectly alright with that change, but it really shouldn't. I'm using clang 11.0.5, so I had another go with GCC 9.2.1 and bang:
Ok, looks like a clang bug, but a reasonably benign one.
Quoted 8 lines
As long as we have sufficient coverage with gcc, we'll get error reporting in a timely manner for any new incorrect assignments, so I think we can do that constant anonymous struct even if it does mean that clang might let some bad cases through (I personally use gcc for build testing, and then clang for building my boot kernels, so I'd catch anything x86-64 allmodconfig in my build tests).
And keeping it unnamed it would avoid a lot of noisy churn..
Hmm. The feedback on the clang bug suggests that GCC is the one in the wrong here (although the argument is based on C11 and I haven't trawled through the standards to see how this has evolved):
https://bugs.llvm.org/show_bug.cgi?id=48755#c1
There is at least some sympathy to generating a warning, so that might be good enough. Otherwise, I suppose we can explicitly mark the fields as 'const' but I won't jump to that immediately.
Will
Re: [RFC PATCH 4/8] mm: Separate fault info out of 'struct vm_fault'
Quoted 4 lines · Will Deacon
Hmm. The feedback on the clang bug suggests that GCC is the one in the wrong here (although the argument is based on C11 and I haven't trawled through the standards to see how this has evolved):
Oh well.
That writing is absolutely the _worst_ kind of weaselwording standards language reading, trying to make excuses for bad behavior by basically depending on "this language is unclear", and trying to say that the buggy behavior is required by C11.
What a disappointment.
Absolutely nothing in the quoted C11 language says to me what that bug entry claims it says.
The argument seems to hinge on
"The members of an anonymous structure or union are considered to
be members of the containing structure or union"
and then it makes the completely uncalled-for leap that that means that because it was "int" in the const struct, it must be "int" in the containing structure too.
Which is complete BS, and doesn't follow logically _or_ grammatically. It would be a "member of the containing structure" even with the "const" qualifier, so the argument they make is just inane.
In fact, the _other_ sentence they quote clearly points to this being just a clang bug:
"A modifiable lvalue is [...] if it is a structure or union, does not
have any member (including, recursively, any member or element of all contained aggregates or unions) with a const- qualified type"
and clearly this recursively is an element with a const-qualified recursive struct.
Whatever. It's one of those "read the documentation squint-eyed to avoid doing the right thing" arguments.
It's not worth arguing with people like that, and let's just ignore the clang bug here.
Linus
Re: [RFC PATCH 4/8] mm: Separate fault info out of 'struct vm_fault'
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Will Deacon
Hmm. The feedback on the clang bug suggests that GCC is the one in the wrong here (although the argument is based on C11 and I haven't trawled through the standards to see how this has evolved):
Quoted 8 lines
Oh well.
That writing is absolutely the _worst_ kind of weaselwording standards language reading, trying to make excuses for bad behavior by basically depending on "this language is unclear", and trying to say that the buggy behavior is required by C11.
What a disappointment.
I don't really understand British humor either, but I assume that's how the language lawyers throw shade on one anothers' standards. Richard is both the WG21 spec editor (C++) and British, IIRC. Apparently, there's a long conversion (behind closed doors; it's the ISO way) going on in regards to the thread Richard has kicked off with them (WG14; C). Moreso on what should happen with the _Atomic qualifier, assignments, and memcpy. So it is still an important thing to nail down the language spec.
Note there were also a lot of discussions lately on "where should the volatile qualifier be allowed, or not."
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1152r0.html
https://www.youtube.com/watch?v=KJW_DLaVXIY
(2018? ok, maybe not lately. Lately for C)
I view this similarly as "where should the const qualifier be allowed, or not."
Signature · 2 lines
Thanks, ~Nick Desaulniers
Will Deacon
maintainerRFC PATCH v3 5/8 mm: Pass 'address' to map to do_set_pte() and drop FAULT_FLAG_PREFAULT
In reply to Will DeaconRather than modifying the 'address' field of the 'struct vm_fault_info' passed to do_set_pte(), leave that to identify the real faulting address and pass in the virtual address to be mapped by the new pte as a separate argument.
This makes FAULT_FLAG_PREFAULT redundant, as a prefault entry can be identified simply by comparing the new address parameter with the faulting address, so remove the redundant flag at the same time.
--- include/linux/mm.h | 7 ++----- mm/filemap.c | 22 +++++++--------------- mm/memory.c | 10 +++++----- 3 files changed, 14 insertions(+), 25 deletions(-)
include/linux/mm.h +2 -5
@@ -434,7 +434,6 @@ extern pgprot_t protection_map[16];* @FAULT_FLAG_REMOTE: The fault is not for current task/mm. * @FAULT_FLAG_INSTRUCTION: The fault was during an instruction fetch. * @FAULT_FLAG_INTERRUPTIBLE: The fault can be interrupted by non-fatal signals. -* @FAULT_FLAG_PREFAULT: Fault was a prefault. * * About @FAULT_FLAG_ALLOW_RETRY and @FAULT_FLAG_TRIED: we can specify * whether we would allow page faults to retry by specifying these two
@@ -465,7 +464,6 @@ extern pgprot_t protection_map[16];#define FAULT_FLAG_REMOTE 0x80#define FAULT_FLAG_INSTRUCTION 0x100#define FAULT_FLAG_INTERRUPTIBLE 0x200 -#define FAULT_FLAG_PREFAULT 0x400/* * The default fault flags that should be used by most of the
@@ -503,8 +501,7 @@ static inline bool fault_flag_allow_retry_first(unsigned int flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ - { FAULT_FLAG_PREFAULT, "PREFAULT" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }/* * vm_fault is filled by the pagefault handler and passed to the vma's
@@ -997,7 +994,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) }vm_fault_t do_set_pmd (struct vm_fault *vmf,struct page *page); -void do_set_pte (struct vm_fault *vmf,struct page *page); +void do_set_pte (struct vm_fault *vmf,struct page *page,unsigned long addr);vm_fault_t finish_fault (struct vm_fault *vmf);vm_fault_t finish_mkwrite_fault (struct vm_fault *vmf);
mm/filemap.c +7 -15
@@ -3018,8 +3018,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,struct file *file = vma->vm_file;struct address_space *mapping = file->f_mapping;pgoff_t last_pgoff = start_pgoff; -unsigned long address = vmf->info.address; -unsigned long flags = vmf->flags; +unsigned long addr;XA_STATE (xas, &mapping->i_pages, start_pgoff);struct page *head, *page;unsigned int mmap_miss =READ_ONCE (file->f_ra.mmap_miss);
@@ -3035,9 +3034,8 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, goto out; } - vmf->info.address = vma->vm_start + ((start_pgoff - vma->vm_pgoff) <<PAGE_SHIFT ); - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, - vmf->info.address, &vmf->ptl); + addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) <<PAGE_SHIFT ); + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, addr, &vmf->ptl); do { page = find_subpage(head, xas.xa_index); if (PageHWPoison(page))
@@ -3046,7 +3044,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,if (mmap_miss >0 ) mmap_miss--; - vmf->info.address += (xas.xa_index - last_pgoff) <<PAGE_SHIFT ; + addr += (xas.xa_index - last_pgoff) <<PAGE_SHIFT ; vmf->pte += xas.xa_index - last_pgoff; last_pgoff = xas.xa_index;
@@ -3054,16 +3052,12 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, goto unlock;/* We're about to handle the fault */ -if (vmf->info.address == address) { - vmf->flags &= ~FAULT_FLAG_PREFAULT ; +if (vmf->info.address == addr) ret =VM_FAULT_NOPAGE ; - }else { - vmf->flags |=FAULT_FLAG_PREFAULT ; - } -do_set_pte (vmf, page); +do_set_pte (vmf, page, addr);/* no need to invalidate: a not-present page won't be cached */ -update_mmu_cache (vma, vmf->info.address, vmf->pte); +update_mmu_cache (vma, addr, vmf->pte);unlock_page (head);continue ; unlock:
@@ -3073,8 +3067,6 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,pte_unmap_unlock (vmf->pte, vmf->ptl); out:rcu_read_unlock (); - vmf->flags = flags; - vmf->info.address = address;WRITE_ONCE (file->f_ra.mmap_miss, mmap_miss);return ret; }
mm/memory.c +5 -5
@@ -3741,11 +3741,11 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) }#endif -void do_set_pte (struct vm_fault *vmf,struct page *page) +void do_set_pte (struct vm_fault *vmf,struct page *page,unsigned long addr) {struct vm_area_struct *vma = vmf->info.vma;bool write = vmf->flags &FAULT_FLAG_WRITE ; -bool prefault = vmf->flags &FAULT_FLAG_PREFAULT ; +bool prefault = vmf->info.address != addr;pte_t entry;flush_icache_page (vma, page);
@@ -3761,13 +3761,13 @@ void do_set_pte(struct vm_fault *vmf, struct page *page)/* copy-on-write page */ if (write && !(vma->vm_flags &VM_SHARED )) {inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES ); -page_add_new_anon_rmap (page, vma, vmf->info.address, false); +page_add_new_anon_rmap (page, vma, addr, false);lru_cache_add_inactive_or_unevictable (page, vma); }else {inc_mm_counter_fast (vma->vm_mm,mm_counter_file (page));page_add_file_rmap (page, false); } -set_pte_at (vma->vm_mm, vmf->info.address, vmf->pte, entry); +set_pte_at (vma->vm_mm, addr, vmf->pte, entry); }/**
@@ -3827,7 +3827,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf) ret =0 ;/* Re-check under ptl */ if (likely (pte_none (*vmf->pte))) -do_set_pte (vmf, page); +do_set_pte (vmf, page, vmf->info.address);else ret =VM_FAULT_NOPAGE ;
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
Re: [RFC PATCH 5/8] mm: Pass 'address' to map to do_set_pte() and drop FAULT_FLAG_PREFAULT
Quoted 5 lines · Will Deacon
Rather than modifying the 'address' field of the 'struct
vm_fault_info' passed todo_set_pte(), leave that to identify the real faulting address and pass in the virtual address to be mapped by the new pte as a separate argument.
Ack.
In fact - apart from the question I had about the 'info' sub-structure
- ack on the whole series. But this one struck me particularly as "that's simpler and clearer" even if that finish_fault() case is now not as pretty (but with an unnamed structure it would be slightly simpler, at least).
Linus
Will Deacon
maintainerRe: [RFC PATCH 5/8] mm: Pass 'address' to map to do_set_pte() and drop FAULT_FLAG_PREFAULT
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Will Deacon
Rather than modifying the 'address' field of the 'struct
vm_fault_info' passed todo_set_pte(), leave that to identify the real faulting address and pass in the virtual address to be mapped by the new pte as a separate argument.
Quoted 7 lines
Ack.
In fact - apart from the question I had about the 'info' sub-structure
- ack on the whole series. But this one struck me particularly as "that's simpler and clearer" even if thatfinish_fault()case is now not as pretty (but with an unnamed structure it would be slightly simpler, at least).
Thanks. I'll post another spin next week to avoid the named structure, but if you (or anybody else) get a chance to glance over some of the other things I mentioned in the cover letter beforehand that would be really great.
Cheers,
Will
Will Deacon
maintainerRFC PATCH v3 6/8 mm: Avoid modifying vmf.info.address in __collapse_huge_page_swapin()
In reply to Will DeaconIn preparation for const-ifying the 'info' field of 'struct vm_fault', rework __collapse_huge_page_swapin() to avoid continously updating vmf.info.address and instead populate a new 'struct vm_fault' on the stack for each page being processed.
--- mm/khugepaged.c | 41 ++++++++++++++++++++--------------------- 1 file changed, 20 insertions(+), 21 deletions(-)
mm/khugepaged.c +20 -21
@@ -991,40 +991,43 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,static bool __collapse_huge_page_swapin (struct mm_struct *mm,struct vm_area_struct *vma, -unsigned long address,pmd_t *pmd, +unsigned long haddr,pmd_t *pmd,int referenced) {int swapped_in =0 ;vm_fault_t ret =0 ; -struct vm_fault vmf = { - .info = { - .vma = vma, - .address = address, - .pgoff =linear_page_index (vma, address), - }, - .flags =FAULT_FLAG_ALLOW_RETRY , - .pmd = pmd, - }; - - vmf.pte =pte_offset_map (pmd, address); -for (; vmf.info.address < address +HPAGE_PMD_NR *PAGE_SIZE ; - vmf.pte++, vmf.info.address +=PAGE_SIZE ) { +unsigned long address, end = haddr + (HPAGE_PMD_NR *PAGE_SIZE ); + +for (address = haddr; address < end; address +=PAGE_SIZE ) { +struct vm_fault vmf = { + .info = { + .vma = vma, + .address = address, + .pgoff =linear_page_index (vma, haddr), + }, + .flags =FAULT_FLAG_ALLOW_RETRY , + .pmd = pmd, + }; + + vmf.pte =pte_offset_map (pmd, address); vmf.orig_pte = *vmf.pte; -if (!is_swap_pte (vmf.orig_pte)) +if (!is_swap_pte (vmf.orig_pte)) { +pte_unmap (vmf.pte);continue ; + } swapped_in++; ret =do_swap_page (&vmf);/* do_swap_page returns VM_FAULT_RETRY with released mmap_lock */ if (ret &VM_FAULT_RETRY ) {mmap_read_lock (mm); -if (hugepage_vma_revalidate (mm, address, &vmf.info.vma)) { +if (hugepage_vma_revalidate (mm, haddr, &vma)) {/* vma is no longer available, don't continue to swapin */ trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced,0 );return false; }/* check if the pmd is still valid */ -if (mm_find_pmd (mm, address) != pmd) { +if (mm_find_pmd (mm, haddr) != pmd) {trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced,0 );return false; }
@@ -1033,11 +1036,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced,0 );return false; } -/* pte is unmapped now, we need to map it */ - vmf.pte =pte_offset_map (pmd, vmf.info.address); } - vmf.pte--; -pte_unmap (vmf.pte);/* Drain LRU add pagevec to remove extra pin on the swapped in pages */ if (swapped_in)
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
Will Deacon
maintainerRFC PATCH v3 7/8 mm: Use static initialisers for 'info' field of 'struct vm_fault'
In reply to Will DeaconIn preparation for const-ifying the 'info' field of 'struct vm_fault', ensure that it is initialised using static initialisers.
--- mm/shmem.c | 8 +++++--- mm/swapfile.c | 13 ++++++++----- 2 files changed, 13 insertions(+), 8 deletions(-)
mm/shmem.c +5 -3
@@ -1520,11 +1520,13 @@ static struct page *shmem_swapin(swp_entry_t swap, gfp_t gfp, {struct vm_area_struct pvma;struct page *page; -struct vm_fault vmf; +struct vm_fault vmf = { + .info = { + .vma = &pvma, + }, + };shmem_pseudo_vma_init (&pvma, info, index); - vmf.info.vma = &pvma; - vmf.info.address =0 ; page =swap_cluster_readahead (swap, gfp, &vmf);shmem_pseudo_vma_destroy (&pvma);
mm/swapfile.c +8 -5
@@ -1951,8 +1951,6 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, si = swap_info[type]; pte =pte_offset_map (pmd, addr); do { - struct vm_fault vmf; - if (!is_swap_pte(*pte)) continue;
@@ -1968,9 +1966,14 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, swap_map = &si->swap_map[offset]; page =lookup_swap_cache (entry, vma, addr);if (!page) { - vmf.info.vma = vma; - vmf.info.address = addr; - vmf.pmd = pmd; +struct vm_fault vmf = { + .info = { + .vma = vma, + .address = addr, + }, + .pmd = pmd, + }; + page =swapin_readahead (entry,GFP_HIGHUSER_MOVABLE , &vmf); }
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
Will Deacon
maintainerRFC PATCH v3 8/8 mm: Mark 'info' field of 'struct vm_fault' as 'const'
In reply to Will DeaconThe field is only ever read after being initialised, so mark it 'const' before somebody tries to modify it again.
--- include/linux/mm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/mm.h +1 -1
@@ -521,7 +521,7 @@ struct vm_fault_info { };struct vm_fault { -struct vm_fault_info info; +const struct vm_fault_info info;unsigned int flags;/* FAULT_FLAG_xxx flags * XXX: should be in vm_fault_info */ pmd_t *pmd;/* Pointer to pmd entry matching
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog