Enhance memory utilization with DMEMFS
Subsystems (15)
cover Enhance memory utilization with DMEMFS
From: Yulei Zhang <[email protected]>
In current system each physical memory page is assocaited with a page structure which is used to track the usage of this page. But due to the memory usage rapidly growing in cloud environment, we find the resource consuming for page structure storage becomes highly remarkable. So is it an expense that we could spare?
This patchset introduces an idea about how to save the extra memory through a new virtual filesystem -- dmemfs.
Dmemfs (Direct Memory filesystem) is device memory or reserved memory based filesystem. This kind of memory is special as it is not managed by kernel and most important it is without 'struct page'. Therefore we can leverage the extra memory from the host system to support more tenants in our cloud service.
We uses a kernel boot parameter 'dmem=' to reserve the system memory when the host system boots up, the details can be checked in /Documentation/admin-guide/kernel-parameters.txt.
Theoretically for each 4k physical page it can save 64 bytes if we drop the 'struct page', so for guest memory with 320G it can save about 5G physical memory totally.
Detailed usage of dmemfs is included in
/Documentation/filesystem/dmemfs.rst.
TODO:
1. we temporary disable the record_steal_time() before entering guest, will enable that after solve the conflict.
2. working on systemcall such as mincore, will update the status and patches soon.
Yulei Zhang (35):
fs: introduce dmemfs module
mm: support direct memory reservation
dmem: implement dmem memory management
dmem: let pat recognize dmem
dmemfs: support mmap
dmemfs: support truncating inode down
dmem: trace core functions
dmem: show some statistic in debugfs
dmemfs: support remote access
dmemfs: introduce max_alloc_try_dpages parameter
mm: export mempolicy interfaces to serve dmem allocator
dmem: introduce mempolicy support
mm, dmem: introduce PFN_DMEM and pfn_t_dmem
mm, dmem: dmem-pmd vs thp-pmd
mm: add pmd_special() check for pmd_trans_huge_lock()
dmemfs: introduce ->split() to dmemfs_vm_ops
mm, dmemfs: support unmap_page_range() for dmemfs pmd
mm: follow_pmd_mask() for dmem huge pmd
mm: gup_huge_pmd() for dmem huge pmd
mm: support dmem huge pmd for vmf_insert_pfn_pmd()
mm: support dmem huge pmd for follow_pfn()
kvm, x86: Distinguish dmemfs page from mmio page
kvm, x86: introduce VM_DMEM
dmemfs: support hugepage for dmemfs
mm, x86, dmem: fix estimation of reserved page for vaddr_get_pfn() mm, dmem: introduce pud_special()
mm: add pud_special() to support dmem huge pud
mm, dmemfs: support huge_fault() for dmemfs
mm: add follow_pte_pud()
dmem: introduce dmem_bitmap_alloc() and dmem_bitmap_free()
dmem: introduce mce handler
mm, dmemfs: register and handle the dmem mce
kvm, x86: temporary disable record_steal_time for dmem
dmem: add dmem unit tests
Add documentation for dmemfs
.../admin-guide/kernel-parameters.txt | 38 + Documentation/filesystems/dmemfs.rst | 59 + arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 32 +- arch/x86/include/asm/pgtable_types.h | 13 +- arch/x86/kernel/setup.c | 3 + arch/x86/kvm/mmu/mmu.c | 5 +- arch/x86/kvm/x86.c | 2 + arch/x86/mm/pat/memtype.c | 21 + drivers/vfio/vfio_iommu_type1.c | 4 + fs/Kconfig | 1 + fs/Makefile | 1 + fs/dmemfs/Kconfig | 16 + fs/dmemfs/Makefile | 8 + fs/dmemfs/inode.c | 1063 ++++++++++++++++ fs/dmemfs/trace.h | 54 + fs/inode.c | 6 + include/linux/dmem.h | 49 + include/linux/fs.h | 1 + include/linux/huge_mm.h | 5 +- include/linux/mempolicy.h | 3 + include/linux/mm.h | 9 + include/linux/pfn_t.h | 17 +- include/linux/pgtable.h | 22 + include/trace/events/dmem.h | 85 ++ include/uapi/linux/magic.h | 1 + mm/Kconfig | 21 + mm/Makefile | 1 + mm/dmem.c | 1075 +++++++++++++++++ mm/dmem_reserve.c | 303 +++++ mm/gup.c | 94 +- mm/huge_memory.c | 19 +- mm/memory-failure.c | 69 +- mm/memory.c | 74 +- mm/mempolicy.c | 4 +- mm/mprotect.c | 7 +- mm/mremap.c | 3 + tools/testing/dmem/Kbuild | 1 + tools/testing/dmem/Makefile | 10 + tools/testing/dmem/dmem-test.c | 184 +++ 40 files changed, 3336 insertions(+), 48 deletions(-) create mode 100644 Documentation/filesystems/dmemfs.rst create mode 100644 fs/dmemfs/Kconfig create mode 100644 fs/dmemfs/Makefile create mode 100644 fs/dmemfs/inode.c create mode 100644 fs/dmemfs/trace.h create mode 100644 include/linux/dmem.h create mode 100644 include/trace/events/dmem.h create mode 100644 mm/dmem.c create mode 100644 mm/dmem_reserve.c create mode 100644 tools/testing/dmem/Kbuild create mode 100644 tools/testing/dmem/Makefile create mode 100644 tools/testing/dmem/dmem-test.c
Signature · 1 line
2.28.0
PATCH 1/35 fs: introduce dmemfs module
From: Yulei Zhang <[email protected]>
dmemfs (Direct Memory filesystem) is device memory or reserved memory based filesystem. This kind of memory is special as it is not managed by kernel and it is without 'struct page'.
The original purpose of dmemfs is to drop the usage of 'struct page' to save extra system memory.
This patch introduces the basic framework of dmemfs and only mkdir and create regular file are supported.
--- fs/Kconfig | 1 + fs/Makefile | 1 + fs/dmemfs/Kconfig | 13 ++ fs/dmemfs/Makefile | 7 + fs/dmemfs/inode.c | 275 +++++++++++++++++++++++++++++++++++++ include/uapi/linux/magic.h | 1 + 6 files changed, 298 insertions(+) create mode 100644 fs/dmemfs/Kconfig create mode 100644 fs/dmemfs/Makefile create mode 100644 fs/dmemfs/inode.c
fs/Kconfig +1 -0
@@ -41,6 +41,7 @@ source "fs/btrfs/Kconfig" source "fs/nilfs2/Kconfig" source "fs/f2fs/Kconfig" source "fs/zonefs/Kconfig" +source "fs/dmemfs/Kconfig" config FS_DAX bool "Direct Access (DAX) support"
fs/Makefile +1 -0
@@ -136,3 +136,4 @@ obj-$(CONFIG_EFIVAR_FS) += efivarfs/ obj-$(CONFIG_EROFS_FS) += erofs/ obj-$(CONFIG_VBOXSF_FS) += vboxsf/ obj-$(CONFIG_ZONEFS_FS) += zonefs/ +obj-$(CONFIG_DMEM_FS) += dmemfs/
fs/dmemfs/Kconfig new file +13 -0
@@ -0,0 +1,13 @@ +config DMEM_FS + tristate "Direct Memory filesystem support" + help + dmemfs (Direct Memory filesystem) is device memory or reserved + memory based filesystem. This kind of memory is special as it + is not managed by kernel and it is without 'struct page'. + + The original purpose of dmemfs is saving extra memory of + 'struct page' that reduces the total cost of ownership (TCO) + for cloud providers. + + To compile this file system support as a module, choose M here: the + module will be called dmemfs.
fs/dmemfs/Makefile new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for the linux dmem-filesystem routines. +# +obj-$(CONFIG_DMEM_FS) += dmemfs.o + +dmemfs-y += inode.o
fs/dmemfs/inode.c new file +275 -0
275 lines not on this page · raw message
include/uapi/linux/magic.h +1 -0
@@ -97,5 +97,6 @@#define DEVMEM_MAGIC 0x454d444d/* "DMEM" */ #define Z3FOLD_MAGIC 0x33#define PPC_CMM_MAGIC 0xc7571590 +#define DMEMFS_MAGIC 0x2ace90c6#endif /* __LINUX_MAGIC_H__ */
Signature · 1 line
2.28.0
Al Viro
maintainerRe: [PATCH 01/35] fs: introduce dmemfs module
fs/dmemfs/inode.c line 61
Quoted 3 lines · On Thu, Oct 08, 2020 at 03:53:51PM +0800, [email protected] wrote:
fs/dmemfs/inode.c new file +3
-0+static struct inode * +dmemfs_get_inode (struct super_block *sb,const struct inode *dir,umode_t mode, +dev_t dev);
WTF is 'dev' for?
Quoted 4 lines
fs/dmemfs/inode.c new file +4
-0+static int +dmemfs_mknod (struct inode *dir,struct dentry *dentry,umode_t mode,dev_t dev) +{ +struct inode *inode =dmemfs_get_inode (dir->i_sb, dir, mode, dev);
fs/dmemfs/inode.c line 76
Quoted 10 lines
fs/dmemfs/inode.c new file +10
-0+int error = -ENOSPC ; + +if (inode) { +d_instantiate (dentry, inode); +dget (dentry);/* Extra count - pin the dentry in core */ + error =0 ; + dir->i_mtime = dir->i_ctime =current_time (inode); + } +return error; +}
... same here, seeing that you only call that thing from the next two functions and you do *not* provide ->mknod() as a method (unsurprisingly - what would device nodes do there?)
Quoted 14 lines
fs/dmemfs/inode.c new file +14
-0+static int dmemfs_create (struct inode *dir,struct dentry *dentry, +umode_t mode,bool excl) +{ +return dmemfs_mknod (dir, dentry, mode |S_IFREG ,0 ); +} + +static int dmemfs_mkdir (struct inode *dir,struct dentry *dentry, +umode_t mode) +{ +int retval =dmemfs_mknod (dir, dentry, mode |S_IFDIR ,0 ); + +if (!retval) +inc_nlink (dir); +return retval;
fs/dmemfs/inode.c line 115
Quoted 10 lines
fs/dmemfs/inode.c new file +9
-0+} +int dmemfs_file_mmap (struct file *file,struct vm_area_struct *vma) +{ +return 0 ; +} + +static const struct file_operations dmemfs_file_operations = { + .mmap = dmemfs_file_mmap, +};
Er... Is that a placeholder for later in the series? Because as it is, it makes no sense whatsoever - "it can be mmapped, but any access to the mapped area will segfault".
Quoted 23 lines
fs/dmemfs/inode.c new file +23
-0+struct inode *dmemfs_get_inode (struct super_block *sb, +const struct inode *dir,umode_t mode,dev_t dev) +{ +struct inode *inode =new_inode (sb); + +if (inode) { + inode->i_ino =get_next_ino (); +inode_init_owner (inode, dir, mode); + inode->i_mapping->a_ops = &empty_aops; +mapping_set_gfp_mask (inode->i_mapping,GFP_HIGHUSER ); +mapping_set_unevictable (inode->i_mapping); + inode->i_atime = inode->i_mtime = inode->i_ctime =current_time (inode); +switch (mode &S_IFMT ) { +default : +init_special_inode (inode, mode, dev); +break ; +case S_IFREG : + inode->i_op = &dmemfs_file_inode_operations; + inode->i_fop = &dmemfs_file_operations; +break ; +case S_IFDIR : + inode->i_op = &dmemfs_dir_inode_operations; + inode->i_fop = &simple_dir_operations;
fs/dmemfs/inode.c line 178
Quoted 10 lines
fs/dmemfs/inode.c new file +10
-0+ +/* +* directory inodes start off with i_nlink == 2 +* (for "." entry) +*/ +inc_nlink (inode); +break ; +case S_IFLNK : + inode->i_op = &page_symlink_inode_operations; +break ;
Where would symlinks come from? Or anything other than regular files and directories, for that matter...
Re: [PATCH 01/35] fs: introduce dmemfs module
Quoted 19 lines · Al Viro
fs/dmemfs/inode.c line 61
Quoted 3 lines · On Thu, Oct 08, 2020 at 03:53:51PM +0800, [email protected] wrote:
fs/dmemfs/inode.c new file +3
-0+static struct inode * +dmemfs_get_inode (struct super_block *sb,const struct inode *dir,umode_t mode, +dev_t dev);WTF is 'dev' for?
Quoted 10 lines
fs/dmemfs/inode.c new file +10
-0+static int +dmemfs_mknod (struct inode *dir,struct dentry *dentry,umode_t mode,dev_t dev) +{ +struct inode *inode =dmemfs_get_inode (dir->i_sb, dir, mode, dev); +int error = -ENOSPC ; + +if (inode) { +d_instantiate (dentry, inode); +dget (dentry);/* Extra count - pin the dentry in core */ + error =0 ;
Quoted 9 lines
fs/dmemfs/inode.c line 76
Quoted 4 lines
fs/dmemfs/inode.c new file +4
-0+ dir->i_mtime = dir->i_ctime =current_time (inode); + } +return error; +}... same here, seeing that you only call that thing from the next two functions and you do *not* provide ->mknod() as a method (unsurprisingly - what would device nodes do there?)
Thanks for pointing this out. we may need support the mknod method, otherwise the dev is redundant and need to be removed.
Quoted 16 lines
Quoted 15 lines
fs/dmemfs/inode.c new file +15
-0+static int dmemfs_create (struct inode *dir,struct dentry *dentry, +umode_t mode,bool excl) +{ +return dmemfs_mknod (dir, dentry, mode |S_IFREG ,0 ); +} + +static int dmemfs_mkdir (struct inode *dir,struct dentry *dentry, +umode_t mode) +{ +int retval =dmemfs_mknod (dir, dentry, mode |S_IFDIR ,0 ); + +if (!retval) +inc_nlink (dir); +return retval; +}
Quoted 13 lines
fs/dmemfs/inode.c line 115
Quoted 8 lines
fs/dmemfs/inode.c new file +8
-0+int dmemfs_file_mmap (struct file *file,struct vm_area_struct *vma) +{ +return 0 ; +} + +static const struct file_operations dmemfs_file_operations = { + .mmap = dmemfs_file_mmap, +};Er... Is that a placeholder for later in the series? Because as it is, it makes no sense whatsoever - "it can be mmapped, but any access to the mapped area will segfault".
Yes, we seperate the full implementation for dmemfs_file_mmap into patch 05/35, it
will assign the interfaces to handle the page fault.
Quoted 28 lines
Quoted 28 lines
fs/dmemfs/inode.c new file +28
-0+struct inode *dmemfs_get_inode (struct super_block *sb, +const struct inode *dir,umode_t mode,dev_t dev) +{ +struct inode *inode =new_inode (sb); + +if (inode) { + inode->i_ino =get_next_ino (); +inode_init_owner (inode, dir, mode); + inode->i_mapping->a_ops = &empty_aops; +mapping_set_gfp_mask (inode->i_mapping,GFP_HIGHUSER ); +mapping_set_unevictable (inode->i_mapping); + inode->i_atime = inode->i_mtime = inode->i_ctime =current_time (inode); +switch (mode &S_IFMT ) { +default : +init_special_inode (inode, mode, dev); +break ; +case S_IFREG : + inode->i_op = &dmemfs_file_inode_operations; + inode->i_fop = &dmemfs_file_operations; +break ; +case S_IFDIR : + inode->i_op = &dmemfs_dir_inode_operations; + inode->i_fop = &simple_dir_operations; + +/* +* directory inodes start off with i_nlink == 2 +* (for "." entry) +*/
Quoted 8 lines
fs/dmemfs/inode.c line 178
Quoted 5 lines
fs/dmemfs/inode.c new file +5
-0+inc_nlink (inode); +break ; +case S_IFLNK : + inode->i_op = &page_symlink_inode_operations; +break ;Where would symlinks come from? Or anything other than regular files and directories, for that matter...
You are right, so far it just supports regular files and directories.
Al Viro
maintainerRe: [PATCH 01/35] fs: introduce dmemfs module
Quoted 7 lines · yulei zhang
Quoted 4 lines
... same here, seeing that you only call that thing from the next two functions and you do *not* provide ->mknod() as a method (unsurprisingly - what would device nodes do there?)
Thanks for pointing this out. we may need support the mknod method, otherwise the dev is redundant and need to be removed.
I'd suggest turning that into (static) __create_file(....) with
static int dmemfs_create(struct inode *dir, struct dentry *dentry,
umode_t mode, bool excl)
{
return __create_file(dir, dentry, mode | S_IFREG);
}static int dmemfs_mkdir(struct inode *dir, struct dentry *dentry,
umode_t mode)
{
return __create_file(dir, dentry, mode | S_IFDIR);
}(i.e. even inc_nlink() of parent folded into that).
[snip]
Quoted 3 lines
Yes, we seperate the full implementation for
dmemfs_file_mmapinto patch 05/35, it
will assign the interfaces to handle the page fault.
It would be less confusing to move the introduction of ->mmap() to that patch, then.
Re: [PATCH 01/35] fs: introduce dmemfs module
Quoted 27 lines · Al Viro
Quoted 7 lines · yulei zhang
Quoted 4 lines
... same here, seeing that you only call that thing from the next two functions and you do *not* provide ->mknod() as a method (unsurprisingly - what would device nodes do there?)
Thanks for pointing this out. we may need support the mknod method, otherwise the dev is redundant and need to be removed.
I'd suggest turning that into (static)
__create_file(....)withstatic int dmemfs_create(struct inode *dir, struct dentry *dentry,
umode_tmode, bool excl){ return __create_file(dir, dentry, mode | S_IFREG); }static int dmemfs_mkdir(struct inode *dir, struct dentry *dentry,
umode_tmode){ return __create_file(dir, dentry, mode | S_IFDIR); }(i.e. even
inc_nlink()of parent folded into that).
Quoted 8 lines
[snip]
Quoted 3 lines
Yes, we seperate the full implementation for
dmemfs_file_mmapinto patch 05/35, it
will assign the interfaces to handle the page fault.It would be less confusing to move the introduction of ->mmap() to that patch, then.
Thanks for the suggestion. will refactor the patches accordingly.
PATCH 2/35 mm: support direct memory reservation
In reply to [email protected]From: Yulei Zhang <[email protected]>
Introduce 'dmem=' to reserve system memory for DMEM (direct memory), comparing with 'mem=' and 'memmap', it reserves memory based on the topology of NUMA, for the detailed info, please refer to kernel-parameters.txt
--- .../admin-guide/kernel-parameters.txt | 38 +++ arch/x86/kernel/setup.c | 3 + include/linux/dmem.h | 16 + mm/Kconfig | 9 + mm/Makefile | 1 + mm/dmem.c | 137 ++++++++ mm/dmem_reserve.c | 303 ++++++++++++++++++ 7 files changed, 507 insertions(+) create mode 100644 include/linux/dmem.h create mode 100644 mm/dmem.c create mode 100644 mm/dmem_reserve.c
Documentation/admin-guide/kernel-parameters.txt +38 -0
@@ -980,6 +980,44 @@ The filter can be disabled or changed to another driver later using sysfs. + dmem=[!]size[KMG] + [KNL, NUMA] When CONFIG_DMEM is set, this means + the size of memory reserved for dmemfs on each numa + memory node and 'size' must be aligned to the default + alignment that is the size of memory section which is + 128M on default on x86_64. If set '!', such amount of + memory on each node will be owned by kernel and dmemfs + own the rest of memory on each node. + Example: Reserve 4G memory on each node for dmemfs + dmem = 4G + + dmem=[!]size[KMG]:align[KMG] + [KNL, NUMA] Ditto. 'align' should be power of two and + it's not smaller than the default alignment. Also + 'size' must be aligned to 'align'. + Example: Bad dmem parameter because 'size' misaligned + dmem=0x40200000:1G + + dmem=size[KMG]@addr[KMG] + [KNL] When CONFIG_DMEM is set, this marks specific + memory as reserved for dmemfs. Region of memory will be + used by dmemfs, from addr to addr + size. Reserving a + certain memory region for kernel is illegal so '!' is + forbidden. Should not assign 'addr' to 0 because kernel + will occupy fixed memory region begin at 0 address. + Ditto, 'size' and 'addr' must be aligned to default + alignment. + Example: Exclude memory from 5G-6G for dmemfs. + dmem=1G@5G + + dmem=size[KMG]@addr[KMG]:align[KMG] + [KNL] Ditto. 'align' should be power of two and it's + not smaller than the default alignment. Also 'size' + and 'addr' must be aligned to 'align'. Specially, + '@addr' and ':align' could occur in any order. + Example: Exclude memory from 5G-6G for dmemfs. + dmem=1G:1G@5G + driver_async_probe= [KNL] List of driver names to be probed asynchronously. Format: <driver_name1>,<driver_name2>...
arch/x86/kernel/setup.c +3 -0
@@ -45,6 +45,7 @@#include <asm/unwind.h> #include <asm/vsyscall.h> #include <linux/vmalloc.h> +#include <linux/dmem.h> /* * max_low_pfn_mapped: highest directly mapped pfn < 4 GB
@@ -1177,6 +1178,8 @@ void __init setup_arch(char **cmdline_p)if (!early_xdbc_setup_hardware ())early_xdbc_register_console (); +dmem_reserve_init (); + x86_init.paging.pagetable_init ();kasan_init ();
include/linux/dmem.h new file +16 -0
@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _LINUX_DMEM_H +#define _LINUX_DMEM_H + +#ifdef CONFIG_DMEM +int dmem_reserve_init (void ); +void dmem_init (void ); +int dmem_region_register (int node,phys_addr_t start,phys_addr_t end); + +#else +static inline int dmem_reserve_init (void ) +{ +return 0 ; +} +#endif +#endif /* _LINUX_DMEM_H */
mm/Kconfig +9 -0
@@ -226,6 +226,15 @@ config BALLOON_COMPACTION scenario aforementioned and helps improving memory defragmentation. # +# support for direct memory basics +config DMEM + bool "Direct Memory Reservation" + def_bool n + depends on SPARSEMEM + help + Allow reservation of memory which could be dedicated usage of dmem. + It's the basics of dmemfs. + # support for memory compaction config COMPACTION bool "Allow for memory compaction"
mm/Makefile +1 -0
@@ -121,3 +121,4 @@ obj-$(CONFIG_MEMFD_CREATE) += memfd.o obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PAGE_REPORTING) += page_reporting.o +obj-$(CONFIG_DMEM) += dmem.o dmem_reserve.o
mm/dmem.c new file +136 -0
136 lines not on this page · raw message
mm/dmem_reserve.c new file +303 -0
303 lines not on this page · raw message
Signature · 1 line
2.28.0
Randy Dunlap
reviewerRe: [PATCH 02/35] mm: support direct memory reservation
mm/Kconfig config BALLOON_COMPACTION · line 232
Quoted 12 lines · On 10/8/20 12:53 AM, [email protected] wrote:
mm/Kconfig +4
-0@@ -226,6 +226,15 @@ config BALLOON_COMPACTION scenario aforementioned and helps improving memory defragmentation. # +# support for direct memory basics +config DMEM + bool "Direct Memory Reservation" + def_bool n
Drop the def_bool line.
mm/Kconfig config BALLOON_COMPACTION · line 235
Quoted 3 lines
mm/Kconfig +3
-0+ depends on SPARSEMEM + help + Allow reservation of memory which could be dedicated usage of dmem.
dedicated to the use of dmem.
or
which could be for the dedicated use of dmem.
mm/Kconfig config BALLOON_COMPACTION · line 236
Quoted 1 line
mm/Kconfig +1
-0+ It's the basics of dmemfs.
basis
Signature · 1 line
~Randy
Randy Dunlap
reviewerRe: [PATCH 02/35] mm: support direct memory reservation
In reply to [email protected]Documentation/admin-guide/kernel-parameters.txt line 985
Quoted 11 lines · On 10/8/20 12:53 AM, [email protected] wrote:
Documentation/admin-guide/kernel-parameters.txt +3
-0@@ -980,6 +980,44 @@ The filter can be disabled or changed to another driver later using sysfs. + dmem=[!]size[KMG] + [KNL, NUMA] When CONFIG_DMEM is set, this means + the size of memory reserved for dmemfs on each numa
NUMA
Documentation/admin-guide/kernel-parameters.txt line 988
Quoted 3 lines
Documentation/admin-guide/kernel-parameters.txt +3
-0+ memory node and 'size' must be aligned to the default + alignment that is the size of memory section which is + 128M on default on x86_64. If set '!', such amount of
by default
Documentation/admin-guide/kernel-parameters.txt line 990
Quoted 2 lines
Documentation/admin-guide/kernel-parameters.txt +2
-0+ memory on each node will be owned by kernel and dmemfs + own the rest of memory on each node.
owns
Documentation/admin-guide/kernel-parameters.txt line 992
Quoted 2 lines
Documentation/admin-guide/kernel-parameters.txt +2
-0+ Example: Reserve 4G memory on each node for dmemfs + dmem = 4G
IIRC, you don't want spaces in this example.
Or did you check? Does the kernel's command line parser accept & ignore spaces like these?
Documentation/admin-guide/kernel-parameters.txt line 996
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ + dmem=[!]size[KMG]:align[KMG] + [KNL, NUMA] Ditto. 'align' should be power of two and + it's not smaller than the default alignment. Also
drop "it's"
Documentation/admin-guide/kernel-parameters.txt line 1007
Quoted 11 lines
Documentation/admin-guide/kernel-parameters.txt +11
-0+ 'size' must be aligned to 'align'. + Example: Bad dmem parameter because 'size' misaligned + dmem=0x40200000:1G + + dmem=size[KMG]@addr[KMG] + [KNL] When CONFIG_DMEM is set, this marks specific + memory as reserved for dmemfs. Region of memory will be + used by dmemfs, from addr to addr + size. Reserving a + certain memory region for kernel is illegal so '!' is + forbidden. Should not assign 'addr' to 0 because kernel + will occupy fixed memory region begin at 0 address.
beginning
Documentation/admin-guide/kernel-parameters.txt line 1014
Quoted 7 lines
Documentation/admin-guide/kernel-parameters.txt +7
-0+ Ditto, 'size' and 'addr' must be aligned to default + alignment. + Example: Exclude memory from 5G-6G for dmemfs. + dmem=1G@5G + + dmem=size[KMG]@addr[KMG]:align[KMG] + [KNL] Ditto. 'align' should be power of two and it's
Drop "it's"
Quoted 9 lines
Documentation/admin-guide/kernel-parameters.txt +6
-0+ not smaller than the default alignment. Also 'size' + and 'addr' must be aligned to 'align'. Specially, + '@addr' and ':align' could occur in any order. + Example: Exclude memory from 5G-6G for dmemfs. + dmem=1G:1G@5G + driver_async_probe= [KNL] List of driver names to be probed asynchronously. Format: <driver_name1>,<driver_name2>...
Signature · 2 lines
~Randy Reported-by: Randy Dunlap <[email protected]>
PATCH 3/35 dmem: implement dmem memory management
In reply to [email protected]From: Yulei Zhang <[email protected]>
It introduces the interfaces to manage dmem pages that include:
- dmem_region_register(), it registers the reserved memory to the dmem management system, later it can be allocated out for dmemfs
- dmem_alloc_init(), initiate dmem allocator, note the page size the allocator used isn't the same thing with the alignment used to reserve dmem memory
- dmem_alloc_pages_from_vma() and dmem_free_pages() are the interfaces allocating and freeing dmem memory, multiple pages can be allocated at one time, but it should be power of two
--- include/linux/dmem.h | 3 + mm/dmem.c | 674 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 677 insertions(+)
include/linux/dmem.h +3 -0
@@ -7,6 +7,9 @@ int dmem_reserve_init(void);void dmem_init (void );int dmem_region_register (int node,phys_addr_t start,phys_addr_t end); +int dmem_alloc_init (unsigned long dpage_shift); +void dmem_alloc_uinit (void ); +#else static inline int dmem_reserve_init (void ) {
mm/dmem.c +673 -0
687 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH 4/35 dmem: let pat recognize dmem
In reply to [email protected]From: Yulei Zhang <[email protected]>
x86 pat uses 'struct page' by only checking if it's system ram, however it is not true if dmem is used, let's teach pat to recognize this case if it is ram but it is !pfn_valid()
We always use WB for dmem and any attempt to change this behavior will be rejected and WARN_ON is triggered
--- arch/x86/mm/pat/memtype.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
arch/x86/mm/pat/memtype.c +21 -0
@@ -511,6 +511,13 @@ static int reserve_ram_pages_type(u64 start, u64 end,for (pfn = (start >>PAGE_SHIFT ); pfn < (end >>PAGE_SHIFT ); ++pfn) {enum page_cache_mode type; +/* +* it's dmem if it's ram but not 'struct page' backend, +* we always use WB +*/ +if (WARN_ON (!pfn_valid (pfn))) +return -EBUSY ; + page =pfn_to_page (pfn); type =get_page_memtype (page);if (type != _PAGE_CACHE_MODE_WB) {
@@ -539,6 +546,13 @@ static int free_ram_pages_type(u64 start, u64 end)u64 pfn;for (pfn = (start >>PAGE_SHIFT ); pfn < (end >>PAGE_SHIFT ); ++pfn) { +/* +* it's dmem, see the comments in +* reserve_ram_pages_type() +*/ +if (WARN_ON (!pfn_valid (pfn))) +continue ; + page =pfn_to_page (pfn);set_page_memtype (page, _PAGE_CACHE_MODE_WB); }
@@ -714,6 +728,13 @@ static enum page_cache_mode lookup_memtype(u64 paddr)if (pat_pagerange_is_ram (paddr, paddr +PAGE_SIZE )) {struct page *page; +/* +* dmem always uses WB, see the comments in +* reserve_ram_pages_type() +*/ +if (!pfn_valid (paddr >>PAGE_SHIFT )) +return rettype; + page =pfn_to_page (paddr >>PAGE_SHIFT );return get_page_memtype (page); }
Signature · 1 line
2.28.0
Paolo Bonzini
maintainerRe: [PATCH 04/35] dmem: let pat recognize dmem
Quoted 11 lines · On 08/10/20 09:53, [email protected] wrote:
From: Yulei Zhang <[email protected]>
x86 pat uses 'struct page' by only checking if it's system ram, however it is not true if dmem is used, let's teach pat to recognize this case if it is ram but it is !pfn_valid()
We always use WB for dmem and any attempt to change this behavior will be rejected and
WARN_ONis triggeredSigned-off-by Xiao Guangrong <[email protected]>Signed-off-by Yulei Zhang <[email protected]>
Hooks like these will make it very hard to merge this series.
I like the idea of struct page-backed memory, but this is a lot of code and I wonder if it's worth adding all these complications.
One can already use mem= to remove the "struct page" cost for most of the host memory, and manage the allocation of the remaining memory in userspace with /dev/mem. What is the advantage of doing this in the kernel?
Paolo
Re: [PATCH 04/35] dmem: let pat recognize dmem
Quoted 16 lines · Paolo Bonzini
Quoted 11 lines · On 08/10/20 09:53, [email protected] wrote:
From: Yulei Zhang <[email protected]>
x86 pat uses 'struct page' by only checking if it's system ram, however it is not true if dmem is used, let's teach pat to recognize this case if it is ram but it is !pfn_valid()
We always use WB for dmem and any attempt to change this behavior will be rejected and
WARN_ONis triggeredSigned-off-by Xiao Guangrong <[email protected]>Signed-off-by Yulei Zhang <[email protected]>Hooks like these will make it very hard to merge this series.
Quoted 9 lines
I like the idea of struct page-backed memory, but this is a lot of code and I wonder if it's worth adding all these complications.
One can already use mem= to remove the "struct page" cost for most of the host memory, and manage the allocation of the remaining memory in userspace with
/dev/mem. What is the advantage of doing this in the kernel?Paolo
hi Paolo,as far as I know there are a few limitations to play with /dev/mem in this case.
1. access to /dev/men is restricted due to the security requirement, but usually our virtual machines are unprivileged processes.
2. what we get from /dev/mem is a whole block of memory, as dynamic VMs running on /dev/mem will cause memory fragment, it needs extra logic to manage the allocation and recovery to avoid wasted memory. dmemfs can support this and also leverage the kernel tlb management.
3. it needs to support hugepage with different page size granularity.
4. MCE recovery capability is also required.
PATCH 5/35 dmemfs: support mmap
In reply to [email protected]From: Yulei Zhang <[email protected]>
It adds mmap support. Note the file will be extended if it's beyond mmap's offset, that drops the requirement of write() operation, however, it has not supported cutting file down.
--- fs/dmemfs/inode.c | 337 ++++++++++++++++++++++++++++++++++++++++++- include/linux/dmem.h | 10 ++ 2 files changed, 345 insertions(+), 2 deletions(-)
fs/dmemfs/inode.c +335 -2
@@ -26,6 +26,7 @@#include <linux/pagevec.h> #include <linux/fs_parser.h> #include <linux/seq_file.h> +#include <linux/dmem.h> MODULE_AUTHOR ("Tencent Corporation" );MODULE_LICENSE ("GPL v2" );
@@ -105,8 +106,250 @@ static const struct inode_operations dmemfs_file_inode_operations = { .getattr = simple_getattr, }; +static unsigned long dmem_pgoff_to_index (struct inode *inode,pgoff_t pgoff) +{ +struct super_block *sb = inode->i_sb; + +return pgoff >> (sb->s_blocksize_bits -PAGE_SHIFT ); +} + +static void *dmem_addr_to_entry (struct inode *inode,phys_addr_t addr) +{ +struct super_block *sb = inode->i_sb; + + addr >>= sb->s_blocksize_bits; +return xa_mk_value (addr); +} + +static phys_addr_t dmem_entry_to_addr (struct inode *inode,void *entry) +{ +struct super_block *sb = inode->i_sb; + +WARN_ON (!xa_is_value (entry)); +return xa_to_value (entry) << sb->s_blocksize_bits; +} + +static unsigned long +dmem_addr_to_pfn (struct inode *inode,phys_addr_t addr,pgoff_t pgoff, +unsigned int fault_shift) +{ +struct super_block *sb = inode->i_sb; +unsigned long pfn = addr >>PAGE_SHIFT ; +unsigned long mask; + + mask = (1UL << ((unsigned int )sb->s_blocksize_bits - fault_shift)) -1 ; + mask <<= fault_shift -PAGE_SHIFT ; + +return pfn + (pgoff & mask); +} + +static inline unsigned long dmem_page_size (struct inode *inode) +{ +return inode->i_sb->s_blocksize; +} + +static int check_inode_size (struct inode *inode,loff_t offset) +{ +WARN_ON_ONCE (!rcu_read_lock_held ()); + +if (offset >=i_size_read (inode)) +return -EINVAL ; + +return 0 ; +} + +static unsigned +dmemfs_find_get_entries (struct address_space *mapping,unsigned long start, +unsigned int nr_entries,void **entries, +unsigned long *indices) +{ +XA_STATE (xas, &mapping->i_pages, start); + +void *entry; +unsigned int ret =0 ; + +if (!nr_entries) +return 0 ; + +rcu_read_lock (); + +xas_for_each (&xas, entry,ULONG_MAX ) { +if (xas_retry (&xas, entry)) +continue ; + +if (xa_is_value (entry)) + goto export; + +if (unlikely (entry !=xas_reload (&xas))) + goto retry; + +export: + indices[ret] = xas.xa_index; + entries[ret] = entry; +if (++ret == nr_entries) +break ; +continue ; +retry: +xas_reset (&xas); + } +rcu_read_unlock (); +return ret; +} + +static void *find_radix_entry_or_next (struct address_space *mapping, +unsigned long start, +unsigned long *eindex) +{ +void *entry =NULL ; + +dmemfs_find_get_entries (mapping, start,1 , &entry, eindex); +return entry; +} + +/* +* find the entry in radix tree based on @index, create it if +* it does not exist +* +* return the entry with rcu locked, otherwise ERR_PTR() +* is returned +*/ +static void * +radix_get_create_entry (struct vm_area_struct *vma,unsigned long fault_addr, +struct inode *inode,pgoff_t pgoff) +{ +struct address_space *mapping = inode->i_mapping; +unsigned long eindex, index; +loff_t offset; +phys_addr_t addr; +gfp_t gfp_masks =mapping_gfp_mask (mapping) & ~__GFP_HIGHMEM; +void *entry; +unsigned int try_dpages, dpages; +int ret; + +retry: + offset = ((loff_t )pgoff <<PAGE_SHIFT ); + index =dmem_pgoff_to_index (inode, pgoff); +rcu_read_lock (); + ret =check_inode_size (inode, offset); +if (ret) { +rcu_read_unlock (); +return ERR_PTR (ret); + } + + try_dpages =dmem_pgoff_to_index (inode, (i_size_read (inode) - offset) + >>PAGE_SHIFT ); + entry =find_radix_entry_or_next (mapping, index, &eindex); +if (entry) { +WARN_ON (!xa_is_value (entry)); +if (eindex == index) +return entry; + +WARN_ON (eindex <= index); + try_dpages = eindex - index; + } +rcu_read_unlock (); + +/* entry does not exist, create it */ + addr =dmem_alloc_pages_vma (vma, fault_addr, try_dpages, &dpages); +if (!addr) { +/* +* do not return -ENOMEM as that will trigger OOM, +* it is useless for reclaiming dmem page +*/ + ret = -EINVAL ; + goto exit; + } + + try_dpages = dpages; +while (dpages) { +rcu_read_lock (); + ret =check_inode_size (inode, offset); +if (ret) + goto unlock_rcu; + + entry =dmem_addr_to_entry (inode, addr); + entry =xa_store (&mapping->i_pages, index, entry, gfp_masks); +if (!xa_is_err (entry)) { + addr += inode->i_sb->s_blocksize; + offset += inode->i_sb->s_blocksize; + dpages--; + mapping->nrexceptional++; + index++; + } + +unlock_rcu: +rcu_read_unlock (); +if (ret) +break ; + } + +if (dpages) +dmem_free_pages (addr, dpages); + +/* we have created some entries, let's retry it */ +if (ret == -EEXIST || try_dpages != dpages) + goto retry; +exit: +return ERR_PTR (ret); +} + +static void radix_put_entry (void ) +{ +rcu_read_unlock (); +} + +static vm_fault_t dmemfs_fault (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +struct inode *inode =file_inode (vma->vm_file); +phys_addr_t addr; +void *entry; +int ret; + +if (vmf->pgoff > (MAX_LFS_FILESIZE >>PAGE_SHIFT )) +return VM_FAULT_SIGBUS ; + + entry =radix_get_create_entry (vma, (unsigned long )vmf->address, + inode, vmf->pgoff); +if (IS_ERR (entry)) { + ret =PTR_ERR (entry); + goto exit; + } + + addr =dmem_entry_to_addr (inode, entry); + ret =vmf_insert_pfn (vma, (unsigned long )vmf->address, +dmem_addr_to_pfn (inode, addr, vmf->pgoff, +PAGE_SHIFT )); +radix_put_entry (); + +exit: +return ret; +} + +static unsigned long dmemfs_pagesize (struct vm_area_struct *vma) +{ +return dmem_page_size (file_inode (vma->vm_file)); +} + +static const struct vm_operations_struct dmemfs_vm_ops = { + .fault = dmemfs_fault, + .pagesize = dmemfs_pagesize, +}; +int dmemfs_file_mmap (struct file *file,struct vm_area_struct *vma) { +struct inode *inode =file_inode (file); + +if (vma->vm_pgoff & ((dmem_page_size (inode) -1 ) >>PAGE_SHIFT )) +return -EINVAL ; + +if (!(vma->vm_flags &VM_SHARED )) +return -EINVAL ; + + vma->vm_flags |=VM_PFNMAP ; + +file_accessed (file); + vma->vm_ops = &dmemfs_vm_ops;return 0 ; }
@@ -189,9 +432,86 @@ static int dmemfs_statfs(struct dentry *dentry, struct kstatfs *buf)return 0 ; } +/* +* should make sure the dmem page in the dropped region is not +* being mapped by any process +*/ +static void inode_drop_dpages (struct inode *inode,loff_t start,loff_t end) +{ +struct address_space *mapping = inode->i_mapping; +struct pagevec pvec; +unsigned long istart, iend, indices[PAGEVEC_SIZE ]; +int i; + +/* we never use normap page */ +WARN_ON (mapping->nrpages); + +/* if no dpage is allocated for the inode */ +if (!mapping->nrexceptional) +return ; + + istart =dmem_pgoff_to_index (inode, start >>PAGE_SHIFT ); + iend =dmem_pgoff_to_index (inode, end >>PAGE_SHIFT ); +pagevec_init (&pvec); +while (istart < iend) { + pvec.nr =dmemfs_find_get_entries (mapping, istart, +min (iend - istart, + (unsigned long )PAGEVEC_SIZE ), + (void **)pvec.pages, + indices); +if (!pvec.nr) +break ; + +for (i =0 ; i <pagevec_count (&pvec); i++) { +phys_addr_t addr; + + istart = indices[i]; +if (istart >= iend) +break ; + +xa_erase (&mapping->i_pages, istart); + mapping->nrexceptional--; + + addr =dmem_entry_to_addr (inode, pvec.pages[i]); +dmem_free_page (addr); + } + +/* +* only exception entries in pagevec, it's safe to +* reinit it +*/ +pagevec_reinit (&pvec); +cond_resched (); + istart++; + } +} + +static void dmemfs_evict_inode (struct inode *inode) +{ +/* no VMA works on it */ +WARN_ON (!RB_EMPTY_ROOT (&inode->i_data.i_mmap.rb_root)); + +inode_drop_dpages (inode,0 ,LLONG_MAX ); +clear_inode (inode); +} + +/* +* Display the mount options in /proc/mounts. +*/ +static int dmemfs_show_options (struct seq_file *m,struct dentry *root) +{ +struct dmemfs_fs_info *fsi = root->d_sb->s_fs_info; + +if (check_dpage_size (fsi->mount_opts.dpage_size)) +seq_printf (m,",pagesize=%lx" , fsi->mount_opts.dpage_size); +return 0 ; +} +static const struct super_operations dmemfs_ops = { .statfs = dmemfs_statfs, + .evict_inode = dmemfs_evict_inode, .drop_inode = generic_delete_inode, + .show_options = dmemfs_show_options, }; static int
@@ -199,6 +519,7 @@ dmemfs_fill_super(struct super_block *sb, struct fs_context *fc) {struct inode *inode;struct dmemfs_fs_info *fsi = sb->s_fs_info; +int ret; sb->s_maxbytes =MAX_LFS_FILESIZE ; sb->s_blocksize = fsi->mount_opts.dpage_size;
@@ -207,11 +528,17 @@ dmemfs_fill_super(struct super_block *sb, struct fs_context *fc) sb->s_op = &dmemfs_ops; sb->s_time_gran =1 ; + ret =dmem_alloc_init (sb->s_blocksize_bits); +if (ret) +return ret; + inode =dmemfs_get_inode (sb,NULL ,S_IFDIR ,0 ); sb->s_root =d_make_root (inode); -if (!sb->s_root) -return -ENOMEM ; +if (!sb->s_root) { +dmem_alloc_uinit (); +return -ENOMEM ; + }return 0 ; }
@@ -247,7 +574,13 @@ int dmemfs_init_fs_context(struct fs_context *fc)static void dmemfs_kill_sb (struct super_block *sb) { +bool has_inode = !!sb->s_root; +kill_litter_super (sb); + +/* do not uninit dmem allocator if mount failed */ +if (has_inode) +dmem_alloc_uinit (); }static struct file_system_type dmemfs_fs_type = {
include/linux/dmem.h +10 -0
@@ -10,6 +10,16 @@ int dmem_region_register(int node, phys_addr_t start, phys_addr_t end);int dmem_alloc_init (unsigned long dpage_shift);void dmem_alloc_uinit (void ); +phys_addr_t +dmem_alloc_pages_nodemask (int nid,nodemask_t *nodemask,unsigned int try_max, +unsigned int *result_nr); + +phys_addr_t +dmem_alloc_pages_vma (struct vm_area_struct *vma,unsigned long addr, +unsigned int try_max,unsigned int *result_nr); + +void dmem_free_pages (phys_addr_t addr,unsigned int dpages_nr); +#define dmem_free_page (addr) dmem_free_pages(addr, 1)#else static inline int dmem_reserve_init (void ) {
Signature · 1 line
2.28.0
PATCH 6/35 dmemfs: support truncating inode down
In reply to [email protected]From: Yulei Zhang <[email protected]>
To support cut inode down, it will
introduce the race between page fault handler and truncating handler as the entry to be deleted is being mapped into process's VMA
in order to make page fault faster (as it's the hot path), we use RCU to sync these two handlers. When inode's size is updated, the handler makes sure the new size is visible to page fault handler who will not use truncated entry anymore and will not create new entry in that region
--- fs/dmemfs/inode.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 66 insertions(+), 1 deletion(-)
fs/dmemfs/inode.c +66 -1
@@ -101,8 +101,73 @@ static const struct inode_operations dmemfs_dir_inode_operations = { .rename = simple_rename, }; +static void inode_drop_dpages (struct inode *inode,loff_t start,loff_t end); + +static int dmemfs_truncate (struct inode *inode,loff_t newsize) +{ +struct super_block *sb = inode->i_sb; +loff_t current_size; + +if (newsize & ((1 << sb->s_blocksize_bits) -1 )) +return -EINVAL ; + + current_size =i_size_read (inode); +i_size_write (inode, newsize); + +if (newsize >= current_size) +return 0 ; + +/* it cuts the inode down */ + +/* +* we should make sure inode->i_size has been updated before +* unmapping and dropping radix entries, so that other sides +* can not create new i_mapping entry beyond inode->i_size +* and the radix entry in the truncated region is not being +* used +* +* see the comments in dmemfs_fault() +*/ +synchronize_rcu (); + +/* +* should unmap all mapping first as dmem pages are freed in +* inode_drop_dpages() +* +* after that, dmem page in the truncated region is not used +* by any process +*/ +unmap_mapping_range (inode->i_mapping, newsize,0 ,1 ); + +inode_drop_dpages (inode, newsize,LLONG_MAX ); +return 0 ; +} + +/* +* same logic as simple_setattr but we need to handle ftruncate +* carefully as we inserted self-defined entry into radix tree +*/ +static int dmemfs_setattr (struct dentry *dentry,struct iattr *iattr) +{ +struct inode *inode = dentry->d_inode; +int error; + + error =setattr_prepare (dentry, iattr); +if (error) +return error; + +if (iattr->ia_valid &ATTR_SIZE ) { + error =dmemfs_truncate (inode, iattr->ia_size); +if (error) +return error; + } +setattr_copy (inode, iattr); +mark_inode_dirty (inode); +return 0 ; +} +static const struct inode_operations dmemfs_file_inode_operations = { - .setattr = simple_setattr, + .setattr = dmemfs_setattr, .getattr = simple_getattr, };
Signature · 1 line
2.28.0
PATCH 7/35 dmem: trace core functions
In reply to [email protected]From: Yulei Zhang <[email protected]>
Add tracepoints for alloc_init, alloc and free functions, that helps us to figure out what is happening inside dmem allocator
--- fs/dmemfs/Makefile | 1 + fs/dmemfs/inode.c | 5 +++ fs/dmemfs/trace.h | 54 +++++++++++++++++++++++++++++ include/trace/events/dmem.h | 68 +++++++++++++++++++++++++++++++++++++ mm/dmem.c | 6 ++++ 5 files changed, 134 insertions(+) create mode 100644 fs/dmemfs/trace.h create mode 100644 include/trace/events/dmem.h
fs/dmemfs/Makefile +1 -0
@@ -2,6 +2,7 @@ # # Makefile for the linux dmem-filesystem routines. # +ccflags-y += -I $(srctree)/$(src) # needed for trace events obj-$(CONFIG_DMEM_FS) += dmemfs.o dmemfs-y += inode.o
fs/dmemfs/inode.c +5 -0
@@ -31,6 +31,9 @@MODULE_AUTHOR ("Tencent Corporation" );MODULE_LICENSE ("GPL v2" ); +#define CREATE_TRACE_POINTS +#include "trace.h" +struct dmemfs_mount_opts {unsigned long dpage_size; };
@@ -339,6 +342,7 @@ radix_get_create_entry(struct vm_area_struct *vma, unsigned long fault_addr, offset += inode->i_sb->s_blocksize; dpages--; mapping->nrexceptional++; +trace_dmemfs_radix_tree_insert (index, entry); index++; }
@@ -535,6 +539,7 @@ static void inode_drop_dpages(struct inode *inode, loff_t start, loff_t end)break ;xa_erase (&mapping->i_pages, istart); +trace_dmemfs_radix_tree_delete (istart, pvec.pages[i]); mapping->nrexceptional--; addr =dmem_entry_to_addr (inode, pvec.pages[i]);
fs/dmemfs/trace.h new file +54 -0
54 lines not on this page · raw message
include/trace/events/dmem.h new file +68 -0
68 lines not on this page · raw message
mm/dmem.c +6 -0
@@ -18,6 +18,8 @@#include <linux/debugfs.h> #include <linux/notifier.h> +#define CREATE_TRACE_POINTS +#include <trace/events/dmem.h> /* * There are two kinds of page in dmem management: * - nature page, it's the CPU's page size, i.e, 4K on x86
@@ -559,6 +561,8 @@ int dmem_alloc_init(unsigned long dpage_shift)mutex_lock (&dmem_pool.lock); +trace_dmem_alloc_init (dpage_shift); +if (dmem_pool.dpage_shift) {/* * double init on the same page size is okay
@@ -686,6 +690,7 @@ dmem_alloc_pages_from_nodelist(int *nodelist, nodemask_t *nodemask, } } +trace_dmem_alloc_pages_node (addr, node, try_max, *result_nr);mutex_unlock (&dmem_pool.lock); }return addr;
@@ -791,6 +796,7 @@ void dmem_free_pages(phys_addr_t addr, unsigned int dpages_nr)mutex_lock (&dmem_pool.lock); +trace_dmem_free_pages (addr, dpages_nr);WARN_ON (!dmem_pool.dpage_shift); dregion =find_dmem_region (addr, &pdnode);
Signature · 1 line
2.28.0
PATCH 8/35 dmem: show some statistic in debugfs
In reply to [email protected]From: Yulei Zhang <[email protected]>
Create 'dmem' directory under debugfs and show some statistic for dmem pool, track total and free dpages on dmem pool and each numa node.
--- mm/Kconfig | 9 +++++ mm/dmem.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 108 insertions(+), 1 deletion(-)
mm/Kconfig +9 -0
@@ -235,6 +235,15 @@ config DMEM Allow reservation of memory which could be dedicated usage of dmem. It's the basics of dmemfs. +config DMEM_DEBUG_FS + bool "Enable debug information for direct memory" + depends on DMEM && DEBUG_FS + def_bool n + help + This option enables showing various statistics of direct memory + in debugfs filesystem. + +# # support for memory compaction config COMPACTION bool "Allow for memory compaction"
mm/dmem.c +99 -1
112 lines not on this page · raw message
Signature · 1 line
2.28.0
Randy Dunlap
reviewerRe: [PATCH 08/35] dmem: show some statistic in debugfs
mm/Kconfig config DMEM · line 241
Quoted 12 lines · On 10/8/20 12:53 AM, [email protected] wrote:
mm/Kconfig +4
-0@@ -235,6 +235,15 @@ config DMEM Allow reservation of memory which could be dedicated usage of dmem. It's the basics of dmemfs. +config DMEM_DEBUG_FS + bool "Enable debug information for direct memory" + depends on DMEM && DEBUG_FS + def_bool n
Drop the def_bool line. 'n' is the default anyway and the symbol is already of type bool from 2 lines above.
Quoted 5 lines
mm/Kconfig +5
-0+ help + This option enables showing various statistics of direct memory + in debugfs filesystem. + +#
Signature · 1 line
~Randy
Re: [PATCH 08/35] dmem: show some statistic in debugfs
Thanks, Randy. I will follow the instructions to modify the patches.
Quoted 28 lines · Randy Dunlap
mm/Kconfig config DMEM · line 241
Quoted 12 lines · On 10/8/20 12:53 AM, [email protected] wrote:
mm/Kconfig +4
-0@@ -235,6 +235,15 @@ config DMEM Allow reservation of memory which could be dedicated usage of dmem. It's the basics of dmemfs. +config DMEM_DEBUG_FS + bool "Enable debug information for direct memory" + depends on DMEM && DEBUG_FS + def_bool nDrop the
def_boolline. 'n' is the default anyway and the symbol is already of type bool from 2 lines above.mm/Kconfig config DMEM · line 246
Quoted 5 lines
mm/Kconfig +5
-0+ help + This option enables showing various statistics of direct memory + in debugfs filesystem. + +#--
~Randy
PATCH 9/35 dmemfs: support remote access
In reply to [email protected]From: Yulei Zhang <[email protected]>
It is required by ptrace_writedata and ptrace_readdata to access dmem memory remotely. The typical user is gdb, after this patch, gdb is able to read & write memory owned by the attached process
--- fs/dmemfs/inode.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+)
fs/dmemfs/inode.c +46 -0
@@ -367,6 +367,51 @@ static void radix_put_entry(void)rcu_read_unlock (); } +static bool check_vma_access (struct vm_area_struct *vma,int write) +{ +vm_flags_t vm_flags = write ?VM_WRITE :VM_READ ; + +return !!(vm_flags & vma->vm_flags); +} + +static int +dmemfs_access_dmem (struct vm_area_struct *vma,unsigned long addr, +void *buf,int len,int write) +{ +struct inode *inode =file_inode (vma->vm_file); +struct super_block *sb = inode->i_sb; +void *entry, *maddr; +int offset, pgoff; + +if (!check_vma_access (vma, write)) +return -EACCES ; + + pgoff =linear_page_index (vma, addr); +if (pgoff > (MAX_LFS_FILESIZE >>PAGE_SHIFT )) +return -EFAULT ; + + entry =radix_get_create_entry (vma, addr, inode, pgoff); +if (IS_ERR (entry)) +return PTR_ERR (entry); + + offset = addr & (sb->s_blocksize -1 ); + addr =dmem_entry_to_addr (inode, entry); + +/* +* it is not beyond vma's region as the vma should be aligned +* to blocksize +*/ + len =min (len, (int )(sb->s_blocksize - offset)); + maddr =__va (addr); +if (write) +memcpy (maddr + offset, buf, len); +else +memcpy (buf, maddr + offset, len); +radix_put_entry (); + +return len; +} +static vm_fault_t dmemfs_fault (struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma;
@@ -403,6 +448,7 @@ static unsigned long dmemfs_pagesize(struct vm_area_struct *vma)static const struct vm_operations_struct dmemfs_vm_ops = { .fault = dmemfs_fault, .pagesize = dmemfs_pagesize, + .access = dmemfs_access_dmem, };int dmemfs_file_mmap (struct file *file,struct vm_area_struct *vma)
Signature · 1 line
2.28.0
PATCH 10/35 dmemfs: introduce max_alloc_try_dpages parameter
In reply to [email protected]From: Yulei Zhang <[email protected]>
It specifies the dmem page number allocated at one time, then multiple radix entries can be created. That will relief the allocation pressure and make page fault more fast.
However that could cause no dmem page mmapped to userspace even if there are some free dmem pages.
Set it to 1 to completely disable this behavior.
--- fs/dmemfs/inode.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
fs/dmemfs/inode.c +41 -0
@@ -34,6 +34,8 @@ MODULE_LICENSE("GPL v2");#define CREATE_TRACE_POINTS #include "trace.h" +static uint __read_mostly max_alloc_try_dpages =1 ; +struct dmemfs_mount_opts {unsigned long dpage_size; };
@@ -46,6 +48,44 @@ enum dmemfs_param { Opt_dpagesize, }; +static int +max_alloc_try_dpages_set (const char *val,const struct kernel_param *kp) +{ +uint sval; +int ret; + + ret =kstrtouint (val,0 , &sval); +if (ret) +return ret; + +/* should be 1 at least */ +if (!sval) +return -EINVAL ; + + max_alloc_try_dpages = sval; +return 0 ; +} + +static struct kernel_param_ops alloc_max_try_dpages_ops = { + .set = max_alloc_try_dpages_set, + .get = param_get_uint, +}; + +/* +* it specifies the dmem page number allocated at one time, then +* multiple radix entries can be created. That will relief the +* allocation pressure and make page fault more fast. +* +* however that could cause no dmem page mmapped to userspace +* even if there are some free dmem pages +* +* set it to 1 to completely disable this behavior +*/ +fs_param_cb (max_alloc_try_dpages, &alloc_max_try_dpages_ops, + &max_alloc_try_dpages,0644 ); +__MODULE_PARM_TYPE (max_alloc_try_dpages,"uint" ); +MODULE_PARM_DESC (max_alloc_try_dpages,"Set the dmem page number allocated at one time, should be 1 at least" ); +const struct fs_parameter_spec dmemfs_fs_parameters[] = {fsparam_string ("pagesize" , Opt_dpagesize), {}
@@ -317,6 +357,7 @@ radix_get_create_entry(struct vm_area_struct *vma, unsigned long fault_addr, }rcu_read_unlock (); + try_dpages =min (try_dpages, max_alloc_try_dpages);/* entry does not exist, create it */ addr =dmem_alloc_pages_vma (vma, fault_addr, try_dpages, &dpages);if (!addr) {
Signature · 1 line
2.28.0
PATCH 11/35 mm: export mempolicy interfaces to serve dmem allocator
In reply to [email protected]From: Yulei Zhang <[email protected]>
Export interface interleave_nid() to serve dmem allocator.
--- include/linux/mempolicy.h | 3 +++ mm/mempolicy.c | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-)
include/linux/mempolicy.h +3 -0
@@ -139,6 +139,9 @@ struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,struct mempolicy *get_task_policy (struct task_struct *p);struct mempolicy *__get_vma_policy (struct vm_area_struct *vma,unsigned long addr); +struct mempolicy *get_vma_policy (struct vm_area_struct *vma,unsigned long addr); +unsigned interleave_nid (struct mempolicy *pol,struct vm_area_struct *vma, +unsigned long addr,int shift);bool vma_policy_mof (struct vm_area_struct *vma);extern void numa_default_policy (void );
mm/mempolicy.c +2 -2
@@ -1816,7 +1816,7 @@ struct mempolicy *__get_vma_policy(struct vm_area_struct *vma,* freeing by another task. It is the caller's responsibility to free the * extra reference for shared policies. */ -static struct mempolicy *get_vma_policy (struct vm_area_struct *vma, +struct mempolicy *get_vma_policy (struct vm_area_struct *vma,unsigned long addr) {struct mempolicy *pol =__get_vma_policy (vma, addr);
@@ -1982,7 +1982,7 @@ static unsigned offset_il_node(struct mempolicy *pol, unsigned long n) }/* Determine a node number for interleave */ -static inline unsigned interleave_nid (struct mempolicy *pol, +unsigned interleave_nid (struct mempolicy *pol,struct vm_area_struct *vma,unsigned long addr,int shift) {if (vma) {
Signature · 1 line
2.28.0
PATCH 12/35 dmem: introduce mempolicy support
In reply to [email protected]From: Yulei Zhang <[email protected]>
It adds mempolicy support for dmem to allocates memory from mempolicy specified nodes.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 7 ++++ arch/x86/include/asm/pgtable_types.h | 13 +++++- fs/dmemfs/Kconfig | 3 ++ include/linux/pgtable.h | 7 ++++ mm/Kconfig | 3 ++ mm/dmem.c | 63 +++++++++++++++++++++++++++- 7 files changed, 94 insertions(+), 3 deletions(-)
arch/x86/Kconfig +1 -0
@@ -73,6 +73,7 @@ config X86 select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE select ARCH_HAS_PMEM_API if X86_64 select ARCH_HAS_PTE_DEVMAP if X86_64 + select ARCH_HAS_PTE_DMEM if X86_64 select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_UACCESS_MCSAFE if X86_64 && X86_MCE
arch/x86/include/asm/pgtable.h +7 -0
@@ -453,6 +453,13 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd)return pmd_set_flags (pmd, _PAGE_DEVMAP); } +#ifdef CONFIG_ARCH_HAS_PTE_DMEM +static inline pmd_t pmd_mkdmem (pmd_t pmd) +{ +return pmd_set_flags (pmd, _PAGE_SPECIAL | _PAGE_DMEM); +} +#endif +static inline pmd_t pmd_mkhuge (pmd_t pmd) {return pmd_set_flags (pmd, _PAGE_PSE);
arch/x86/include/asm/pgtable_types.h +12 -1
@@ -23,6 +23,15 @@#define _PAGE_BIT_SOFTW2 10/* " */ #define _PAGE_BIT_SOFTW3 11/* " */ #define _PAGE_BIT_PAT_LARGE 12/* On 2MB or 1GB pages */ +#define _PAGE_BIT_DMEM 57/* Flag used to indicate dmem pmd. +* Since _PAGE_BIT_SPECIAL is defined +* same as _PAGE_BIT_CPA_TEST, we can +* not only use _PAGE_BIT_SPECIAL, so +* add _PAGE_BIT_DMEM to help +* indicate it. Since dmem pte will +* never be splitting, setting +* _PAGE_BIT_SPECIAL for pte is enough. +*/ #define _PAGE_BIT_SOFTW4 58/* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59/* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60/* Protection Keys, bit 2/4 */
@@ -112,9 +121,11 @@#if defined(CONFIG_X86_64 ) || defined(CONFIG_X86_PAE )#define _PAGE_NX (_AT(pteval_t, 1) << _PAGE_BIT_NX)#define _PAGE_DEVMAP (_AT(u64, 1) << _PAGE_BIT_DEVMAP) +#define _PAGE_DMEM (_AT(u64, 1) << _PAGE_BIT_DMEM)#else #define _PAGE_NX (_AT(pteval_t, 0))#define _PAGE_DEVMAP (_AT(pteval_t, 0)) +#define _PAGE_DMEM (_AT(pteval_t, 0))#endif #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE)
@@ -128,7 +139,7 @@#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY | \ _PAGE_SOFT_DIRTY | _PAGE_DEVMAP | _PAGE_ENC | \ - _PAGE_UFFD_WP) + _PAGE_UFFD_WP | _PAGE_DMEM)#define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE)/*
fs/dmemfs/Kconfig +3 -0
@@ -1,5 +1,8 @@ config DMEM_FS tristate "Direct Memory filesystem support" + depends on DMEM + depends on TRANSPARENT_HUGEPAGE + depends on ARCH_HAS_PTE_DMEM help dmemfs (Direct Memory filesystem) is device memory or reserved memory based filesystem. This kind of memory is special as it
include/linux/pgtable.h +7 -0
@@ -1129,6 +1129,13 @@ static inline int pud_trans_unstable(pud_t *pud)#endif } +#ifndef CONFIG_ARCH_HAS_PTE_DMEM +static inline pmd_t pmd_mkdmem (pmd_t pmd) +{ +return pmd; +} +#endif +#ifndef pmd_read_atomicstatic inline pmd_t pmd_read_atomic (pmd_t *pmdp) {
mm/Kconfig +3 -0
@@ -795,6 +795,9 @@ config IDLE_PAGE_TRACKING config ARCH_HAS_PTE_DEVMAP bool +config ARCH_HAS_PTE_DMEM + bool + config ZONE_DEVICE bool "Device memory (pmem, HMM, etc...) hotplug support" depends on MEMORY_HOTPLUG
mm/dmem.c +61 -2
86 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH 13/35 mm, dmem: introduce PFN_DMEM and pfn_t_dmem
In reply to [email protected]From: Yulei Zhang <[email protected]>
Introduce PFN_DMEM as a new pfn flag for dmem pfn, define it by setting (BITS_PER_LONG_LONG - 6) bit.
Introduce pfn_t_dmem() helper to recognize dmem pfn.
--- include/linux/pfn_t.h | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
include/linux/pfn_t.h +16 -1
@@ -11,6 +11,7 @@* PFN_MAP - pfn has a dynamic page mapping established by a device driver * PFN_SPECIAL - for CONFIG_FS_DAX_LIMITED builds to allow XIP, but not * get_user_pages +* PFN_DMEM - pfn references a dmem page */ #define PFN_FLAGS_MASK (((u64) (~PAGE_MASK)) << (BITS_PER_LONG_LONG - PAGE_SHIFT))#define PFN_SG_CHAIN (1ULL << (BITS_PER_LONG_LONG - 1))
@@ -18,13 +19,15 @@#define PFN_DEV (1ULL << (BITS_PER_LONG_LONG - 3))#define PFN_MAP (1ULL << (BITS_PER_LONG_LONG - 4))#define PFN_SPECIAL (1ULL << (BITS_PER_LONG_LONG - 5)) +#define PFN_DMEM (1ULL << (BITS_PER_LONG_LONG - 6))#define PFN_FLAGS_TRACE \ { PFN_SPECIAL, "SPECIAL" }, \ { PFN_SG_CHAIN, "SG_CHAIN" }, \ { PFN_SG_LAST, "SG_LAST" }, \ { PFN_DEV, "DEV" }, \ - { PFN_MAP, "MAP" } + { PFN_MAP, "MAP" }, \ + { PFN_DMEM, "DMEM" }static inline pfn_t __pfn_to_pfn_t (unsigned long pfn,u64 flags) {
@@ -128,4 +131,16 @@ static inline bool pfn_t_special(pfn_t pfn)return false; }#endif /* CONFIG_ARCH_HAS_PTE_SPECIAL */ + +#ifdef CONFIG_ARCH_HAS_PTE_DMEM +static inline bool pfn_t_dmem (pfn_t pfn) +{ +return (pfn.val &PFN_DMEM ) ==PFN_DMEM ; +} +#else +static inline bool pfn_t_dmem (pfn_t pfn) +{ +return false; +} +#endif /* CONFIG_ARCH_HAS_PTE_DMEM */ #endif /* _LINUX_PFN_T_H_ */
Signature · 1 line
2.28.0
PATCH 14/35 mm, dmem: dmem-pmd vs thp-pmd
In reply to [email protected]From: Yulei Zhang <[email protected]>
A dmem huge page is ultimately not a transparent huge page. As we decided to use pmd_special() to distinguish dmem-pmd from thp-pmd, we should make some slightly different semantics between pmd_special() and pmd_trans_huge(), just as pmd_devmap() in upstream. This distinction is especially important in some mm-core paths such as zap_pmd_range().
Explicitly mark the pmd_trans_huge() helpers that dmem needs by adding pmd_special() checks. This method could be reused in many mm-core paths.
--- arch/x86/include/asm/pgtable.h | 10 +++++++++- include/linux/pgtable.h | 5 +++++ 2 files changed, 14 insertions(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +9 -1
@@ -260,7 +260,7 @@ static inline int pmd_large(pmd_t pte)/* NOTE: when predicate huge page, consider also pmd_devmap, or use pmd_large */ static inline int pmd_trans_huge (pmd_t pmd) { -return (pmd_val (pmd) & (_PAGE_PSE|_PAGE_DEVMAP)) == _PAGE_PSE; +return (pmd_val (pmd) & (_PAGE_PSE|_PAGE_DEVMAP|_PAGE_DMEM)) == _PAGE_PSE; }#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD
@@ -276,6 +276,14 @@ static inline int has_transparent_hugepage(void)return boot_cpu_has (X86_FEATURE_PSE ); } +#ifdef CONFIG_ARCH_HAS_PTE_DMEM +static inline int pmd_special (pmd_t pmd) +{ +return (pmd_val (pmd) & (_PAGE_SPECIAL | _PAGE_DMEM)) == + (_PAGE_SPECIAL | _PAGE_DMEM); +} +#endif +#ifdef CONFIG_ARCH_HAS_PTE_DEVMAP static inline int pmd_devmap (pmd_t pmd) {
include/linux/pgtable.h +5 -0
@@ -1134,6 +1134,11 @@ static inline pmd_t pmd_mkdmem(pmd_t pmd) {return pmd; } + +static inline int pmd_special (pmd_t pmd) +{ +return 0 ; +}#endif #ifndef pmd_read_atomic
Signature · 1 line
2.28.0
PATCH 15/35 mm: add pmd_special() check for pmd_trans_huge_lock()
In reply to [email protected]From: Yulei Zhang <[email protected]>
As dmem-pmd had been distinguished from thp-pmd, we need to add pmd_special() such that pmd_trans_huge_lock could fetch ptl for dmem huge pmd and treat it as stable pmd.
--- include/linux/huge_mm.h | 3 ++- mm/huge_memory.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
include/linux/huge_mm.h +2 -1
@@ -245,7 +245,8 @@ static inline int is_swap_pmd(pmd_t pmd)static inline spinlock_t *pmd_trans_huge_lock (pmd_t *pmd,struct vm_area_struct *vma) { -if (is_swap_pmd (*pmd) ||pmd_trans_huge (*pmd) ||pmd_devmap (*pmd)) +if (is_swap_pmd (*pmd) ||pmd_trans_huge (*pmd) + ||pmd_devmap (*pmd) ||pmd_special (*pmd))return __pmd_trans_huge_lock (pmd, vma);else return NULL ;
mm/huge_memory.c +1 -1
@@ -1862,7 +1862,7 @@ spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma)spinlock_t *ptl; ptl =pmd_lock (vma->vm_mm, pmd);if (likely (is_swap_pmd (*pmd) ||pmd_trans_huge (*pmd) || -pmd_devmap (*pmd))) +pmd_devmap (*pmd) ||pmd_special (*pmd)))return ptl;spin_unlock (ptl);return NULL ;
Signature · 1 line
2.28.0
PATCH 16/35 dmemfs: introduce ->split() to dmemfs_vm_ops
In reply to [email protected]From: Yulei Zhang <[email protected]>
It is required by __split_vma() to adjust vma. munmap() which create hole unaligned to pagesize in dmemfs-mapping should be forbidden.
--- fs/dmemfs/inode.c | 8 ++++++++ 1 file changed, 8 insertions(+)
fs/dmemfs/inode.c +8 -0
@@ -453,6 +453,13 @@ dmemfs_access_dmem(struct vm_area_struct *vma, unsigned long addr,return len; } +static int dmemfs_split (struct vm_area_struct *vma,unsigned long addr) +{ +if (addr & (dmem_page_size (file_inode (vma->vm_file)) -1 )) +return -EINVAL ; +return 0 ; +} +static vm_fault_t dmemfs_fault (struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma;
@@ -487,6 +494,7 @@ static unsigned long dmemfs_pagesize(struct vm_area_struct *vma) }static const struct vm_operations_struct dmemfs_vm_ops = { + .split = dmemfs_split, .fault = dmemfs_fault, .pagesize = dmemfs_pagesize, .access = dmemfs_access_dmem,
Signature · 1 line
2.28.0
PATCH 17/35 mm, dmemfs: support unmap_page_range() for dmemfs pmd
In reply to [email protected]From: Yulei Zhang <[email protected]>
It is required by munmap() for dmemfs mapping.
--- mm/huge_memory.c | 2 ++ mm/memory.c | 8 +++++--- 2 files changed, 7 insertions(+), 3 deletions(-)
mm/huge_memory.c +2 -0
@@ -1636,6 +1636,8 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,spin_unlock (ptl);if (is_huge_zero_pmd (orig_pmd))tlb_remove_page_size (tlb,pmd_page (orig_pmd),HPAGE_PMD_SIZE ); + }else if (pmd_special (orig_pmd)) { +spin_unlock (ptl); }else if (is_huge_zero_pmd (orig_pmd)) {zap_deposited_table (tlb->mm, pmd);spin_unlock (ptl);
mm/memory.c +5 -3
@@ -1178,10 +1178,12 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, pmd =pmd_offset (pud, addr); do { next = pmd_addr_end(addr, end); - if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) { - if (next - addr != HPAGE_PMD_SIZE) + if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || + pmd_devmap(*pmd) || pmd_special(*pmd)) { + if (next - addr != HPAGE_PMD_SIZE) { + VM_BUG_ON(pmd_special(*pmd)); __split_huge_pmd(vma, pmd, addr, false, NULL); - else if (zap_huge_pmd(tlb, vma, pmd, addr)) + } else if (zap_huge_pmd(tlb, vma, pmd, addr)) goto next; /* fall through */ }
Signature · 1 line
2.28.0
PATCH 18/35 mm: follow_pmd_mask() for dmem huge pmd
In reply to [email protected]From: Yulei Zhang <[email protected]>
While follow_pmd_mask(), dmem huge pmd should be recognized and return error pointer of '-EEXIST' to indicate that proper page table entry exists in pmd special but no corresponding struct page, because dmem page means non struct page backend. We update pmd if foll_flags takes FOLL_TOUCH.
--- mm/gup.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
mm/gup.c +42 -0
@@ -380,6 +380,42 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address,return -EEXIST ; } +static struct page * +follow_special_pmd (struct vm_area_struct *vma,unsigned long address, +pmd_t *pmd,unsigned int flags) +{ +spinlock_t *ptl; + +if (flags &FOLL_DUMP ) +/* Avoid special (like zero) pages in core dumps */ +return ERR_PTR (-EFAULT ); + +/* No page to get reference */ +if (flags &FOLL_GET ) +return ERR_PTR (-EFAULT ); + +if (flags &FOLL_TOUCH ) { +pmd_t _pmd; + + ptl =pmd_lock (vma->vm_mm, pmd); +if (!pmd_special (*pmd)) { +spin_unlock (ptl); +return NULL ; + } + _pmd =pmd_mkyoung (*pmd); +if (flags &FOLL_WRITE ) + _pmd =pmd_mkdirty (_pmd); +if (pmdp_set_access_flags (vma, address &HPAGE_PMD_MASK , + pmd, _pmd, + flags &FOLL_WRITE )) +update_mmu_cache_pmd (vma, address, pmd); +spin_unlock (ptl); + } + +/* Proper page table entry exists, but no corresponding struct page */ +return ERR_PTR (-EEXIST ); +} +/* * FOLL_FORCE can write to even unwritable pte's, but only * after we've gone through a COW cycle and they are dirty.
@@ -564,6 +600,12 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma,return page;return no_page_table (vma, flags); } +if (pmd_special (*pmd)) { + page =follow_special_pmd (vma, address, pmd, flags); +if (page) +return page; +return no_page_table (vma, flags); + }if (is_hugepd (__hugepd (pmd_val (pmdval)))) { page = follow_huge_pd(vma, address, __hugepd(pmd_val(pmdval)), flags,
Signature · 1 line
2.28.0
PATCH 19/35 mm: gup_huge_pmd() for dmem huge pmd
In reply to [email protected]From: Yulei Zhang <[email protected]>
Add pmd_special() check in gup_huge_pmd() to support dmem huge pmd. GUP will return zero if enconter dmem page, and we could handle it outside GUP routine.
--- mm/gup.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
mm/gup.c +5 -1
@@ -2440,6 +2440,10 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,if (!pmd_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; +/* Bypass dmem huge pmd. It will be handled in outside routine. */ +if (pmd_special (orig)) +return 0 ; +if (pmd_devmap (orig)) {if (unlikely (flags &FOLL_LONGTERM ))return 0 ;
@@ -2542,7 +2546,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,return 0 ;if (unlikely (pmd_trans_huge (pmd) ||pmd_huge (pmd) || -pmd_devmap (pmd))) { +pmd_devmap (pmd) ||pmd_special (pmd))) {/* * NUMA hinting faults need to be handled in the GUP * slowpath for accounting purposes and so that they
Signature · 1 line
2.28.0
PATCH 20/35 mm: support dmem huge pmd for vmf_insert_pfn_pmd()
In reply to [email protected]From: Yulei Zhang <[email protected]>
Since vmf_insert_pfn_pmd will BUG_ON non-pmd-devmap, we make pfn dmem pass the check.
Dmem huge pmd will be marked with _PAGE_SPECIAL and _PAGE_DMEM, so that follow_pfn() could recognize it.
--- mm/huge_memory.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/huge_memory.c +3 -1
@@ -781,6 +781,8 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, entry =pmd_mkhuge (pfn_t_pmd (pfn, prot));if (pfn_t_devmap (pfn)) entry =pmd_mkdevmap (entry); +else if (pfn_t_dmem (pfn)) + entry =pmd_mkdmem (entry);if (write) { entry =pmd_mkyoung (pmd_mkdirty (entry)); entry =maybe_pmd_mkwrite (entry, vma);
@@ -827,7 +829,7 @@ vm_fault_t vmf_insert_pfn_pmd_prot(struct vm_fault *vmf, pfn_t pfn,* can't support a 'special' bit. */ BUG_ON (!(vma->vm_flags & (VM_PFNMAP |VM_MIXEDMAP )) && - !pfn_t_devmap (pfn)); + !pfn_t_devmap (pfn) && !pfn_t_dmem (pfn));BUG_ON ((vma->vm_flags & (VM_PFNMAP |VM_MIXEDMAP )) == (VM_PFNMAP |VM_MIXEDMAP ));BUG_ON ((vma->vm_flags &VM_PFNMAP ) &&is_cow_mapping (vma->vm_flags));
Signature · 1 line
2.28.0
PATCH 21/35 mm: support dmem huge pmd for follow_pfn()
In reply to [email protected]From: Yulei Zhang <[email protected]>
follow_pfn() will get pfn of pmd if huge pmd is encountered.
--- mm/memory.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
mm/memory.c +11 -3
@@ -4644,15 +4644,23 @@ int follow_pfn(struct vm_area_struct *vma, unsigned long address,int ret = -EINVAL ;spinlock_t *ptl;pte_t *ptep; +pmd_t *pmdp =NULL ;if (!(vma->vm_flags & (VM_IO |VM_PFNMAP )))return ret; - ret =follow_pte (vma->vm_mm, address, &ptep, &ptl); + ret =follow_pte_pmd (vma->vm_mm, address,NULL , &ptep, &pmdp, &ptl);if (ret)return ret; - *pfn =pte_pfn (*ptep); -pte_unmap_unlock (ptep, ptl); + +if (pmdp) { + *pfn =pmd_pfn (*pmdp) + ((address & ~PMD_MASK ) >>PAGE_SHIFT ); +spin_unlock (ptl); + }else { + *pfn =pte_pfn (*ptep); +pte_unmap_unlock (ptep, ptl); + } +return 0 ; }EXPORT_SYMBOL (follow_pfn);
Signature · 1 line
2.28.0
PATCH 22/35 kvm, x86: Distinguish dmemfs page from mmio page
In reply to [email protected]From: Yulei Zhang <[email protected]>
Dmem page is pfn invalid but not mmio. Support cacheable dmem page for kvm.
--- arch/x86/kvm/mmu/mmu.c | 5 +++-- include/linux/dmem.h | 7 +++++++ mm/dmem.c | 7 +++++++ 3 files changed, 17 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -2
@@ -41,6 +41,7 @@#include <linux/hash.h> #include <linux/kern_levels.h> #include <linux/kthread.h> +#include <linux/dmem.h> #include <asm/page.h> #include <asm/memtype.h>
@@ -2962,9 +2963,9 @@ static bool kvm_is_mmio_pfn(kvm_pfn_t pfn)*/ (!pat_enabled () ||pat_pfn_immune_to_uc_mtrr (pfn)); -return !e820__mapped_raw_any (pfn_to_hpa (pfn), +return (!e820__mapped_raw_any (pfn_to_hpa (pfn),pfn_to_hpa (pfn +1 ) -1 , -E820_TYPE_RAM ); +E820_TYPE_RAM )) || (!is_dmem_pfn (pfn)); }/* Bits which may be returned by set_spte() */
include/linux/dmem.h +7 -0
@@ -19,11 +19,18 @@ dmem_alloc_pages_vma(struct vm_area_struct *vma, unsigned long addr, unsigned int try_max, unsigned int *result_nr);void dmem_free_pages (phys_addr_t addr,unsigned int dpages_nr); +bool is_dmem_pfn (unsigned long pfn);#define dmem_free_page (addr) dmem_free_pages(addr, 1)#else static inline int dmem_reserve_init (void ) {return 0 ; } + +static inline bool is_dmem_pfn (unsigned long pfn) +{ +return 0 ; +} +#endif #endif /* _LINUX_DMEM_H */
mm/dmem.c +7 -0
@@ -972,3 +972,10 @@ void dmem_free_pages(phys_addr_t addr, unsigned int dpages_nr) }EXPORT_SYMBOL (dmem_free_pages); +bool is_dmem_pfn (unsigned long pfn) +{ +struct dmem_node *dnode; + +return !!find_dmem_region (__pfn_to_phys (pfn), &dnode); +} +EXPORT_SYMBOL (is_dmem_pfn);
Signature · 1 line
2.28.0
Re: [PATCH 22/35] kvm, x86: Distinguish dmemfs page from mmio page
Quoted 26 lines · On Thu, Oct 08, 2020 at 03:54:12PM +0800, [email protected] wrote:
From: Yulei Zhang <[email protected]>
Dmem page is pfn invalid but not mmio. Support cacheable dmem page for kvm.
Signed-off-by Chen Zhuo <[email protected]>Signed-off-by Yulei Zhang <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 5 +++-- include/linux/dmem.h | 7 +++++++ mm/dmem.c | 7 +++++++ 3 files changed, 17 insertions(+), 2 deletions(-)arch/x86/kvm/mmu/mmu.c +1
-0@@ -41,6 +41,7 @@#include <linux/hash.h> #include <linux/kern_levels.h> #include <linux/kthread.h> +#include <linux/dmem.h> #include <asm/page.h> #include <asm/memtype.h>
arch/x86/kvm/mmu/mmu.c static bool kvm_is_mmio_pfn(kvm_pfn_t pfn) · line 2968
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +2
-2@@ -2962,9 +2963,9 @@ static bool kvm_is_mmio_pfn(kvm_pfn_t pfn)*/ (!pat_enabled () ||pat_pfn_immune_to_uc_mtrr (pfn)); -return !e820__mapped_raw_any (pfn_to_hpa (pfn), +return (!e820__mapped_raw_any (pfn_to_hpa (pfn),pfn_to_hpa (pfn +1 ) -1 , -E820_TYPE_RAM ); +E820_TYPE_RAM )) || (!is_dmem_pfn (pfn));
This is wrong. As is, the logic reads "A PFN is MMIO if it is INVALID && (!RAM || !DMEM)". The obvious fix would be to change it to "INVALID && !RAM && !DMEM", but that begs the question of whether or DMEM is reported as RAM. I don't see any e820 related changes in the series, i.e. no evidence that dmem yanks its memory out of the e820 tables, which makes me think this change is unnecessary.
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +0
-0}/* Bits which may be returned by set_spte() */
Re: [PATCH 22/35] kvm, x86: Distinguish dmemfs page from mmio page
Quoted 30 lines · Sean Christopherson
Quoted 29 lines · On Thu, Oct 08, 2020 at 03:54:12PM +0800, [email protected] wrote:
From: Yulei Zhang <[email protected]>
Dmem page is pfn invalid but not mmio. Support cacheable dmem page for kvm.
Signed-off-by Chen Zhuo <[email protected]>Signed-off-by Yulei Zhang <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 5 +++-- include/linux/dmem.h | 7 +++++++ mm/dmem.c | 7 +++++++ 3 files changed, 17 insertions(+), 2 deletions(-)arch/x86/kvm/mmu/mmu.c +1
-0@@ -41,6 +41,7 @@#include <linux/hash.h> #include <linux/kern_levels.h> #include <linux/kthread.h> +#include <linux/dmem.h> #include <asm/page.h> #include <asm/memtype.h> @@ -2962,9 +2963,9 @@ static bool kvm_is_mmio_pfn(kvm_pfn_t pfn)*/ (!pat_enabled () ||pat_pfn_immune_to_uc_mtrr (pfn));
Quoted 13 lines
arch/x86/kvm/mmu/mmu.c static bool kvm_is_mmio_pfn(kvm_pfn_t pfn) · line 2968
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +2
-2-return !e820__mapped_raw_any (pfn_to_hpa (pfn), +return (!e820__mapped_raw_any (pfn_to_hpa (pfn),pfn_to_hpa (pfn +1 ) -1 , -E820_TYPE_RAM ); +E820_TYPE_RAM )) || (!is_dmem_pfn (pfn));This is wrong. As is, the logic reads "A PFN is MMIO if it is INVALID && (!RAM || !DMEM)". The obvious fix would be to change it to "INVALID && !RAM && !DMEM", but that begs the question of whether or DMEM is reported as RAM. I don't see any e820 related changes in the series, i.e. no evidence that dmem yanks its memory out of the e820 tables, which makes me think this change is unnecessary.
Even if there would exist e820 changes, e820__mapped_raw_any() checks against hardware-provided e820 that we are given before any changes happen i.e. not the one kernel has changed (e820_table_firmware). So unless you're having that memory carved from an MMIO range (which would be wrong), or the BIOS is misrepresenting its memory map... the e820__mapped_raw_any(E820_TYPE_RAM) ought to be enough to cover RAM.
Or at least that has been my experience with similar work.
Joao
Re: [PATCH 22/35] kvm, x86: Distinguish dmemfs page from mmio page
Sean and Joao, thanks for the feedback. Probably we can drop this change.
Quoted 54 lines · Joao Martins
Quoted 30 lines · Sean Christopherson
Quoted 29 lines · On Thu, Oct 08, 2020 at 03:54:12PM +0800, [email protected] wrote:
From: Yulei Zhang <[email protected]>
Dmem page is pfn invalid but not mmio. Support cacheable dmem page for kvm.
Signed-off-by Chen Zhuo <[email protected]>Signed-off-by Yulei Zhang <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 5 +++-- include/linux/dmem.h | 7 +++++++ mm/dmem.c | 7 +++++++ 3 files changed, 17 insertions(+), 2 deletions(-)arch/x86/kvm/mmu/mmu.c +1
-0@@ -41,6 +41,7 @@#include <linux/hash.h> #include <linux/kern_levels.h> #include <linux/kthread.h> +#include <linux/dmem.h> #include <asm/page.h> #include <asm/memtype.h> @@ -2962,9 +2963,9 @@ static bool kvm_is_mmio_pfn(kvm_pfn_t pfn)*/ (!pat_enabled () ||pat_pfn_immune_to_uc_mtrr (pfn));Quoted 13 lines
arch/x86/kvm/mmu/mmu.c static bool kvm_is_mmio_pfn(kvm_pfn_t pfn) · line 2968
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +2
-2-return !e820__mapped_raw_any (pfn_to_hpa (pfn), +return (!e820__mapped_raw_any (pfn_to_hpa (pfn),pfn_to_hpa (pfn +1 ) -1 , -E820_TYPE_RAM ); +E820_TYPE_RAM )) || (!is_dmem_pfn (pfn));This is wrong. As is, the logic reads "A PFN is MMIO if it is INVALID && (!RAM || !DMEM)". The obvious fix would be to change it to "INVALID && !RAM && !DMEM", but that begs the question of whether or DMEM is reported as RAM. I don't see any e820 related changes in the series, i.e. no evidence that dmem yanks its memory out of the e820 tables, which makes me think this change is unnecessary.
Even if there would exist e820 changes,
e820__mapped_raw_any()checks against hardware-provided e820 that we are given before any changes happen i.e. not the one kernel has changed (e820_table_firmware). So unless you're having that memory carved from an MMIO range (which would be wrong), or the BIOS is misrepresenting its memory map... thee820__mapped_raw_any(E820_TYPE_RAM)ought to be enough to cover RAM.Or at least that has been my experience with similar work.
Joao
PATCH 23/35 kvm, x86: introduce VM_DMEM
In reply to [email protected]From: Yulei Zhang <[email protected]>
Currently dmemfs do not support memory readonly, so change_protection() will be disabled for dmemfs vma. Since vma->vm_flags could be changed to new flag in mprotect_fixup(), so we introduce a new vma flag VM_DMEM and check this flag in mprotect_fixup() to avoid changing vma->vm_flags.
We also check it in vma_to_resize() to disable mremap() for dmemfs vma.
--- fs/dmemfs/inode.c | 2 +- include/linux/mm.h | 7 +++++++ mm/mprotect.c | 5 ++++- mm/mremap.c | 3 +++ 4 files changed, 15 insertions(+), 2 deletions(-)
fs/dmemfs/inode.c +1 -1
@@ -510,7 +510,7 @@ int dmemfs_file_mmap(struct file *file, struct vm_area_struct *vma)if (!(vma->vm_flags &VM_SHARED ))return -EINVAL ; - vma->vm_flags |=VM_PFNMAP ; + vma->vm_flags |=VM_PFNMAP |VM_DMEM |VM_IO ;file_accessed (file); vma->vm_ops = &dmemfs_vm_ops;
include/linux/mm.h +7 -0
@@ -309,6 +309,8 @@ extern unsigned int kobjsize(const void *objp);#define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4)#endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ +#define VM_DMEM BIT(38)/* Dmem page VM */ +#ifdef CONFIG_ARCH_HAS_PKEYS # define VM_PKEY_SHIFT VM_HIGH_ARCH_BIT_0# define VM_PKEY_BIT0 VM_HIGH_ARCH_0/* A protection key is a 4-bit value */
@@ -656,6 +658,11 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma)return vma->vm_flags &VM_ACCESS_FLAGS ; } +static inline bool vma_is_dmem (struct vm_area_struct *vma) +{ +return !!(vma->vm_flags &VM_DMEM ); +} +#ifdef CONFIG_SHMEM /* * The vma_is_shmem is not inline because it is used only by slow
mm/mprotect.c +4 -1
@@ -236,7 +236,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,* for all the checks. */ if (!is_swap_pmd (*pmd) && !pmd_devmap (*pmd) && -pmd_none_or_clear_bad_unless_trans_huge (pmd)) +pmd_none_or_clear_bad_unless_trans_huge (pmd) && !pmd_special (*pmd)) goto next;/* invoke the mmu notifier if the pmd is populated */
@@ -412,6 +412,9 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,return 0 ; } +if (vma_is_dmem (vma)) +return -EINVAL ; +/* * Do PROT_NONE PFN permission checks here when we can still * bail out without undoing a lot of state. This is a rather
mm/mremap.c +3 -0
@@ -482,6 +482,9 @@ static struct vm_area_struct *vma_to_resize(unsigned long addr,if (!vma || vma->vm_start > addr)return ERR_PTR (-EFAULT ); +if (vma_is_dmem (vma)) +return ERR_PTR (-EINVAL ); +/* * !old_len is a special case where an attempt is made to 'duplicate' * a mapping. This makes no sense for private mappings as it will
Signature · 1 line
2.28.0
PATCH 24/35 dmemfs: support hugepage for dmemfs
In reply to [email protected]From: Yulei Zhang <[email protected]>
It add hugepage support for dmemfs. We use PFN_DMEM to notify vmf_insert_pfn_pmd, and dmem huge pmd will be marked with _PAGE_SPECIAL and _PAGE_DMEM. So that GUP-fast can separate dmemfs page from other page type and handle it correctly.
--- fs/dmemfs/inode.c | 113 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 111 insertions(+), 2 deletions(-)
fs/dmemfs/inode.c +111 -2
@@ -460,7 +460,7 @@ static int dmemfs_split(struct vm_area_struct *vma, unsigned long addr)return 0 ; } -static vm_fault_t dmemfs_fault (struct vm_fault *vmf) +static vm_fault_t __dmemfs_fault (struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma;struct inode *inode =file_inode (vma->vm_file);
@@ -488,6 +488,63 @@ static vm_fault_t dmemfs_fault(struct vm_fault *vmf)return ret; } +static vm_fault_t __dmemfs_pmd_fault (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +unsigned long pmd_addr = vmf->address &PMD_MASK ; +unsigned long page_addr; +struct inode *inode =file_inode (vma->vm_file); +void *entry; +phys_addr_t phys; +pfn_t pfn; +int ret; + +if (dmem_page_size (inode) <PMD_SIZE ) +return VM_FAULT_FALLBACK ; + +WARN_ON (pmd_addr < vma->vm_start || + vma->vm_end < pmd_addr +PMD_SIZE ); + + page_addr = vmf->address & ~(dmem_page_size (inode) -1 ); + entry =radix_get_create_entry (vma, page_addr, inode, +linear_page_index (vma, page_addr)); +if (IS_ERR (entry)) +return (PTR_ERR (entry) == -ENOMEM ) ? +VM_FAULT_OOM :VM_FAULT_SIGBUS ; + + phys =dmem_addr_to_pfn (inode,dmem_entry_to_addr (inode, entry), +linear_page_index (vma, pmd_addr),PMD_SHIFT ); + phys <<=PAGE_SHIFT ; + pfn =phys_to_pfn_t (phys,PFN_DMEM ); + ret =vmf_insert_pfn_pmd (vmf, pfn, !!(vma->vm_flags &VM_WRITE )); + +radix_put_entry (); +return ret; +} + +static vm_fault_t dmemfs_huge_fault (struct vm_fault *vmf,enum page_entry_size pe_size) +{ +int ret; + +switch (pe_size) { +case PE_SIZE_PTE : + ret =__dmemfs_fault (vmf); +break ; +case PE_SIZE_PMD : + ret =__dmemfs_pmd_fault (vmf); +break ; +default : + ret =VM_FAULT_SIGBUS ; + } + +return ret; +} + +static vm_fault_t dmemfs_fault (struct vm_fault *vmf) +{ +return dmemfs_huge_fault (vmf,PE_SIZE_PTE ); +} +static unsigned long dmemfs_pagesize (struct vm_area_struct *vma) {return dmem_page_size (file_inode (vma->vm_file));
@@ -498,6 +555,7 @@ static const struct vm_operations_struct dmemfs_vm_ops = { .fault = dmemfs_fault, .pagesize = dmemfs_pagesize, .access = dmemfs_access_dmem, + .huge_fault = dmemfs_huge_fault, };int dmemfs_file_mmap (struct file *file,struct vm_area_struct *vma)
@@ -510,15 +568,66 @@ int dmemfs_file_mmap(struct file *file, struct vm_area_struct *vma)if (!(vma->vm_flags &VM_SHARED ))return -EINVAL ; - vma->vm_flags |=VM_PFNMAP |VM_DMEM |VM_IO ; + vma->vm_flags |=VM_PFNMAP |VM_DONTCOPY |VM_DMEM |VM_IO ; + +if (dmem_page_size (inode) !=PAGE_SIZE ) + vma->vm_flags |=VM_HUGEPAGE ;file_accessed (file); vma->vm_ops = &dmemfs_vm_ops;return 0 ; } +/* +* If the size of area returned by mm->get_unmapped_area() is one +* dmem pagesize larger than 'len', the returned addr by +* mm->get_unmapped_area() could be aligned to dmem pagesize to +* meet alignment demand. +*/ +static unsigned long +dmemfs_get_unmapped_area (struct file *file,unsigned long addr, +unsigned long len,unsigned long pgoff, +unsigned long flags) +{ +unsigned long len_pad; +unsigned long off = pgoff <<PAGE_SHIFT ; +unsigned long align; + + align =dmem_page_size (file_inode (file)); + +/* For pud or pmd pagesize, could not support fault fallback. */ +if (len & (align -1 )) +return -EINVAL ; +if (len >TASK_SIZE ) +return -ENOMEM ; + +if (flags &MAP_FIXED ) { +if (addr & (align -1 )) +return -EINVAL ; +return addr; + } + +/* +* Pad a extra align space for 'len', as we want to find a unmapped +* area which is larger enough to align with dmemfs pagesize, if +* pagesize of dmem is larger than 4K. +*/ + len_pad = (align ==PAGE_SIZE ) ? len : len + align; + +/* 'len' or 'off' is too large for pad. */ +if (len_pad < len || (off + len_pad) < off) +return -EINVAL ; + + addr = current->mm->get_unmapped_area (file, addr, len_pad, + pgoff, flags); + +/* Now 'addr' could be aligned to upper boundary. */ +return IS_ERR_VALUE (addr) ? addr :round_up (addr, align); +} +static const struct file_operations dmemfs_file_operations = { .mmap = dmemfs_file_mmap, + .get_unmapped_area = dmemfs_get_unmapped_area, };static int dmemfs_parse_param (struct fs_context *fc,struct fs_parameter *param)
Signature · 1 line
2.28.0
PATCH 25/35 mm, x86, dmem: fix estimation of reserved page for vaddr_get_pfn()
In reply to [email protected]From: Yulei Zhang <[email protected]>
Fix estimation of reserved page for vaddr_get_pfn() and check 'ret' before checking writable permission
--- drivers/vfio/vfio_iommu_type1.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/vfio/vfio_iommu_type1.c +4 -0
@@ -471,6 +471,10 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,if (ret == -EAGAIN ) goto retry; +if (!ret && (prot &IOMMU_WRITE ) && + !(vma->vm_flags &VM_WRITE )) + ret = -EFAULT ; +if (!ret && !is_invalid_reserved_pfn (*pfn)) ret = -EFAULT ; }
Signature · 1 line
2.28.0
PATCH 26/35 mm, dmem: introduce pud_special()
In reply to [email protected]From: Yulei Zhang <[email protected]>
pud_special() will check both _PAGE_SPECIAL and _PAGE_DMEM bit as pmd_special() does.
--- arch/x86/include/asm/pgtable.h | 13 +++++++++++++ include/linux/pgtable.h | 10 ++++++++++ 2 files changed, 23 insertions(+)
arch/x86/include/asm/pgtable.h +13 -0
@@ -282,6 +282,12 @@ static inline int pmd_special(pmd_t pmd)return (pmd_val (pmd) & (_PAGE_SPECIAL | _PAGE_DMEM)) == (_PAGE_SPECIAL | _PAGE_DMEM); } + +static inline int pud_special (pud_t pud) +{ +return (pud_val (pud) & (_PAGE_SPECIAL | _PAGE_DMEM)) == + (_PAGE_SPECIAL | _PAGE_DMEM); +}#endif #ifdef CONFIG_ARCH_HAS_PTE_DEVMAP
@@ -517,6 +523,13 @@ static inline pud_t pud_mkdirty(pud_t pud)return pud_set_flags (pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); } +#ifdef CONFIG_ARCH_HAS_PTE_DMEM +static inline pud_t pud_mkdmem (pud_t pud) +{ +return pud_set_flags (pud, _PAGE_SPECIAL | _PAGE_DMEM); +} +#endif +static inline pud_t pud_mkdevmap (pud_t pud) {return pud_set_flags (pud, _PAGE_DEVMAP);
include/linux/pgtable.h +10 -0
@@ -1139,6 +1139,16 @@ static inline int pmd_special(pmd_t pmd) {return 0 ; } + +static inline pud_t pud_mkdmem (pud_t pud) +{ +return pud; +} + +static inline int pud_special (pud_t pud) +{ +return 0 ; +}#endif #ifndef pmd_read_atomic
Signature · 1 line
2.28.0
PATCH 27/35 mm: add pud_special() to support dmem huge pud
In reply to [email protected]From: Yulei Zhang <[email protected]>
Add pud_special() and follow_special_pud() to support dmem huge pud as we do for dmem huge pmd.
--- arch/x86/include/asm/pgtable.h | 2 +- include/linux/huge_mm.h | 2 +- mm/gup.c | 46 ++++++++++++++++++++++++++++++++++ mm/huge_memory.c | 11 +++++--- mm/memory.c | 4 +-- mm/mprotect.c | 2 ++ 6 files changed, 59 insertions(+), 8 deletions(-)
arch/x86/include/asm/pgtable.h +1 -1
@@ -266,7 +266,7 @@ static inline int pmd_trans_huge(pmd_t pmd)#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD static inline int pud_trans_huge (pud_t pud) { -return (pud_val (pud) & (_PAGE_PSE|_PAGE_DEVMAP)) == _PAGE_PSE; +return (pud_val (pud) & (_PAGE_PSE|_PAGE_DEVMAP|_PAGE_DMEM)) == _PAGE_PSE; }#endif
include/linux/huge_mm.h +1 -1
@@ -254,7 +254,7 @@ static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,static inline spinlock_t *pud_trans_huge_lock (pud_t *pud,struct vm_area_struct *vma) { -if (pud_trans_huge (*pud) ||pud_devmap (*pud)) +if (pud_trans_huge (*pud) ||pud_devmap (*pud) ||pud_special (*pud))return __pud_trans_huge_lock (pud, vma);else return NULL ;
mm/gup.c +46 -0
64 lines not on this page · raw message
mm/huge_memory.c +7 -4
@@ -883,6 +883,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, entry =pud_mkhuge (pfn_t_pud (pfn, prot));if (pfn_t_devmap (pfn)) entry =pud_mkdevmap (entry); +if (pfn_t_dmem (pfn)) + entry =pud_mkdmem (entry);if (write) { entry =pud_mkyoung (pud_mkdirty (entry)); entry =maybe_pud_mkwrite (entry, vma);
@@ -919,7 +921,7 @@ vm_fault_t vmf_insert_pfn_pud_prot(struct vm_fault *vmf, pfn_t pfn,* can't support a 'special' bit. */ BUG_ON (!(vma->vm_flags & (VM_PFNMAP |VM_MIXEDMAP )) && - !pfn_t_devmap (pfn)); + !pfn_t_devmap (pfn) && !pfn_t_dmem (pfn));BUG_ON ((vma->vm_flags & (VM_PFNMAP |VM_MIXEDMAP )) == (VM_PFNMAP |VM_MIXEDMAP ));BUG_ON ((vma->vm_flags &VM_PFNMAP ) &&is_cow_mapping (vma->vm_flags));
@@ -1883,7 +1885,7 @@ spinlock_t *__pud_trans_huge_lock(pud_t *pud, struct vm_area_struct *vma)spinlock_t *ptl; ptl =pud_lock (vma->vm_mm, pud); -if (likely (pud_trans_huge (*pud) ||pud_devmap (*pud))) +if (likely (pud_trans_huge (*pud) ||pud_devmap (*pud) ||pud_special (*pud)))return ptl;spin_unlock (ptl);return NULL ;
@@ -1894,6 +1896,7 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t *pud, unsigned long addr) {spinlock_t *ptl; +pud_t orig_pud; ptl =__pud_trans_huge_lock (pud, vma);if (!ptl)
@@ -1904,9 +1907,9 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,* pgtable_trans_huge_withdraw after finishing pudp related * operations. */ -pudp_huge_get_and_clear_full (tlb->mm, addr, pud, tlb->fullmm); + orig_pud =pudp_huge_get_and_clear_full (tlb->mm, addr, pud, tlb->fullmm);tlb_remove_pud_tlb_entry (tlb, pud, addr); -if (vma_is_special_huge (vma)) { +if (vma_is_special_huge (vma) ||pud_special (orig_pud)) {spin_unlock (ptl);/* No zero page support yet */ }else {
mm/memory.c +2 -2
@@ -922,7 +922,7 @@ static inline int copy_pud_range(struct mm_struct *dst_mm, struct mm_struct *src src_pud =pud_offset (src_p4d, addr); do { next = pud_addr_end(addr, end); - if (pud_trans_huge(*src_pud) || pud_devmap(*src_pud)) { + if (pud_trans_huge(*src_pud) || pud_devmap(*src_pud) || pud_special(*src_pud)) { int err; VM_BUG_ON_VMA(next-addr != HPAGE_PUD_SIZE, vma);
@@ -1215,7 +1215,7 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb, pud =pud_offset (p4d, addr); do { next = pud_addr_end(addr, end); - if (pud_trans_huge(*pud) || pud_devmap(*pud)) { + if (pud_trans_huge(*pud) || pud_devmap(*pud) || pud_special(*pud)) { if (next - addr != HPAGE_PUD_SIZE) { mmap_assert_locked(tlb->mm); split_huge_pud(vma, pud, addr);
mm/mprotect.c +2 -0
@@ -292,6 +292,8 @@ static inline unsigned long change_pud_range(struct vm_area_struct *vma, pud =pud_offset (p4d, addr); do { next = pud_addr_end(addr, end); + if (pud_special(*pud)) + continue; if (pud_none_or_clear_bad(pud)) continue; pages += change_pmd_range(vma, pud, addr, next, newprot,
Signature · 1 line
2.28.0
PATCH 28/35 mm, dmemfs: support huge_fault() for dmemfs
In reply to [email protected]From: Yulei Zhang <[email protected]>
Introduce __dmemfs_huge_fault() to handle 1G huge pud for dmemfs.
--- fs/dmemfs/inode.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
fs/dmemfs/inode.c +40 -0
@@ -522,6 +522,43 @@ static vm_fault_t __dmemfs_pmd_fault(struct vm_fault *vmf)return ret; } +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +static vm_fault_t __dmemfs_huge_fault (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +unsigned long pud_addr = vmf->address &PUD_MASK ; +struct inode *inode =file_inode (vma->vm_file); +void *entry; +phys_addr_t phys; +pfn_t pfn; +int ret; + +if (dmem_page_size (inode) <PUD_SIZE ) +return VM_FAULT_FALLBACK ; + +WARN_ON (pud_addr < vma->vm_start || + vma->vm_end < pud_addr +PUD_SIZE ); + + entry =radix_get_create_entry (vma, pud_addr, inode, +linear_page_index (vma, pud_addr)); +if (IS_ERR (entry)) +return (PTR_ERR (entry) == -ENOMEM ) ? +VM_FAULT_OOM :VM_FAULT_SIGBUS ; + + phys =dmem_entry_to_addr (inode, entry); + pfn =phys_to_pfn_t (phys,PFN_DMEM ); + ret =vmf_insert_pfn_pud (vmf, pfn, !!(vma->vm_flags &VM_WRITE )); + +radix_put_entry (); +return ret; +} +#else +static vm_fault_t __dmemfs_huge_fault (struct vm_fault *vmf) +{ +return VM_FAULT_FALLBACK ; +} +#endif /* !CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +static vm_fault_t dmemfs_huge_fault (struct vm_fault *vmf,enum page_entry_size pe_size) {int ret;
@@ -533,6 +570,9 @@ static vm_fault_t dmemfs_huge_fault(struct vm_fault *vmf, enum page_entry_size pcase PE_SIZE_PMD : ret =__dmemfs_pmd_fault (vmf);break ; +case PE_SIZE_PUD : + ret =__dmemfs_huge_fault (vmf); +break ;default : ret =VM_FAULT_SIGBUS ; }
Signature · 1 line
2.28.0
PATCH 29/35 mm: add follow_pte_pud()
In reply to [email protected]From: Yulei Zhang <[email protected]>
Since we had supported dmem huge pud, here support dmem huge pud for hva_to_pfn().
Similar to follow_pte_pmd(), follow_pte_pud() allows a PTE lead or a huge page PMD or huge page PUD to be found and returned.
--- mm/memory.c | 52 ++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 44 insertions(+), 8 deletions(-)
mm/memory.c +44 -8
@@ -4535,9 +4535,9 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address) }#endif /* __PAGETABLE_PMD_FOLDED */ -static int __follow_pte_pmd (struct mm_struct *mm,unsigned long address, +static int __follow_pte_pud (struct mm_struct *mm,unsigned long address,struct mmu_notifier_range *range, -pte_t **ptepp,pmd_t **pmdpp,spinlock_t **ptlp) +pte_t **ptepp,pmd_t **pmdpp,pud_t **pudpp,spinlock_t **ptlp) {pgd_t *pgd;p4d_t *p4d;
@@ -4554,6 +4554,26 @@ static int __follow_pte_pmd(struct mm_struct *mm, unsigned long address, goto out; pud =pud_offset (p4d, address); +VM_BUG_ON (pud_trans_huge (*pud)); +if (pud_huge (*pud)) { +if (!pudpp) + goto out; + +if (range) { +mmu_notifier_range_init (range,MMU_NOTIFY_CLEAR ,0 , +NULL , mm, address &PUD_MASK , + (address &PUD_MASK ) +PUD_SIZE ); +mmu_notifier_invalidate_range_start (range); + } + *ptlp =pud_lock (mm, pud); +if (pud_huge (*pud)) { + *pudpp = pud; +return 0 ; + } +spin_unlock (*ptlp); +if (range) +mmu_notifier_invalidate_range_end (range); + }if (pud_none (*pud) ||unlikely (pud_bad (*pud))) goto out;
@@ -4609,8 +4629,8 @@ static inline int follow_pte(struct mm_struct *mm, unsigned long address,/* (void) is needed to make gcc happy */ (void )__cond_lock (*ptlp, - !(res =__follow_pte_pmd (mm, address,NULL , - ptepp,NULL , ptlp))); + !(res =__follow_pte_pud (mm, address,NULL , + ptepp,NULL ,NULL , ptlp)));return res; }
@@ -4622,12 +4642,24 @@ int follow_pte_pmd(struct mm_struct *mm, unsigned long address,/* (void) is needed to make gcc happy */ (void )__cond_lock (*ptlp, - !(res =__follow_pte_pmd (mm, address, range, - ptepp, pmdpp, ptlp))); + !(res =__follow_pte_pud (mm, address, range, + ptepp, pmdpp,NULL , ptlp)));return res; }EXPORT_SYMBOL (follow_pte_pmd); +int follow_pte_pud (struct mm_struct *mm,unsigned long address, +struct mmu_notifier_range *range, +pte_t **ptepp,pmd_t **pmdpp,pud_t **pudpp,spinlock_t **ptlp) +{ +int res; + +/* (void) is needed to make gcc happy */ + (void )__cond_lock (*ptlp, + !(res =__follow_pte_pud (mm, address, range, + ptepp, pmdpp, pudpp, ptlp))); +return res; +}/** * follow_pfn - look up PFN at a user virtual address * @vma: memory mapping
@@ -4645,15 +4677,19 @@ int follow_pfn(struct vm_area_struct *vma, unsigned long address,spinlock_t *ptl;pte_t *ptep;pmd_t *pmdp =NULL ; +pud_t *pudp =NULL ;if (!(vma->vm_flags & (VM_IO |VM_PFNMAP )))return ret; - ret =follow_pte_pmd (vma->vm_mm, address,NULL , &ptep, &pmdp, &ptl); + ret =follow_pte_pud (vma->vm_mm, address,NULL , &ptep, &pmdp, &pudp, &ptl);if (ret)return ret; -if (pmdp) { +if (pudp) { + *pfn =pud_pfn (*pudp) + ((address & ~PUD_MASK ) >>PAGE_SHIFT ); +spin_unlock (ptl); + }else if (pmdp) { *pfn =pmd_pfn (*pmdp) + ((address & ~PMD_MASK ) >>PAGE_SHIFT );spin_unlock (ptl); }else {
Signature · 1 line
2.28.0
PATCH 30/35 dmem: introduce dmem_bitmap_alloc() and dmem_bitmap_free()
In reply to [email protected]From: Yulei Zhang <[email protected]>
If dmem contained in dmem region is too large and dmemfs is mounted as 4K pagesize, size of bitmap in this dmem region maybe exceed maximal available memory of kzalloc(). It would cause kzalloc() fail.
So introduce dmem_bitmap_alloc() and use vzalloc() if bitmap is larger than PAGE_SIZE as vzalloc() will get sparse page.
--- fs/inode.c | 6 ++++ include/linux/fs.h | 1 + mm/dmem.c | 69 +++++++++++++++++++++++++++++----------------- 3 files changed, 50 insertions(+), 26 deletions(-)
fs/inode.c +6 -0
@@ -208,6 +208,12 @@ int inode_init_always(struct super_block *sb, struct inode *inode) }EXPORT_SYMBOL (inode_init_always); +struct inode *alloc_inode_nonrcu (void ) +{ +return kmem_cache_alloc (inode_cachep,GFP_KERNEL ); +} +EXPORT_SYMBOL (alloc_inode_nonrcu); +void free_inode_nonrcu (struct inode *inode) {kmem_cache_free (inode_cachep, inode);
include/linux/fs.h +1 -0
@@ -2984,6 +2984,7 @@ extern void clear_inode(struct inode *);extern void __destroy_inode (struct inode *);extern struct inode *new_inode_pseudo (struct super_block *sb);extern struct inode *new_inode (struct super_block *sb); +extern struct inode *alloc_inode_nonrcu (void );extern void free_inode_nonrcu (struct inode *inode);extern int should_remove_suid (struct dentry *);extern int file_remove_privs (struct file *);
mm/dmem.c +43 -26
123 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH 31/35 dmem: introduce mce handler
In reply to [email protected]From: Yulei Zhang <[email protected]>
dmem handle the mce if the pfn belongs to dmem when mce occurs.
1. check whether the pfn is handled by dmem. return if true.
2. mark the pfn in a new error bitmap defined in page.
3. a series of mechanism to ensure that the mce pfn is not allocated.
--- include/linux/dmem.h | 6 +++ include/trace/events/dmem.h | 17 ++++++ mm/dmem.c | 103 +++++++++++++++++++++++++----------- mm/memory-failure.c | 6 +++ 4 files changed, 102 insertions(+), 30 deletions(-)
include/linux/dmem.h +6 -0
@@ -21,6 +21,8 @@ dmem_alloc_pages_vma(struct vm_area_struct *vma, unsigned long addr,void dmem_free_pages (phys_addr_t addr,unsigned int dpages_nr);bool is_dmem_pfn (unsigned long pfn);#define dmem_free_page (addr) dmem_free_pages(addr, 1) + +bool dmem_memory_failure (unsigned long pfn,int flags);#else static inline int dmem_reserve_init (void ) {
@@ -32,5 +34,9 @@ static inline bool is_dmem_pfn(unsigned long pfn)return 0 ; } +static inline bool dmem_memory_failure (unsigned long pfn,int flags) +{ +return false; +}#endif #endif /* _LINUX_DMEM_H */
include/trace/events/dmem.h +17 -0
@@ -62,6 +62,23 @@ TRACE_EVENT(dmem_free_pages,TP_printk ("addr %#lx dpages_nr %d" , (unsigned long )__entry->addr, __entry->dpages_nr) ); + +TRACE_EVENT (dmem_memory_failure, + TP_PROTO(unsigned long pfn, bool used), +TP_ARGS (pfn, used), + +TP_STRUCT__entry ( + __field(unsigned long, pfn) +__field (bool, used) + ), + +TP_fast_assign ( + __entry->pfn = pfn; + __entry->used = used; + ), + +TP_printk ("pfn=%#lx used=%d" , __entry->pfn, __entry->used) +);#endif /* This part must be outside protection */
mm/dmem.c +73 -30
127 lines not on this page · raw message
mm/memory-failure.c +6 -0
@@ -35,6 +35,7 @@*/ #include <linux/kernel.h> #include <linux/mm.h> +#include <linux/dmem.h> #include <linux/page-flags.h> #include <linux/kernel-page-flags.h> #include <linux/sched/signal.h>
@@ -1280,6 +1281,11 @@ int memory_failure(unsigned long pfn, int flags)if (!sysctl_memory_failure_recovery)panic ("Memory failure on page %lx" , pfn); +if (dmem_memory_failure (pfn, flags)) { +pr_info ("MCE %#lx: handled by dmem\n" , pfn); +return 0 ; + } + p =pfn_to_online_page (pfn);if (!p) {if (pfn_valid (pfn)) {
Signature · 1 line
2.28.0
PATCH 32/35 mm, dmemfs: register and handle the dmem mce
In reply to [email protected]From: Yulei Zhang <[email protected]>
dmemfs register the mce handler, send signal to the procs whose vma is mapped in mce pfn.
--- fs/dmemfs/inode.c | 141 +++++++++++++++++++++++++++++++++++++++++++ include/linux/dmem.h | 7 +++ include/linux/mm.h | 2 + mm/dmem.c | 34 +++++++++++ mm/memory-failure.c | 63 ++++++++++++++----- 5 files changed, 231 insertions(+), 16 deletions(-)
fs/dmemfs/inode.c +141 -0
@@ -36,6 +36,47 @@ MODULE_LICENSE("GPL v2");static uint __read_mostly max_alloc_try_dpages =1 ; +struct dmemfs_inode { +struct inode *inode; +struct list_head link; +}; + +static LIST_HEAD (dmemfs_inode_list); +static DEFINE_SPINLOCK (dmemfs_inode_lock); + +static struct dmemfs_inode * +dmemfs_create_dmemfs_inode (struct inode *inode) +{ +struct dmemfs_inode *dmemfs_inode; + +spin_lock (&dmemfs_inode_lock); + dmemfs_inode =kmalloc (sizeof (struct dmemfs_inode ),GFP_NOIO ); +if (!dmemfs_inode) { +pr_err ("DMEMFS: Out of memory while getting dmemfs inode\n" ); + goto out; + } + dmemfs_inode->inode = inode; +list_add_tail (&dmemfs_inode->link, &dmemfs_inode_list); +out: +spin_unlock (&dmemfs_inode_lock); +return dmemfs_inode; +} + +static void dmemfs_delete_dmemfs_inode (struct inode *inode) +{ +struct dmemfs_inode *i, *next; + +spin_lock (&dmemfs_inode_lock); +list_for_each_entry_safe (i, next, &dmemfs_inode_list, link) { +if (i->inode == inode) { +list_del (&i->link); +kfree (i); +break ; + } + } +spin_unlock (&dmemfs_inode_lock); +} +struct dmemfs_mount_opts {unsigned long dpage_size; };
@@ -221,6 +262,13 @@ static unsigned long dmem_pgoff_to_index(struct inode *inode, pgoff_t pgoff)return pgoff >> (sb->s_blocksize_bits -PAGE_SHIFT ); } +static pgoff_t dmem_index_to_pgoff (struct inode *inode,unsigned long index) +{ +struct super_block *sb = inode->i_sb; + +return index << (sb->s_blocksize_bits -PAGE_SHIFT ); +} +static void *dmem_addr_to_entry (struct inode *inode,phys_addr_t addr) {struct super_block *sb = inode->i_sb;
@@ -809,6 +857,23 @@ static void dmemfs_evict_inode(struct inode *inode)clear_inode (inode); } +static struct inode *dmemfs_alloc_inode (struct super_block *sb) +{ +struct inode *inode; + + inode =alloc_inode_nonrcu (); +if (inode) +dmemfs_create_dmemfs_inode (inode); +return inode; +} + +static void dmemfs_destroy_inode (struct inode *inode) +{ +if (inode) +dmemfs_delete_dmemfs_inode (inode); +free_inode_nonrcu (inode); +} +/* * Display the mount options in /proc/mounts. */
@@ -822,9 +887,11 @@ static int dmemfs_show_options(struct seq_file *m, struct dentry *root) }static const struct super_operations dmemfs_ops = { + .alloc_inode = dmemfs_alloc_inode, .statfs = dmemfs_statfs, .evict_inode = dmemfs_evict_inode, .drop_inode = generic_delete_inode, + .destroy_inode = dmemfs_destroy_inode, .show_options = dmemfs_show_options, };
@@ -904,17 +971,91 @@ static struct file_system_type dmemfs_fs_type = { .kill_sb = dmemfs_kill_sb, }; +static struct inode * +dmemfs_find_inode_by_addr (phys_addr_t addr,pgoff_t *pgoff) +{ +struct dmemfs_inode *di; +struct inode *inode; +struct address_space *mapping; +void *entry, **slot; +void *mce_entry; + +list_for_each_entry (di, &dmemfs_inode_list, link) { + inode = di->inode; + mapping = inode->i_mapping; + mce_entry =dmem_addr_to_entry (inode, addr); +XA_STATE (xas, &mapping->i_pages,0 ); +rcu_read_lock (); + +xas_for_each (&xas, entry,ULONG_MAX ) { +if (xas_retry (&xas, entry)) +continue ; + +if (unlikely (entry !=xas_reload (&xas))) + goto retry; + +if (mce_entry != entry) +continue ; + *pgoff =dmem_index_to_pgoff (inode, xas.xa_index); +rcu_read_unlock (); +return inode; +retry: +xas_reset (&xas); + } +rcu_read_unlock (); + } +return NULL ; +} + +static int dmemfs_mce_handler (struct notifier_block *this,unsigned long pfn, +void *v) +{ +struct dmem_mce_notifier_info *info = + (struct dmem_mce_notifier_info *)v; +int flags = info->flags; +struct inode *inode; +phys_addr_t mce_addr =__pfn_to_phys (pfn); +pgoff_t pgoff; + +spin_lock (&dmemfs_inode_lock); + inode =dmemfs_find_inode_by_addr (mce_addr, &pgoff); +if (!inode || !atomic_read (&inode->i_count)) + goto out; + +collect_procs_and_signal_inode (inode, pgoff, pfn, flags); +out: +spin_unlock (&dmemfs_inode_lock); +return 0 ; +} + +static struct notifier_block dmemfs_mce_notifier = { + .notifier_call = dmemfs_mce_handler, +}; +static int __init dmemfs_init (void ) {int ret; +pr_info ("dmemfs initialized\n" ); ret =register_filesystem (&dmemfs_fs_type); +if (ret) + goto reg_fs_fail; + + ret =dmem_register_mce_notifier (&dmemfs_mce_notifier); +if (ret) + goto reg_notifier_fail; +return 0 ; + +reg_notifier_fail: +unregister_filesystem (&dmemfs_fs_type); +reg_fs_fail:return ret; } static void __exit dmemfs_uninit(void) { +dmem_unregister_mce_notifier (&dmemfs_mce_notifier);unregister_filesystem (&dmemfs_fs_type); }
include/linux/dmem.h +7 -0
@@ -23,6 +23,13 @@ bool is_dmem_pfn(unsigned long pfn);#define dmem_free_page (addr) dmem_free_pages(addr, 1)bool dmem_memory_failure (unsigned long pfn,int flags); + +struct dmem_mce_notifier_info { +int flags; +}; + +int dmem_register_mce_notifier (struct notifier_block *nb); +int dmem_unregister_mce_notifier (struct notifier_block *nb);#else static inline int dmem_reserve_init (void ) {
include/linux/mm.h +2 -0
@@ -3006,6 +3006,8 @@ extern int memory_failure(unsigned long pfn, int flags);extern void memory_failure_queue (unsigned long pfn,int flags);extern void memory_failure_queue_kick (int cpu);extern int unpoison_memory (unsigned long pfn); +extern void collect_procs_and_signal_inode (struct inode *inode,pgoff_t pgoff, +unsigned long pfn,int flags);extern int get_hwpoison_page (struct page *page);#define put_hwpoison_page (page) put_page(page)extern int sysctl_memory_failure_early_kill;
mm/dmem.c +34 -0
@@ -70,6 +70,7 @@ struct dmem_node {struct dmem_pool {struct mutex lock; +struct raw_notifier_head mce_notifier_chain;unsigned long region_num;unsigned long registered_pages;
@@ -92,6 +93,7 @@ struct dmem_pool {static struct dmem_pool dmem_pool = { .lock =__MUTEX_INITIALIZER (dmem_pool.lock), + .mce_notifier_chain =RAW_NOTIFIER_INIT (dmem_pool.mce_notifier_chain), };#define DMEM_PAGE_SIZE (1UL << dmem_pool.dpage_shift)
@@ -121,6 +123,35 @@ static struct dmem_pool dmem_pool = {#define for_each_dmem_region (_dnode, _dregion) \ list_for_each_entry(_dregion, &(_dnode)->regions, node) +int dmem_register_mce_notifier (struct notifier_block *nb) +{ +int ret; + +mutex_lock (&dmem_pool.lock); + ret =raw_notifier_chain_register (&dmem_pool.mce_notifier_chain, nb); +mutex_unlock (&dmem_pool.lock); +return ret; +} +EXPORT_SYMBOL (dmem_register_mce_notifier); + +int dmem_unregister_mce_notifier (struct notifier_block *nb) +{ +int ret; + +mutex_lock (&dmem_pool.lock); + ret =raw_notifier_chain_unregister (&dmem_pool.mce_notifier_chain, nb); +mutex_unlock (&dmem_pool.lock); +return ret; +} +EXPORT_SYMBOL (dmem_unregister_mce_notifier); + +static int dmem_mce_notify (unsigned long pfn, +struct dmem_mce_notifier_info *info) +{ +return raw_notifier_call_chain (&dmem_pool.mce_notifier_chain, + pfn, info); +} +static inline int *dmem_nodelist (int nid) {return nid_to_dnode (nid)->nodelist;
@@ -1003,6 +1034,7 @@ bool dmem_memory_failure(unsigned long pfn, int flags)u64 pos;phys_addr_t addr =__pfn_to_phys (pfn);bool used = false; +struct dmem_mce_notifier_info info; dregion =find_dmem_region (addr, &pdnode);if (!dregion)
@@ -1022,6 +1054,8 @@ bool dmem_memory_failure(unsigned long pfn, int flags) pos =phys_to_dpage (addr) - dregion->dpage_start_pfn;if (__test_and_set_bit (pos, dregion->bitmap)) { used = true; + info.flags = flags; +dmem_mce_notify (pfn, &info); }else {pr_info ("MCE: free dpage, mark %#lx disabled in dmem\n" , pfn);dnode_count_free_dpages (pdnode,-1 );
mm/memory-failure.c +47 -16
113 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH 33/35 kvm, x86: temporary disable record_steal_time for dmem
In reply to [email protected]From: Yulei Zhang <[email protected]>
Temporarily disable record_steal_time when entering the guest for dmem.
--- arch/x86/kvm/x86.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kvm/x86.c +2 -0
@@ -2789,6 +2789,7 @@ static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu)static void record_steal_time (struct kvm_vcpu *vcpu) { +#if 0 struct kvm_host_map map;struct kvm_steal_time *st;
@@ -2830,6 +2831,7 @@ static void record_steal_time(struct kvm_vcpu *vcpu) st->version +=1 ;kvm_unmap_gfn (vcpu, &map, &vcpu->arch.st.cache, true, false); +#endif }int kvm_set_msr_common (struct kvm_vcpu *vcpu,struct msr_data *msr_info)
Signature · 1 line
2.28.0
PATCH 34/35 dmem: add dmem unit tests
In reply to [email protected]From: Yulei Zhang <[email protected]>
This test case is used to test dmem management system.
--- tools/testing/dmem/Kbuild | 1 + tools/testing/dmem/Makefile | 10 ++ tools/testing/dmem/dmem-test.c | 184 +++++++++++++++++++++++++++++++++ 3 files changed, 195 insertions(+) create mode 100644 tools/testing/dmem/Kbuild create mode 100644 tools/testing/dmem/Makefile create mode 100644 tools/testing/dmem/dmem-test.c
tools/testing/dmem/Kbuild new file +1 -0
@@ -0,0 +1 @@ +obj-m += dmem-test.o
tools/testing/dmem/Makefile new file +10 -0
@@ -0,0 +1,10 @@ +KDIR ?= ../../../ + +default: + $(MAKE) -C $(KDIR) M=$$PWD + +install: default + $(MAKE) -C $(KDIR) M=$$PWD modules_install + +clean: + rm -f *.o *.ko Module.* modules.* *.mod.c
tools/testing/dmem/dmem-test.c new file +184 -0
184 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH 35/35 Add documentation for dmemfs
In reply to [email protected]From: Yulei Zhang <[email protected]>
Introduce dmemfs.rst to document the basic usage of dmemfs.
--- Documentation/filesystems/dmemfs.rst | 59 ++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 Documentation/filesystems/dmemfs.rst
Documentation/filesystems/dmemfs.rst new file +57 -0
@@ -0,0 +1,57 @@ +.. SPDX-License-Identifier: GPL-2.0 + +===================================== +The Direct Memory Filesystem - DMEMFS +===================================== + + +.. Table of contents + + - Overview + - Compilation + - Usage + +Overview +======== + +Dmemfs (Direct Memory filesystem) is device memory or reserved +memory based filesystem. This kind of memory is special as it +is not managed by kernel and it is without 'struct page'. Therefore +it can save extra memory from the host system for various usage, +especially for guest virtual machines. + +It uses a kernel boot parameter ``dmem=`` to reserve the system +memory when the host system boots up, the details can be checked +in /Documentation/admin-guide/kernel-parameters.txt. + +Compilation +=========== + +The filesystem should be enabled by turning on the kernel configuration +options:: + + CONFIG_DMEM_FS - Direct Memory filesystem support + CONFIG_DMEM - Allow reservation of memory for dmem + + +Additionally, the following can be turned on to aid debugging:: + + CONFIG_DMEM_DEBUG_FS - Enable debug information for dmem + +Usage +======== + +Dmemfs supports mapping ``4K``, ``2M`` and ``1G`` size of pages to +the userspace, for example :: + + # mount -t dmemfs none -o pagesize=4K /mnt/ + +The it can create the backing storage with 4G size :: + + # truncate /mnt/dmemfs-uuid --size 4G + +To use as backing storage for virtual machine starts with qemu, just need +to specify the memory-backed-file in the qemu command line like this :: + + # -object memory-backend-file,id=ram-node0,mem-path=/mnt/dmemfs-uuid \ + share=yes,size=4G,host-nodes=0,policy=preferred -numa node,nodeid=0,memdev=ram-node0
Signature · 1 line
2.28.0
Randy Dunlap
reviewerRe: [PATCH 35/35] Add documentation for dmemfs
Quoted 4 lines · On 10/8/20 12:54 AM, [email protected] wrote:
From: Yulei Zhang <[email protected]>
Introduce
dmemfs.rstto document the basic usage of dmemfs.
Please add dmemfs as an entry in Documentation/filesystems/index.rst also.
Quoted 22 lines
Signed-off-by Yulei Zhang <[email protected]>--- Documentation/filesystems/dmemfs.rst | 59 ++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 Documentation/filesystems/dmemfs.rstDocumentation/filesystems/dmemfs.rst new file +10
-0@@ -0,0 +1,57 @@ +.. SPDX-License-Identifier: GPL-2.0 + +===================================== +The Direct Memory Filesystem - DMEMFS +===================================== + + +.. Table of contents + + - Overview
Documentation/filesystems/dmemfs.rst line 20
Quoted 10 lines
Documentation/filesystems/dmemfs.rst new file +10
-0+ - Compilation + - Usage + +Overview +======== + +Dmemfs (Direct Memory filesystem) is device memory or reserved +memory based filesystem. This kind of memory is special as it +is not managed by kernel and it is without 'struct page'. Therefore +it can save extra memory from the host system for various usage,
usages,
Documentation/filesystems/dmemfs.rst line 24
Quoted 4 lines
Documentation/filesystems/dmemfs.rst new file +4
-0+especially for guest virtual machines. + +It uses a kernel boot parameter ``dmem=`` to reserve the system +memory when the host system boots up, the details can be checked
up. The details
Documentation/filesystems/dmemfs.rst line 34
Quoted 10 lines
Documentation/filesystems/dmemfs.rst new file +10
-0+in /Documentation/admin-guide/kernel-parameters.txt. + +Compilation +=========== + +The filesystem should be enabled by turning on the kernel configuration +options:: + + CONFIG_DMEM_FS - Direct Memory filesystem support + CONFIG_DMEM - Allow reservation of memory for dmem
Hm, is there a good reason for having both of these options? Is one of them usable without the other one?
If not, there should only be one Kconfig option for DMEMFS.
Quoted 5 lines
Documentation/filesystems/dmemfs.rst new file +5
-0+ + +Additionally, the following can be turned on to aid debugging:: + + CONFIG_DMEM_DEBUG_FS - Enable debug information for dmem
Documentation/filesystems/dmemfs.rst line 49
Quoted 10 lines
Documentation/filesystems/dmemfs.rst new file +10
-0+ +Usage +======== + +Dmemfs supports mapping ``4K``, ``2M`` and ``1G`` size of pages to +the userspace, for example :: + + # mount -t dmemfs none -o pagesize=4K /mnt/ + +The it can create the backing storage with 4G size ::
Then
Quoted 9 lines
Documentation/filesystems/dmemfs.rst new file +8
-0+ + # truncate /mnt/dmemfs-uuid --size 4G + +To use as backing storage for virtual machine starts with qemu, just need +to specify the memory-backed-file in the qemu command line like this :: + + # -object memory-backend-file,id=ram-node0,mem-path=/mnt/dmemfs-uuid \ + share=yes,size=4G,host-nodes=0,policy=preferred -numa node,nodeid=0,memdev=ram-node0
Signature · 1 line
~Randy
[adding a couple folks that directly or indirectly work on the subject]
Quoted 8 lines · On 10/8/20 8:53 AM, [email protected] wrote:
From: Yulei Zhang <[email protected]>
In current system each physical memory page is assocaited with a page structure which is used to track the usage of this page. But due to the memory usage rapidly growing in cloud environment, we find the resource consuming for page structure storage becomes highly remarkable. So is it an expense that we could spare?
Happy to see another person working to solve the same problem!
I am really glad to see more folks being interested in solving this problem and I hope we can join efforts?
BTW, there is also a second benefit in removing struct page - which is carving out memory from the direct map.
Quoted 9 lines
This patchset introduces an idea about how to save the extra memory through a new virtual filesystem -- dmemfs.
Dmemfs (Direct Memory filesystem) is device memory or reserved memory based filesystem. This kind of memory is special as it is not managed by kernel and most important it is without 'struct page'. Therefore we can leverage the extra memory from the host system to support more tenants in our cloud service.
This is like a walk down the memory lane.
About a year ago we followed the same exact idea/motivation to have memory outside of the direct map (and removing struct page overhead) and started with our own layer/thingie. However we realized that DAX is one the subsystems which already gives you direct access to memory for free (and is already upstream), plus a couple of things which we found more handy.
So we sent an RFC a couple months ago:
https://lore.kernel.org/linux-mm/[email protected]/
Since then majority of the work has been in improving DAX[1]. But now that is done I am going to follow up with the above patchset.
(Give me a couple of days and I will send you the link to the latest patches on a git-tree - would love feedback!)
The struct page removal for DAX would then be small, and ticks the same bells and whistles (MCE handling, reserving PAT memtypes, ptrace support) that we both do, with a smaller diffstat and it doesn't touch KVM (not at least fundamentally).
15 files changed, 401 insertions(+), 38 deletions(-)
The things needed in core-mm is for handling PMD/PUD PAGE_SPECIAL much like we both do. Furthermore there wouldn't be a need for a new vm type, consuming an extra page bit (in addition to PAGE_SPECIAL) or new filesystem.
Quoted 8 lines
We uses a kernel boot parameter 'dmem=' to reserve the system memory when the host system boots up, the details can be checked in /Documentation/admin-guide/kernel-parameters.txt.
Theoretically for each 4k physical page it can save 64 bytes if we drop the 'struct page', so for guest memory with 320G it can save about 5G physical memory totally.
Also worth mentioning that if you only care about 'struct page' cost, and not on the security boundary, there's also some work on hugetlbfs preallocation of hugepages into tricking vmemmap in reusing tail pages.
https://lore.kernel.org/linux-mm/[email protected]/
Going forward that could also make sense for device-dax to avoid so many struct pages allocated (which would require its transition to compound struct pages like hugetlbfs which we are looking at too). In addition an idea <handwaving> would be perhaps to have a stricter mode in DAX where we initialize/use the metadata ('struct page') but remove the underlaying PFNs (of the 'struct page') from the direct map having to bear the cost of mapping/unmapping on gup/pup.
Joao
Joao, thanks a lot for the feedback. One more thing needs to mention is that dmemfs also support fine-grained
memory management which makes it more flexible for tenants with different requirements.
Quoted 89 lines · Joao Martins
[adding a couple folks that directly or indirectly work on the subject]
Quoted 8 lines · On 10/8/20 8:53 AM, [email protected] wrote:
From: Yulei Zhang <[email protected]>
In current system each physical memory page is assocaited with a page structure which is used to track the usage of this page. But due to the memory usage rapidly growing in cloud environment, we find the resource consuming for page structure storage becomes highly remarkable. So is it an expense that we could spare?
Happy to see another person working to solve the same problem!
I am really glad to see more folks being interested in solving this problem and I hope we can join efforts?
BTW, there is also a second benefit in removing struct page - which is carving out memory from the direct map.
Quoted 9 lines
This patchset introduces an idea about how to save the extra memory through a new virtual filesystem -- dmemfs.
Dmemfs (Direct Memory filesystem) is device memory or reserved memory based filesystem. This kind of memory is special as it is not managed by kernel and most important it is without 'struct page'. Therefore we can leverage the extra memory from the host system to support more tenants in our cloud service.
This is like a walk down the memory lane.
About a year ago we followed the same exact idea/motivation to have memory outside of the direct map (and removing struct page overhead) and started with our own layer/thingie. However we realized that DAX is one the subsystems which already gives you direct access to memory for free (and is already upstream), plus a couple of things which we found more handy.
So we sent an RFC a couple months ago:
https://lore.kernel.org/linux-mm/[email protected]/
Since then majority of the work has been in improving DAX[1]. But now that is done I am going to follow up with the above patchset.
(Give me a couple of days and I will send you the link to the latest patches on a git-tree - would love feedback!)
The struct page removal for DAX would then be small, and ticks the same bells and whistles (MCE handling, reserving PAT memtypes, ptrace support) that we both do, with a smaller diffstat and it doesn't touch KVM (not at least fundamentally).
15 files changed, 401 insertions(+), 38 deletions(-)The things needed in core-mm is for handling PMD/PUD
PAGE_SPECIALmuch like we both do. Furthermore there wouldn't be a need for a new vm type, consuming an extra page bit (in addition toPAGE_SPECIAL) or new filesystem.Quoted 8 lines
We uses a kernel boot parameter 'dmem=' to reserve the system memory when the host system boots up, the details can be checked in /Documentation/admin-guide/kernel-parameters.txt.
Theoretically for each 4k physical page it can save 64 bytes if we drop the 'struct page', so for guest memory with 320G it can save about 5G physical memory totally.
Also worth mentioning that if you only care about 'struct page' cost, and not on the security boundary, there's also some work on hugetlbfs preallocation of hugepages into tricking vmemmap in reusing tail pages.
https://lore.kernel.org/linux-mm/[email protected]/
Going forward that could also make sense for device-dax to avoid so many struct pages allocated (which would require its transition to compound struct pages like hugetlbfs which we are looking at too). In addition an idea <handwaving> would be perhaps to have a stricter mode in DAX where we initialize/use the metadata ('struct page') but remove the underlaying PFNs (of the 'struct page') from the direct map having to bear the cost of mapping/unmapping on gup/pup.
Joao
Quoted 5 lines · yulei zhang
Joao, thanks a lot for the feedback. One more thing needs to mention is that dmemfs also support fine-grained
memory management which makes it more flexible for tenants with different requirements.
So as DAX when it allows to partition a region (starting 5.10). Meaning you have a region which you dedicated to userspace. That region can then be partitioning into devices which give you access to multiple (possibly discontinuous) extents with at a given page granularity (selectable when you create the device), accessed through mmap(). You can then give that device to a cgroup. Or you can return that memory back to the kernel (should you run into OOM situation), or you recreate the same mappings across reboot/kexec.
I probably need to read your patches again, but can you extend on the 'dmemfs also support fine-grained memory management' to understand what is the gap that you mention?
Quoted 90 lines · Joao Martins
Quoted 89 lines
[adding a couple folks that directly or indirectly work on the subject]
Quoted 8 lines · On 10/8/20 8:53 AM, [email protected] wrote:
From: Yulei Zhang <[email protected]>
In current system each physical memory page is assocaited with a page structure which is used to track the usage of this page. But due to the memory usage rapidly growing in cloud environment, we find the resource consuming for page structure storage becomes highly remarkable. So is it an expense that we could spare?
Happy to see another person working to solve the same problem!
I am really glad to see more folks being interested in solving this problem and I hope we can join efforts?
BTW, there is also a second benefit in removing struct page - which is carving out memory from the direct map.
Quoted 9 lines
This patchset introduces an idea about how to save the extra memory through a new virtual filesystem -- dmemfs.
Dmemfs (Direct Memory filesystem) is device memory or reserved memory based filesystem. This kind of memory is special as it is not managed by kernel and most important it is without 'struct page'. Therefore we can leverage the extra memory from the host system to support more tenants in our cloud service.
This is like a walk down the memory lane.
About a year ago we followed the same exact idea/motivation to have memory outside of the direct map (and removing struct page overhead) and started with our own layer/thingie. However we realized that DAX is one the subsystems which already gives you direct access to memory for free (and is already upstream), plus a couple of things which we found more handy.
So we sent an RFC a couple months ago:
https://lore.kernel.org/linux-mm/[email protected]/
Since then majority of the work has been in improving DAX[1]. But now that is done I am going to follow up with the above patchset.
(Give me a couple of days and I will send you the link to the latest patches on a git-tree - would love feedback!)
The struct page removal for DAX would then be small, and ticks the same bells and whistles (MCE handling, reserving PAT memtypes, ptrace support) that we both do, with a smaller diffstat and it doesn't touch KVM (not at least fundamentally).
15 files changed, 401 insertions(+), 38 deletions(-)The things needed in core-mm is for handling PMD/PUD
PAGE_SPECIALmuch like we both do. Furthermore there wouldn't be a need for a new vm type, consuming an extra page bit (in addition toPAGE_SPECIAL) or new filesystem.Quoted 8 lines
We uses a kernel boot parameter 'dmem=' to reserve the system memory when the host system boots up, the details can be checked in /Documentation/admin-guide/kernel-parameters.txt.
Theoretically for each 4k physical page it can save 64 bytes if we drop the 'struct page', so for guest memory with 320G it can save about 5G physical memory totally.
Also worth mentioning that if you only care about 'struct page' cost, and not on the security boundary, there's also some work on hugetlbfs preallocation of hugepages into tricking vmemmap in reusing tail pages.
https://lore.kernel.org/linux-mm/[email protected]/
Going forward that could also make sense for device-dax to avoid so many struct pages allocated (which would require its transition to compound struct pages like hugetlbfs which we are looking at too). In addition an idea <handwaving> would be perhaps to have a stricter mode in DAX where we initialize/use the metadata ('struct page') but remove the underlaying PFNs (of the 'struct page') from the direct map having to bear the cost of mapping/unmapping on gup/pup.
Joao
Quoted 15 lines · Joao Martins
Quoted 5 lines · yulei zhang
Joao, thanks a lot for the feedback. One more thing needs to mention is that dmemfs also support fine-grained
memory management which makes it more flexible for tenants with different requirements.So as DAX when it allows to partition a region (starting 5.10). Meaning you have a region which you dedicated to userspace. That region can then be partitioning into devices which give you access to multiple (possibly discontinuous) extents with at a given page granularity (selectable when you create the device), accessed through
mmap(). You can then give that device to a cgroup. Or you can return that memory back to the kernel (should you run into OOM situation), or you recreate the same mappings across reboot/kexec.
Quoted 3 lines
I probably need to read your patches again, but can you extend on the 'dmemfs also support fine-grained memory management' to understand what is the gap that you mention?
sure, dmemfs uses bitmap to track the memory usage in the reserved memory region in
a given page size granularity. And for each user the memory can be discrete as well.
Quoted 90 lines
Quoted 90 lines · Joao Martins
Quoted 89 lines
[adding a couple folks that directly or indirectly work on the subject]
Quoted 8 lines · On 10/8/20 8:53 AM, [email protected] wrote:
From: Yulei Zhang <[email protected]>
In current system each physical memory page is assocaited with a page structure which is used to track the usage of this page. But due to the memory usage rapidly growing in cloud environment, we find the resource consuming for page structure storage becomes highly remarkable. So is it an expense that we could spare?
Happy to see another person working to solve the same problem!
I am really glad to see more folks being interested in solving this problem and I hope we can join efforts?
BTW, there is also a second benefit in removing struct page - which is carving out memory from the direct map.
Quoted 9 lines
This patchset introduces an idea about how to save the extra memory through a new virtual filesystem -- dmemfs.
Dmemfs (Direct Memory filesystem) is device memory or reserved memory based filesystem. This kind of memory is special as it is not managed by kernel and most important it is without 'struct page'. Therefore we can leverage the extra memory from the host system to support more tenants in our cloud service.
This is like a walk down the memory lane.
About a year ago we followed the same exact idea/motivation to have memory outside of the direct map (and removing struct page overhead) and started with our own layer/thingie. However we realized that DAX is one the subsystems which already gives you direct access to memory for free (and is already upstream), plus a couple of things which we found more handy.
So we sent an RFC a couple months ago:
https://lore.kernel.org/linux-mm/[email protected]/
Since then majority of the work has been in improving DAX[1]. But now that is done I am going to follow up with the above patchset.
(Give me a couple of days and I will send you the link to the latest patches on a git-tree - would love feedback!)
The struct page removal for DAX would then be small, and ticks the same bells and whistles (MCE handling, reserving PAT memtypes, ptrace support) that we both do, with a smaller diffstat and it doesn't touch KVM (not at least fundamentally).
15 files changed, 401 insertions(+), 38 deletions(-)The things needed in core-mm is for handling PMD/PUD
PAGE_SPECIALmuch like we both do. Furthermore there wouldn't be a need for a new vm type, consuming an extra page bit (in addition toPAGE_SPECIAL) or new filesystem.Quoted 8 lines
We uses a kernel boot parameter 'dmem=' to reserve the system memory when the host system boots up, the details can be checked in /Documentation/admin-guide/kernel-parameters.txt.
Theoretically for each 4k physical page it can save 64 bytes if we drop the 'struct page', so for guest memory with 320G it can save about 5G physical memory totally.
Also worth mentioning that if you only care about 'struct page' cost, and not on the security boundary, there's also some work on hugetlbfs preallocation of hugepages into tricking vmemmap in reusing tail pages.
https://lore.kernel.org/linux-mm/[email protected]/
Going forward that could also make sense for device-dax to avoid so many struct pages allocated (which would require its transition to compound struct pages like hugetlbfs which we are looking at too). In addition an idea <handwaving> would be perhaps to have a stricter mode in DAX where we initialize/use the metadata ('struct page') but remove the underlaying PFNs (of the 'struct page') from the direct map having to bear the cost of mapping/unmapping on gup/pup.
Joao
Quoted 15 lines · yulei zhang
Quoted 14 lines · Joao Martins
Quoted 5 lines · yulei zhang
Joao, thanks a lot for the feedback. One more thing needs to mention is that dmemfs also support fine-grained
memory management which makes it more flexible for tenants with different requirements.So as DAX when it allows to partition a region (starting 5.10). Meaning you have a region which you dedicated to userspace. That region can then be partitioning into devices which give you access to multiple (possibly discontinuous) extents with at a given page granularity (selectable when you create the device), accessed through
mmap(). You can then give that device to a cgroup. Or you can return that memory back to the kernel (should you run into OOM situation), or you recreate the same mappings across reboot/kexec.
Quoted 8 lines
Quoted 3 lines
I probably need to read your patches again, but can you extend on the 'dmemfs also support fine-grained memory management' to understand what is the gap that you mention?
sure, dmemfs uses bitmap to track the memory usage in the reserved memory region in
a given page size granularity. And for each user the memory can be discrete as well.
That same functionality of tracking reserved region usage across different users at any page granularity is covered the DAX series I mentioned below. The discrete part -- IIUC what you meant -- is then reduced using DAX ABI/tools to create a device file vs a filesystem.
Quoted 90 lines
Quoted 90 lines
Quoted 90 lines · Joao Martins
Quoted 89 lines
[adding a couple folks that directly or indirectly work on the subject]
Quoted 8 lines · On 10/8/20 8:53 AM, [email protected] wrote:
From: Yulei Zhang <[email protected]>
In current system each physical memory page is assocaited with a page structure which is used to track the usage of this page. But due to the memory usage rapidly growing in cloud environment, we find the resource consuming for page structure storage becomes highly remarkable. So is it an expense that we could spare?
Happy to see another person working to solve the same problem!
I am really glad to see more folks being interested in solving this problem and I hope we can join efforts?
BTW, there is also a second benefit in removing struct page - which is carving out memory from the direct map.
Quoted 9 lines
This patchset introduces an idea about how to save the extra memory through a new virtual filesystem -- dmemfs.
Dmemfs (Direct Memory filesystem) is device memory or reserved memory based filesystem. This kind of memory is special as it is not managed by kernel and most important it is without 'struct page'. Therefore we can leverage the extra memory from the host system to support more tenants in our cloud service.
This is like a walk down the memory lane.
About a year ago we followed the same exact idea/motivation to have memory outside of the direct map (and removing struct page overhead) and started with our own layer/thingie. However we realized that DAX is one the subsystems which already gives you direct access to memory for free (and is already upstream), plus a couple of things which we found more handy.
So we sent an RFC a couple months ago:
https://lore.kernel.org/linux-mm/[email protected]/
Since then majority of the work has been in improving DAX[1]. But now that is done I am going to follow up with the above patchset.
(Give me a couple of days and I will send you the link to the latest patches on a git-tree - would love feedback!)
The struct page removal for DAX would then be small, and ticks the same bells and whistles (MCE handling, reserving PAT memtypes, ptrace support) that we both do, with a smaller diffstat and it doesn't touch KVM (not at least fundamentally).
15 files changed, 401 insertions(+), 38 deletions(-)The things needed in core-mm is for handling PMD/PUD
PAGE_SPECIALmuch like we both do. Furthermore there wouldn't be a need for a new vm type, consuming an extra page bit (in addition toPAGE_SPECIAL) or new filesystem.Quoted 8 lines
We uses a kernel boot parameter 'dmem=' to reserve the system memory when the host system boots up, the details can be checked in /Documentation/admin-guide/kernel-parameters.txt.
Theoretically for each 4k physical page it can save 64 bytes if we drop the 'struct page', so for guest memory with 320G it can save about 5G physical memory totally.
Also worth mentioning that if you only care about 'struct page' cost, and not on the security boundary, there's also some work on hugetlbfs preallocation of hugepages into tricking vmemmap in reusing tail pages.
https://lore.kernel.org/linux-mm/[email protected]/
Going forward that could also make sense for device-dax to avoid so many struct pages allocated (which would require its transition to compound struct pages like hugetlbfs which we are looking at too). In addition an idea <handwaving> would be perhaps to have a stricter mode in DAX where we initialize/use the metadata ('struct page') but remove the underlaying PFNs (of the 'struct page') from the direct map having to bear the cost of mapping/unmapping on gup/pup.
Joao
On Mon, Oct 12, 2020 at 4:00 AM Joao Martins <[email protected]> wrote:
[..]
Quoted 19 lines · yulei zhang
Quoted 18 lines · Joao Martins
Quoted 17 lines · yulei zhang
Quoted 5 lines
Joao, thanks a lot for the feedback. One more thing needs to mention is that dmemfs also support fine-grained
memory management which makes it more flexible for tenants with different requirements.So as DAX when it allows to partition a region (starting 5.10). Meaning you have a region which you dedicated to userspace. That region can then be partitioning into devices which give you access to multiple (possibly discontinuous) extents with at a given page granularity (selectable when you create the device), accessed through
mmap(). You can then give that device to a cgroup. Or you can return that memory back to the kernel (should you run into OOM situation), or you recreate the same mappings across reboot/kexec.I probably need to read your patches again, but can you extend on the 'dmemfs also support fine-grained memory management' to understand what is the gap that you mention?
Quoted 8 lines
Quoted 5 lines
sure, dmemfs uses bitmap to track the memory usage in the reserved memory region in
a given page size granularity. And for each user the memory can be discrete as well.That same functionality of tracking reserved region usage across different users at any page granularity is covered the DAX series I mentioned below. The discrete part -- IIUC what you meant -- is then reduced using DAX ABI/tools to create a device file vs a filesystem.
Put another way. Linux already has a fine grained memory management system, the page allocator. Now, with recent device-dax extensions, it also has a coarse grained memory management system for physical address-space partitioning and a path for struct-page-less backing for VMs. What feature gaps remain vs dmemfs, and can those gaps be closed with incremental improvements to the 2 existing memory-management systems?
Paolo Bonzini
maintainerQuoted 6 lines · Dan Williams
Now, with recent device-dax extensions, it
also has a coarse grained memory management system for physical address-space partitioning and a path for struct-page-less backing for VMs. What feature gaps remain vs dmemfs, and can those gaps be closed with incremental improvements to the 2 existing memory-management systems?
If I understand correctly, devm_memremap_pages() on ZONE_DEVICE memory would still create the "struct page" albeit lazily? KVM then would use the usual get_user_pages() path.
Looking more closely at the implementation of dmemfs, I don't understand is why dmemfs needs VM_DMEM etc. and cannot provide access to mmap-ed memory using remap_pfn_range and VM_PFNMAP, just like /dev/mem. If it did that KVM would get physical addresses using fixup_user_fault and never need pfn_to_page() or get_user_pages(). I'm not saying that would instantly be an approval, but it would make remove a lot of hooks.
Paolo
Quoted 8 lines · Paolo Bonzini
Quoted 6 lines · Dan Williams
Now, with recent device-dax extensions, it
also has a coarse grained memory management system for physical address-space partitioning and a path for struct-page-less backing for VMs. What feature gaps remain vs dmemfs, and can those gaps be closed with incremental improvements to the 2 existing memory-management systems?
Quoted 4 lines
If I understand correctly,
devm_memremap_pages()onZONE_DEVICEmemory would still create the "struct page" albeit lazily? KVM then would use the usualget_user_pages()path.
Correct.
The removal of struct page would be one of the added incremental improvements, like a 'map' with 'raw' sysfs attribute for dynamic dax regions that wouldn't online/create the struct pages. The remaining plumbing (...)
Quoted 7 lines
Looking more closely at the implementation of dmemfs, I don't understand is why dmemfs needs
VM_DMEMetc. and cannot provide access to mmap-ed memory usingremap_pfn_rangeandVM_PFNMAP, just like/dev/mem. If it did that KVM would get physical addresses usingfixup_user_faultand never needpfn_to_page()orget_user_pages(). I'm not saying that would instantly be an approval, but it would make remove a lot of hooks.
(...) is similar to what you describe above. Albeit there's probably no need to do a remap_pfn_range at mmap(), as DAX supplies a fault/huge_fault. Also, using that means it's limited to a single contiguous PFN chunk.
KVM has the bits to make it work without struct pages, I don't think there's a need for new pg/pfn_t/VM_* bits (aside from relying on {PFN,PAGE}_SPECIAL) as mentioned at the start of the thread. I'm storing my wip here:
https://github.com/jpemartins/linux pageless-dax
Which is based on the first series that had been submitted earlier this year:
https://lore.kernel.org/kvm/[email protected]/
Joao
Quoted 36 lines · Joao Martins
Quoted 8 lines · Paolo Bonzini
Quoted 6 lines · Dan Williams
Now, with recent device-dax extensions, it
also has a coarse grained memory management system for physical address-space partitioning and a path for struct-page-less backing for VMs. What feature gaps remain vs dmemfs, and can those gaps be closed with incremental improvements to the 2 existing memory-management systems?Quoted 4 lines
If I understand correctly,
devm_memremap_pages()onZONE_DEVICEmemory would still create the "struct page" albeit lazily? KVM then would use the usualget_user_pages()path.Correct.
The removal of struct page would be one of the added incremental improvements, like a 'map' with 'raw' sysfs attribute for dynamic dax regions that wouldn't online/create the struct pages. The remaining plumbing (...)
Quoted 7 lines
Looking more closely at the implementation of dmemfs, I don't understand is why dmemfs needs
VM_DMEMetc. and cannot provide access to mmap-ed memory usingremap_pfn_rangeandVM_PFNMAP, just like/dev/mem. If it did that KVM would get physical addresses usingfixup_user_faultand never needpfn_to_page()orget_user_pages(). I'm not saying that would instantly be an approval, but it would make remove a lot of hooks.(...) is similar to what you describe above. Albeit there's probably no need to do a
remap_pfn_rangeatmmap(), as DAX supplies a fault/huge_fault. Also, using that means it's limited to a single contiguous PFN chunk.KVM has the bits to make it work without struct pages, I don't think there's a need for new pg/pfn_t/VM_* bits (aside from relying on {PFN,PAGE}_SPECIAL) as mentioned at the start of the thread. I'm storing my wip here:
Quoted 7 lines
https://github.com/jpemartins/linux pageless-dax
Which is based on the first series that had been submitted earlier this year:
https://lore.kernel.org/kvm/[email protected]/Joao
Just as Joao mentioned, remap_pfn_range() may request a single contiguous PFN range, which
is not our intention. And for VM_DMEM, I think we may drop it in the next version, and to use the
existing bits as much as possible to minimize the modifications.
Quoted 28 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Thursday, October 08, 2020 3:54 PM
To: [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yulei Zhang
Subject: [PATCH 00/35] Enhance memory utilization with DMEMFSFrom: Yulei Zhang <[email protected]>
In current system each physical memory page is assocaited with a page structure which is used to track the usage of this page. But due to the memory usage rapidly growing in cloud environment, we find the resource consuming for page structure storage becomes highly remarkable. So is it an expense that we could spare?
This patchset introduces an idea about how to save the extra memory through a new virtual filesystem -- dmemfs.
Dmemfs (Direct Memory filesystem) is device memory or reserved memory based filesystem. This kind of memory is special as it is not managed by kernel and most important it is without 'struct page'. Therefore we can leverage the extra memory from the host system to support more tenants in our cloud service.
Quoted 7 lines
We uses a kernel boot parameter 'dmem=' to reserve the system memory when the host system boots up, the details can be checked in /Documentation/admin-guide/kernel-parameters.txt.
Theoretically for each 4k physical page it can save 64 bytes if we drop the 'struct page', so for guest memory with 320G it can save about 5G physical memory totally.
Sounds interesting, but seems your patch only support x86, have you
considered aarch64?
Regards
Zengtao
Quoted 37 lines · Zengtao (B)
Quoted 29 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Thursday, October 08, 2020 3:54 PM
To: [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Yulei Zhang
Subject: [PATCH 00/35] Enhance memory utilization with DMEMFSFrom: Yulei Zhang <[email protected]>
In current system each physical memory page is assocaited with a page structure which is used to track the usage of this page. But due to the memory usage rapidly growing in cloud environment, we find the resource consuming for page structure storage becomes highly remarkable. So is it an expense that we could spare?
This patchset introduces an idea about how to save the extra memory through a new virtual filesystem -- dmemfs.
Dmemfs (Direct Memory filesystem) is device memory or reserved memory based filesystem. This kind of memory is special as it is not managed by kernel and most important it is without 'struct page'. Therefore we can leverage the extra memory from the host system to support more tenants in our cloud service.
Quoted 7 lines
We uses a kernel boot parameter 'dmem=' to reserve the system memory when the host system boots up, the details can be checked in /Documentation/admin-guide/kernel-parameters.txt.
Theoretically for each 4k physical page it can save 64 bytes if we drop the 'struct page', so for guest memory with 320G it can save about 5G physical memory totally.
Quoted 5 lines
Sounds interesting, but seems your patch only support x86, have you
considered aarch64?Regards
Zengtao
Thanks, so far we only verify it on x86 server, may extend to arm platform in the future.