RAM_NORESERVE, MAP_NORESERVE and hostmem "reserve" property
Subsystems (3)
v2 cover RAM_NORESERVE, MAP_NORESERVE and hostmem "reserve" property
Some cleanups previously sent in other context (resizeable allocations), followed by RAM_NORESERVE, implementing it under POSIX using MAP_NORESERVE, and letting users configure it for memory backens using the "reserve" property (default: true).
MAP_NORESERVE under Linux has in the context of QEMU an effect on
1) Private anonymous memory
-> memory-backend-ram,id=mem0,size=10G
2) Private file-based mappings
-> memory-backend-file,id=mem0,size=10G,mem-path=/dev/shm/0
3) Private/shared hugetlbfs memory
-> memory-backend-memfd,id=mem0,size=10G,hugetlb=on,hugetlbsize=2M
With MAP_NORESERVE/"reserve=off", we won't be reserving swap space (1/2) or huge pages (3) for the whole memory region.
The target use case is virtio-mem, which dynamically exposes memory inside a large, sparse memory area to the VM. MAP_NORESERVE tells the OS "this mapping might be very sparse". This essentially allows avoiding to set "/proc/sys/vm/overcommit_memory == 1") when using virtio-mem and also supporting hugetlbfs in the future.
virtio-mem currently only supports anonymous memory, in the future we want to also support private memfd, shared file-based and shared hugetlbfs mappings.
virtio-mem features I am currently working on include:
1. Introducing a prealloc option for virtio-mem (e.g., using fallocate() when plugging blocks) to fail nicely when running out of backing storage like huge pages.
2. Supporting resizable RAM block/memmory regions, such that we won't always expose a large, sparse memory region to the VM.
3. Handling virtio-mem requests via an iothread to not hold the BQL while populating/preallocating memory
4. Protecting unplugged memory e.g., using userfaultfd.
5. (resizeable allocations / optimized mmap handling when resizing RAM blocks)
Based-on: [email protected]
v1 -> v2:
- Rebased to upstream and phs_mem_alloc simplifications
-- Upsteam added the "map_offset" parameter to many RAM allocation
interfaces.
- "softmmu/physmem: Drop "shared" parameter from ram_block_add()" -- Use local variable "shared"
- "memory: introduce RAM_NORESERVE and wire it up in qemu_ram_mmap()" -- Simplify due to phs_mem_alloc changes
- "util/mmap-alloc: Support RAM_NORESERVE via MAP_NORESERVE" -- Add a whole bunch of comments.
-- Exclude shared anonymous memory that QEMU doesn't use -- Special-case readonly mappings
David Hildenbrand (9):
softmmu/physmem: Drop "shared" parameter from ram_block_add() util/mmap-alloc: Factor out calculation of the pagesize for the guard
page
util/mmap-alloc: Factor out reserving of a memory region to
mmap_reserve()
util/mmap-alloc: Factor out activating of memory to mmap_activate() softmmu/memory: Pass ram_flags into qemu_ram_alloc_from_fd() softmmu/memory: Pass ram_flags into
memory_region_init_ram_shared_nomigrate()
memory: introduce RAM_NORESERVE and wire it up in qemu_ram_mmap() util/mmap-alloc: Support RAM_NORESERVE via MAP_NORESERVE
hostmem: Wire up RAM_NORESERVE via "reserve" property
backends/hostmem-file.c | 11 +- backends/hostmem-memfd.c | 8 +- backends/hostmem-ram.c | 7 +- backends/hostmem.c | 33 +++ hw/m68k/next-cube.c | 4 +- hw/misc/ivshmem.c | 5 +- include/exec/cpu-common.h | 1 + include/exec/memory.h | 43 ++-- include/exec/ram_addr.h | 9 +- include/qemu/mmap-alloc.h | 2 + include/qemu/osdep.h | 3 +- include/sysemu/hostmem.h | 2 +- migration/ram.c | 3 +- .../memory-region-housekeeping.cocci | 8 +- softmmu/memory.c | 27 ++- softmmu/physmem.c | 41 ++-- util/mmap-alloc.c | 220 ++++++++++++------ util/oslib-posix.c | 6 +- util/oslib-win32.c | 13 +- 19 files changed, 299 insertions(+), 147 deletions(-)
Signature · 1 line
2.29.2
PATCH v2 1/9 softmmu/physmem: Drop "shared" parameter from ram_block_add()
Properly store it in the flags of the ram block instead (and the flag even already exists and is used).
E.g., qemu_ram_is_shared() now properly succeeds on all ram blocks that are actually shared.
--- softmmu/physmem.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
softmmu/physmem.c +7 -3
@@ -1927,8 +1927,9 @@ static void dirty_memory_extend(ram_addr_t old_ram_size, } } -static void ram_block_add (RAMBlock *new_block,Error **errp,bool shared) +static void ram_block_add (RAMBlock *new_block,Error **errp) { +const bool shared =qemu_ram_is_shared (new_block);RAMBlock *block;RAMBlock *last_block =NULL ;ram_addr_t old_ram_size, new_ram_size;
@@ -2064,7 +2065,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,return NULL ; } -ram_block_add (new_block, &local_err, ram_flags &RAM_SHARED ); +ram_block_add (new_block, &local_err);if (local_err) {g_free (new_block);error_propagate (errp, local_err);
@@ -2127,10 +2128,13 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,if (host) { new_block->flags |=RAM_PREALLOC ; } +if (share) { + new_block->flags |=RAM_SHARED ; + }if (resizeable) { new_block->flags |=RAM_RESIZEABLE ; } -ram_block_add (new_block, &local_err, share); +ram_block_add (new_block, &local_err);if (local_err) {g_free (new_block);error_propagate (errp, local_err);
Signature · 1 line
2.29.2
PATCH v2 2/9 util/mmap-alloc: Factor out calculation of the pagesize for the guard page
In reply to David HildenbrandLet's factor out calculating the size of the guard page and rename the variable to make it clearer that this pagesize only applies to the guard page.
--- util/mmap-alloc.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-)
util/mmap-alloc.c +16 -15
@@ -82,6 +82,16 @@ size_t qemu_mempath_getpagesize(const char *mem_path)return qemu_real_host_page_size; } +static inline size_t mmap_guard_pagesize (int fd) +{ +#if defined(__powerpc64__) && defined(__linux__) +/* Mappings in the same segment must share the same page size */ +return qemu_fd_getpagesize (fd); +#else +return qemu_real_host_page_size; +#endif +} + void *qemu_ram_mmap(int fd, size_t size, size_t align,
@@ -90,12 +100,12 @@ void *qemu_ram_mmap(int fd, bool is_pmem, off_t map_offset) { +const size_t guard_pagesize =mmap_guard_pagesize (fd);int prot;int flags;int map_sync_flags =0 ;int guardfd;size_t offset; -size_t pagesize;size_t total;void *guardptr;void *ptr;
@@ -116,8 +126,7 @@ void *qemu_ram_mmap(int fd,* anonymous memory is OK. */ flags =MAP_PRIVATE ; - pagesize =qemu_fd_getpagesize (fd); -if (fd ==-1 || pagesize == qemu_real_host_page_size) { +if (fd ==-1 || guard_pagesize == qemu_real_host_page_size) { guardfd =-1 ; flags |=MAP_ANONYMOUS ; }else {
@@ -126,7 +135,6 @@ void *qemu_ram_mmap(int fd, }#else guardfd =-1 ; - pagesize = qemu_real_host_page_size; flags =MAP_PRIVATE |MAP_ANONYMOUS ;#endif
@@ -138,7 +146,7 @@ void *qemu_ram_mmap(int fd,assert (is_power_of_2 (align));/* Always align to host page size */ -assert (align >= pagesize); +assert (align >= guard_pagesize); flags =MAP_FIXED ; flags |= fd ==-1 ?MAP_ANONYMOUS :0 ;
@@ -193,8 +201,8 @@ void *qemu_ram_mmap(int fd,* a guard page guarding against potential buffer overflows. */ total -= offset; -if (total > size + pagesize) { -munmap (ptr + size + pagesize, total - size - pagesize); +if (total > size + guard_pagesize) { +munmap (ptr + size + guard_pagesize, total - size - guard_pagesize); }return ptr;
@@ -202,15 +210,8 @@ void *qemu_ram_mmap(int fd,void qemu_ram_munmap (int fd,void *ptr,size_t size) { -size_t pagesize; -if (ptr) {/* Unmap both the RAM block and the guard page */ -#if defined(__powerpc64__) && defined(__linux__) - pagesize =qemu_fd_getpagesize (fd); -#else - pagesize = qemu_real_host_page_size; -#endif -munmap (ptr, size + pagesize); +munmap (ptr, size +mmap_guard_pagesize (fd)); } }
Signature · 1 line
2.29.2
PATCH v2 3/9 util/mmap-alloc: Factor out reserving of a memory region to mmap_reserve()
In reply to David HildenbrandWe want to reserve a memory region without actually populating memory. Let's factor that out.
--- util/mmap-alloc.c | 58 +++++++++++++++++++++++++++-------------------- 1 file changed, 33 insertions(+), 25 deletions(-)
util/mmap-alloc.c +33 -25
@@ -82,6 +82,38 @@ size_t qemu_mempath_getpagesize(const char *mem_path)return qemu_real_host_page_size; } +/* +* Reserve a new memory region of the requested size to be used for mapping +* from the given fd (if any). +*/ +static void *mmap_reserve (size_t size,int fd) +{ +int flags =MAP_PRIVATE ; + +#if defined(__powerpc64__) && defined(__linux__) +/* +* On ppc64 mappings in the same segment (aka slice) must share the same +* page size. Since we will be re-allocating part of this segment +* from the supplied fd, we should make sure to use the same page size, to +* this end we mmap the supplied fd. In this case, set MAP_NORESERVE to +* avoid allocating backing store memory. +* We do this unless we are using the system page size, in which case +* anonymous memory is OK. +*/ +if (fd ==-1 ||qemu_fd_getpagesize (fd) == qemu_real_host_page_size) { + fd =-1 ; + flags |=MAP_ANONYMOUS ; + }else { + flags |=MAP_NORESERVE ; + } +#else + fd =-1 ; + flags |=MAP_ANONYMOUS ; +#endif + +return mmap (0 , size,PROT_NONE , flags, fd,0 ); +} +static inline size_t mmap_guard_pagesize (int fd) {#if defined(__powerpc64__) && defined(__linux__)
@@ -104,7 +136,6 @@ void *qemu_ram_mmap(int fd,int prot;int flags;int map_sync_flags =0 ; -int guardfd;size_t offset;size_t total;void *guardptr;
@@ -116,30 +147,7 @@ void *qemu_ram_mmap(int fd,*/ total = size + align; -#if defined(__powerpc64__) && defined(__linux__) -/* On ppc64 mappings in the same segment (aka slice) must share the same -* page size. Since we will be re-allocating part of this segment -* from the supplied fd, we should make sure to use the same page size, to -* this end we mmap the supplied fd. In this case, set MAP_NORESERVE to -* avoid allocating backing store memory. -* We do this unless we are using the system page size, in which case -* anonymous memory is OK. -*/ - flags =MAP_PRIVATE ; -if (fd ==-1 || guard_pagesize == qemu_real_host_page_size) { - guardfd =-1 ; - flags |=MAP_ANONYMOUS ; - }else { - guardfd = fd; - flags |=MAP_NORESERVE ; - } -#else - guardfd =-1 ; - flags =MAP_PRIVATE |MAP_ANONYMOUS ; -#endif - - guardptr =mmap (0 , total,PROT_NONE , flags, guardfd,0 ); - + guardptr =mmap_reserve (total, fd);if (guardptr ==MAP_FAILED ) {return MAP_FAILED ; }
Signature · 1 line
2.29.2
PATCH v2 4/9 util/mmap-alloc: Factor out activating of memory to mmap_activate()
In reply to David HildenbrandWe want to activate memory within a reserved memory region, to make it accessible. Let's factor that out.
--- util/mmap-alloc.c | 94 +++++++++++++++++++++++++---------------------- 1 file changed, 50 insertions(+), 44 deletions(-)
util/mmap-alloc.c +50 -44
@@ -114,6 +114,52 @@ static void *mmap_reserve(size_t size, int fd)return mmap (0 , size,PROT_NONE , flags, fd,0 ); } +/* +* Activate memory in a reserved region from the given fd (if any), to make +* it accessible. +*/ +static void *mmap_activate (void *ptr,size_t size,int fd,bool readonly, +bool shared,bool is_pmem,off_t map_offset) +{ +const int prot =PROT_READ | (readonly ?0 :PROT_WRITE ); +int map_sync_flags =0 ; +int flags =MAP_FIXED ; +void *activated_ptr; + + flags |= fd ==-1 ?MAP_ANONYMOUS :0 ; + flags |= shared ?MAP_SHARED :MAP_PRIVATE ; +if (shared && is_pmem) { + map_sync_flags =MAP_SYNC |MAP_SHARED_VALIDATE ; + } + + activated_ptr =mmap (ptr, size, prot, flags | map_sync_flags, fd, + map_offset); +if (activated_ptr ==MAP_FAILED && map_sync_flags) { +if (errno ==ENOTSUP ) { +char *proc_link =g_strdup_printf ("/proc/self/fd/%d" , fd); +char *file_name =g_malloc0 (PATH_MAX ); +int len =readlink (proc_link, file_name,PATH_MAX -1 ); + +if (len <0 ) { + len =0 ; + } + file_name[len] ='\0' ; +fprintf (stderr,"Warning: requesting persistence across crashes " +"for backend file %s failed. Proceeding without " +"persistence, data might become corrupted in case of host " +"crash.\n" , file_name); +g_free (proc_link); +g_free (file_name); + } +/* +* If mmap failed with MAP_SHARED_VALIDATE | MAP_SYNC, we will try +* again without these flags to handle backwards compatibility. +*/ + activated_ptr =mmap (ptr, size, prot, flags, fd, map_offset); + } +return activated_ptr; +} +static inline size_t mmap_guard_pagesize (int fd) {#if defined(__powerpc64__) && defined(__linux__)
@@ -133,13 +179,8 @@ void *qemu_ram_mmap(int fd, off_t map_offset) {const size_t guard_pagesize =mmap_guard_pagesize (fd); -int prot; -int flags; -int map_sync_flags =0 ; -size_t offset; -size_t total; -void *guardptr; -void *ptr; +size_t offset, total; +void *ptr, *guardptr;/* * Note: this always allocates at least one extra page of virtual address
@@ -156,45 +197,10 @@ void *qemu_ram_mmap(int fd,/* Always align to host page size */ assert (align >= guard_pagesize); - flags =MAP_FIXED ; - flags |= fd ==-1 ?MAP_ANONYMOUS :0 ; - flags |= shared ?MAP_SHARED :MAP_PRIVATE ; -if (shared && is_pmem) { - map_sync_flags =MAP_SYNC |MAP_SHARED_VALIDATE ; - } - offset =QEMU_ALIGN_UP ((uintptr_t )guardptr, align) - (uintptr_t )guardptr; - prot =PROT_READ | (readonly ?0 :PROT_WRITE ); - - ptr =mmap (guardptr + offset, size, prot, - flags | map_sync_flags, fd, map_offset); - -if (ptr ==MAP_FAILED && map_sync_flags) { -if (errno ==ENOTSUP ) { -char *proc_link, *file_name; -int len; - proc_link =g_strdup_printf ("/proc/self/fd/%d" , fd); - file_name =g_malloc0 (PATH_MAX ); - len =readlink (proc_link, file_name,PATH_MAX -1 ); -if (len <0 ) { - len =0 ; - } - file_name[len] ='\0' ; -fprintf (stderr,"Warning: requesting persistence across crashes " -"for backend file %s failed. Proceeding without " -"persistence, data might become corrupted in case of host " -"crash.\n" , file_name); -g_free (proc_link); -g_free (file_name); - } -/* -* if map failed with MAP_SHARED_VALIDATE | MAP_SYNC, -* we will remove these flags to handle compatibility. -*/ - ptr =mmap (guardptr + offset, size, prot, flags, fd, map_offset); - } - + ptr =mmap_activate (guardptr + offset, size, fd, readonly, shared, is_pmem, + map_offset);if (ptr ==MAP_FAILED ) {munmap (guardptr, total);return MAP_FAILED ;
Signature · 1 line
2.29.2
PATCH v2 5/9 softmmu/memory: Pass ram_flags into qemu_ram_alloc_from_fd()
In reply to David HildenbrandLet's pass in ram flags just like we do with qemu_ram_alloc_from_file(), to clean up and prepare for more flags.
Simplify the documentation of passed ram flags: Looking at our documentation of RAM_SHARED and RAM_PMEM is sufficient, no need to be repetitive.
--- backends/hostmem-memfd.c | 7 ++++--- hw/misc/ivshmem.c | 5 ++--- include/exec/memory.h | 9 +++------ include/exec/ram_addr.h | 6 +----- softmmu/memory.c | 7 +++---- 5 files changed, 13 insertions(+), 21 deletions(-)
backends/hostmem-memfd.c +4 -3
@@ -36,6 +36,7 @@ static void memfd_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) {HostMemoryBackendMemfd *m =MEMORY_BACKEND_MEMFD (backend); +uint32_t ram_flags;char *name;int fd;
@@ -53,9 +54,9 @@ memfd_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) } name =host_memory_backend_get_name (backend); -memory_region_init_ram_from_fd (&backend->mr,OBJECT (backend), - name, backend->size, - backend->share, fd,0 , errp); + ram_flags = backend->share ?RAM_SHARED :0 ; +memory_region_init_ram_from_fd (&backend->mr,OBJECT (backend), name, + backend->size, ram_flags, fd,0 , errp);g_free (name); }
hw/misc/ivshmem.c +2 -3
@@ -494,9 +494,8 @@ static void process_msg_shmem(IVShmemState *s, int fd, Error **errp) size = buf.st_size;/* mmap the region and map into the BAR2 */ -memory_region_init_ram_from_fd (&s->server_bar2,OBJECT (s), -"ivshmem.bar2" , size, true, fd,0 , - &local_err); +memory_region_init_ram_from_fd (&s->server_bar2,OBJECT (s),"ivshmem.bar2" , + size,RAM_SHARED , fd,0 , &local_err);if (local_err) {error_propagate (errp, local_err);return ;
include/exec/memory.h +3 -6
@@ -967,10 +967,7 @@ void memory_region_init_resizeable_ram(MemoryRegion *mr,* @size: size of the region. * @align: alignment of the region base address; if 0, the default alignment * (getpagesize()) will be used. -* @ram_flags: Memory region features: -* - RAM_SHARED: memory must be mmaped with the MAP_SHARED flag -* - RAM_PMEM: the memory is persistent memory -* Other bits are ignored now. +* @ram_flags: RamBlock flags. Supported flags: RAM_SHARED, RAM_PMEM. * @path: the path in which to allocate the RAM. * @readonly: true to open @path for reading, false for read/write. * @errp: pointer to Error*, to store an error if it happens.
@@ -996,7 +993,7 @@ void memory_region_init_ram_from_file(MemoryRegion *mr,* @owner: the object that tracks the region's reference count * @name: the name of the region. * @size: size of the region. -* @share: %true if memory must be mmaped with the MAP_SHARED flag +* @ram_flags: RamBlock flags. Supported flags: RAM_SHARED, RAM_PMEM. * @fd: the fd to mmap. * @offset: offset within the file referenced by fd * @errp: pointer to Error*, to store an error if it happens.
@@ -1008,7 +1005,7 @@ void memory_region_init_ram_from_fd(MemoryRegion *mr,struct Object *owner, const char *name, uint64_t size, - bool share, + uint32_t ram_flags, int fd, ram_addr_t offset, Error **errp);
include/exec/ram_addr.h +1 -5
@@ -104,11 +104,7 @@ long qemu_maxrampagesize(void);* Parameters: * @size: the size in bytes of the ram block * @mr: the memory region where the ram block is -* @ram_flags: specify the properties of the ram block, which can be one -* or bit-or of following values -* - RAM_SHARED: mmap the backing file or device with MAP_SHARED -* - RAM_PMEM: the backend @mem_path or @fd is persistent memory -* Other bits are ignored. +* @ram_flags: RamBlock flags. Supported flags: RAM_SHARED, RAM_PMEM. * @mem_path or @fd: specify the backing file or device * @readonly: true to open @path for reading, false for read/write. * @errp: pointer to Error*, to store an error if it happens
softmmu/memory.c +3 -4
@@ -1610,7 +1610,7 @@ void memory_region_init_ram_from_fd(MemoryRegion *mr,struct Object *owner, const char *name, uint64_t size, - bool share, + uint32_t ram_flags, int fd, ram_addr_t offset, Error **errp)
@@ -1620,9 +1620,8 @@ void memory_region_init_ram_from_fd(MemoryRegion *mr, mr->ram = true; mr->terminates = true; mr->destructor = memory_region_destructor_ram; - mr->ram_block =qemu_ram_alloc_from_fd (size, mr, - share ?RAM_SHARED :0 , - fd, offset, false, &err); + mr->ram_block =qemu_ram_alloc_from_fd (size, mr, ram_flags, fd, offset, + false, &err);if (err) { mr->size =int128_zero ();object_unparent (OBJECT (mr));
Signature · 1 line
2.29.2
PATCH v2 6/9 softmmu/memory: Pass ram_flags into memory_region_init_ram_shared_nomigrate()
In reply to David HildenbrandLet's forward ram_flags instead, renaming memory_region_init_ram_shared_nomigrate() into memory_region_init_ram_flags_nomigrate(). Forward flags to qemu_ram_alloc() and qemu_ram_alloc_internal().
--- backends/hostmem-ram.c | 6 +++-- hw/m68k/next-cube.c | 4 ++-- include/exec/memory.h | 24 +++++++++---------- include/exec/ram_addr.h | 2 +- .../memory-region-housekeeping.cocci | 8 +++---- softmmu/memory.c | 20 ++++++++-------- softmmu/physmem.c | 24 ++++++++----------- 7 files changed, 43 insertions(+), 45 deletions(-)
backends/hostmem-ram.c +4 -2
@@ -19,6 +19,7 @@static void ram_backend_memory_alloc (HostMemoryBackend *backend,Error **errp) { +uint32_t ram_flags;char *name;if (!backend->size) {
@@ -27,8 +28,9 @@ ram_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) } name =host_memory_backend_get_name (backend); -memory_region_init_ram_shared_nomigrate (&backend->mr,OBJECT (backend), name, - backend->size, backend->share, errp); + ram_flags = backend->share ?RAM_SHARED :0 ; +memory_region_init_ram_flags_nomigrate (&backend->mr,OBJECT (backend), name, + backend->size, ram_flags, errp);g_free (name); }
hw/m68k/next-cube.c +2 -2
@@ -986,8 +986,8 @@ static void next_cube_init(MachineState *machine)sysbus_mmio_map (SYS_BUS_DEVICE (pcdev),1 ,0x02100000 );/* BMAP memory */ -memory_region_init_ram_shared_nomigrate (bmapm1,NULL ,"next.bmapmem" ,64 , - true, &error_fatal); +memory_region_init_ram_flags_nomigrate (bmapm1,NULL ,"next.bmapmem" ,64 , +RAM_SHARED , &error_fatal);memory_region_add_subregion (sysmem,0x020c0000 , bmapm1);/* The Rev_2.5_v66.bin firmware accesses it at 0x820c0020, too */ memory_region_init_alias (bmapm2,NULL ,"next.bmapmem2" , bmapm1,0x0 ,64 );
include/exec/memory.h +12 -12
@@ -904,27 +904,27 @@ void memory_region_init_ram_nomigrate(MemoryRegion *mr, Error **errp);/** -* memory_region_init_ram_shared_nomigrate: Initialize RAM memory region. -* Accesses into the region will -* modify memory directly. +* memory_region_init_ram_flags_nomigrate: Initialize RAM memory region. +* Accesses into the region will +* modify memory directly. * * @mr: the #MemoryRegion to be initialized. * @owner: the object that tracks the region's reference count * @name: Region name, becomes part of RAMBlock name used in migration stream * must be unique within any device * @size: size of the region. -* @share: allow remapping RAM to different addresses +* @ram_flags: RamBlock flags. Supported flags: RAM_SHARED. * @errp: pointer to Error*, to store an error if it happens. * -* Note that this function is similar to memory_region_init_ram_nomigrate. -* The only difference is part of the RAM region can be remapped. +* Note that this function does not do anything to cause the data in the +* RAM memory region to be migrated; that is the responsibility of the caller. */ -void memory_region_init_ram_shared_nomigrate (MemoryRegion *mr, -struct Object *owner, -const char *name, -uint64_t size, -bool share, -Error **errp); +void memory_region_init_ram_flags_nomigrate (MemoryRegion *mr, +struct Object *owner, +const char *name, +uint64_t size, +uint32_t ram_flags, +Error **errp);/** * memory_region_init_resizeable_ram: Initialize memory region with resizeable
include/exec/ram_addr.h +1 -1
@@ -122,7 +122,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,RAMBlock *qemu_ram_alloc_from_ptr (ram_addr_t size,void *host,MemoryRegion *mr,Error **errp); -RAMBlock *qemu_ram_alloc (ram_addr_t size,bool share,MemoryRegion *mr, +RAMBlock *qemu_ram_alloc (ram_addr_t size,uint32_t ram_flags,MemoryRegion *mr,Error **errp); RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t max_size, void (*resized)(const char*,
scripts/coccinelle/memory-region-housekeeping.cocci +4 -4
@@ -127,8 +127,8 @@ static void device_fn(DeviceState *dev, ...) - memory_region_init_rom(E1, NULL, E2, E3, E4); + memory_region_init_rom(E1, obj, E2, E3, E4); | -- memory_region_init_ram_shared_nomigrate(E1, NULL, E2, E3, E4, E5); -+ memory_region_init_ram_shared_nomigrate(E1, obj, E2, E3, E4, E5); +- memory_region_init_ram_flags_nomigrate(E1, NULL, E2, E3, E4, E5); ++ memory_region_init_ram_flags_nomigrate(E1, obj, E2, E3, E4, E5); ) ...+> }
@@ -152,8 +152,8 @@ static void device_fn(DeviceState *dev, ...) - memory_region_init_rom(E1, NULL, E2, E3, E4); + memory_region_init_rom(E1, OBJECT(dev), E2, E3, E4); | -- memory_region_init_ram_shared_nomigrate(E1, NULL, E2, E3, E4, E5); -+ memory_region_init_ram_shared_nomigrate(E1, OBJECT(dev), E2, E3, E4, E5); +- memory_region_init_ram_flags_nomigrate(E1, NULL, E2, E3, E4, E5); ++ memory_region_init_ram_flags_nomigrate(E1, OBJECT(dev), E2, E3, E4, E5); ) ...+> }
softmmu/memory.c +10 -10
@@ -1532,22 +1532,22 @@ void memory_region_init_ram_nomigrate(MemoryRegion *mr,uint64_t size, Error **errp) { -memory_region_init_ram_shared_nomigrate (mr ,owner ,name ,size ,false ,errp ); + memory_region_init_ram_flags_nomigrate(mr, owner, name, size, 0, errp); } -void memory_region_init_ram_shared_nomigrate (MemoryRegion *mr, -Object *owner, -const char *name, -uint64_t size, -bool share, -Error **errp) +void memory_region_init_ram_flags_nomigrate (MemoryRegion *mr, +Object *owner, +const char *name, +uint64_t size, +uint32_t ram_flags, +Error **errp) {Error *err =NULL ;memory_region_init (mr, owner, name, size); mr->ram = true; mr->terminates = true; mr->destructor = memory_region_destructor_ram; - mr->ram_block =qemu_ram_alloc (size, share, mr, &err); + mr->ram_block =qemu_ram_alloc (size, ram_flags, mr, &err);if (err) { mr->size =int128_zero ();object_unparent (OBJECT (mr));
@@ -1683,7 +1683,7 @@ void memory_region_init_rom_nomigrate(MemoryRegion *mr,uint64_t size, Error **errp) { -memory_region_init_ram_shared_nomigrate (mr ,owner ,name ,size ,false ,errp ); + memory_region_init_ram_flags_nomigrate(mr, owner, name, size, 0, errp); mr->readonly = true; }
@@ -1703,7 +1703,7 @@ void memory_region_init_rom_device_nomigrate(MemoryRegion *mr, mr->terminates = true; mr->rom_device = true; mr->destructor = memory_region_destructor_ram; - mr->ram_block =qemu_ram_alloc (size, false, mr, &err); + mr->ram_block =qemu_ram_alloc (size,0 , mr, &err);if (err) { mr->size =int128_zero ();object_unparent (OBJECT (mr));
softmmu/physmem.c +10 -14
@@ -2108,12 +2108,14 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,void (*resized)(const char *,uint64_t length,void *host), - void *host, bool resizeable, bool share, + void *host, uint32_t ram_flags, MemoryRegion *mr, Error **errp) { RAMBlock *new_block;Error *local_err =NULL ; +assert ((ram_flags & ~(RAM_SHARED |RAM_RESIZEABLE )) ==0 ); + size =HOST_PAGE_ALIGN (size); max_size =HOST_PAGE_ALIGN (max_size); new_block =g_malloc0 (sizeof (*new_block));
@@ -2125,15 +2127,10 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size, new_block->fd =-1 ; new_block->page_size = qemu_real_host_page_size; new_block->host = host; + new_block->flags = ram_flags;if (host) { new_block->flags |=RAM_PREALLOC ; } -if (share) { - new_block->flags |=RAM_SHARED ; - } -if (resizeable) { - new_block->flags |=RAM_RESIZEABLE ; - }ram_block_add (new_block, &local_err);if (local_err) {g_free (new_block);
@@ -2146,15 +2143,14 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,RAMBlock *qemu_ram_alloc_from_ptr (ram_addr_t size,void *host,MemoryRegion *mr,Error **errp) { -return qemu_ram_alloc_internal (size, size,NULL , host, false, - false, mr, errp); +return qemu_ram_alloc_internal (size, size,NULL , host,0 , mr, errp); } -RAMBlock *qemu_ram_alloc (ram_addr_t size,bool share, +RAMBlock *qemu_ram_alloc (ram_addr_t size,uint32_t ram_flags,MemoryRegion *mr,Error **errp) { -return qemu_ram_alloc_internal (size, size,NULL ,NULL , false, - share, mr, errp); +assert ((ram_flags & ~RAM_SHARED ) ==0 ); +return qemu_ram_alloc_internal (size, size,NULL ,NULL , ram_flags, mr, errp); } RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t maxsz,
@@ -2163,8 +2159,8 @@ RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t maxsz, void *host), MemoryRegion *mr, Error **errp) { - return qemu_ram_alloc_internal(size, maxsz, resized, NULL, true, -false ,mr ,errp ); + return qemu_ram_alloc_internal(size, maxsz, resized, NULL, +RAM_RESIZEABLE ,mr ,errp ); }static void reclaim_ramblock (RAMBlock *block)
Signature · 1 line
2.29.2
PATCH v2 7/9 memory: introduce RAM_NORESERVE and wire it up in qemu_ram_mmap()
In reply to David HildenbrandLet's introduce RAM_NORESERVE, allowing mmap'ing with MAP_NORESERVE. The new flag has the following semantics:
RAM is mmap-ed with MAP_NORESERVE. When set, reserving swap space (or huge pages on Linux) is skipped: will bail out if not supported. When not set, the OS might reserve swap space (or huge pages on Linux), depending on OS support.
Allow passing it into:
- memory_region_init_ram_nomigrate()
- memory_region_init_resizeable_ram()
- memory_region_init_ram_from_file()
... and teach qemu_ram_mmap() and qemu_anon_ram_alloc() about the flag. Bail out if the flag is not supported, which is the case right now for both, POSIX and win32. We will add the POSIX mmap implementation next and allow specifying RAM_NORESERVE via memory backends.
The target use case is virtio-mem, which dynamically exposes memory inside a large, sparse memory area to the VM.
--- include/exec/cpu-common.h | 1 + include/exec/memory.h | 16 +++++++++++++--- include/exec/ram_addr.h | 3 ++- include/qemu/mmap-alloc.h | 2 ++ include/qemu/osdep.h | 3 ++- migration/ram.c | 3 +-- softmmu/physmem.c | 16 +++++++++++----- util/mmap-alloc.c | 7 +++++++ util/oslib-posix.c | 6 ++++-- util/oslib-win32.c | 13 ++++++++++++- 10 files changed, 55 insertions(+), 15 deletions(-)
include/exec/cpu-common.h +1 -0
@@ -58,6 +58,7 @@ void *qemu_ram_get_host_addr(RAMBlock *rb);ram_addr_t qemu_ram_get_offset (RAMBlock *rb);ram_addr_t qemu_ram_get_used_length (RAMBlock *rb);bool qemu_ram_is_shared (RAMBlock *rb); +bool qemu_ram_is_noreserve (RAMBlock *rb);bool qemu_ram_is_uf_zeroable (RAMBlock *rb);void qemu_ram_set_uf_zeroable (RAMBlock *rb);bool qemu_ram_is_migratable (RAMBlock *rb);
include/exec/memory.h +13 -3
@@ -155,6 +155,14 @@ typedef struct IOMMUTLBEvent {*/ #define RAM_UF_WRITEPROTECT (1 << 6) +/* +* RAM is mmap-ed with MAP_NORESERVE. When set, reserving swap space (or huge +* pages Linux) is skipped: will bail out if not supported. When not set, the +* OS might reserve swap space (or huge pages on Linux), depending on OS +* support. +*/ +#define RAM_NORESERVE (1 << 7) + static inline void iommu_notifier_init(IOMMUNotifier *n, IOMMUNotify fn, IOMMUNotifierFlag flags, hwaddr start, hwaddr end,
@@ -913,7 +921,7 @@ void memory_region_init_ram_nomigrate(MemoryRegion *mr,* @name: Region name, becomes part of RAMBlock name used in migration stream * must be unique within any device * @size: size of the region. -* @ram_flags: RamBlock flags. Supported flags: RAM_SHARED. +* @ram_flags: RamBlock flags. Supported flags: RAM_SHARED, RAM_NORESERVE. * @errp: pointer to Error*, to store an error if it happens. * * Note that this function does not do anything to cause the data in the
@@ -967,7 +975,8 @@ void memory_region_init_resizeable_ram(MemoryRegion *mr,* @size: size of the region. * @align: alignment of the region base address; if 0, the default alignment * (getpagesize()) will be used. -* @ram_flags: RamBlock flags. Supported flags: RAM_SHARED, RAM_PMEM. +* @ram_flags: RamBlock flags. Supported flags: RAM_SHARED, RAM_PMEM, +* RAM_NORESERVE, * @path: the path in which to allocate the RAM. * @readonly: true to open @path for reading, false for read/write. * @errp: pointer to Error*, to store an error if it happens.
@@ -993,7 +1002,8 @@ void memory_region_init_ram_from_file(MemoryRegion *mr,* @owner: the object that tracks the region's reference count * @name: the name of the region. * @size: size of the region. -* @ram_flags: RamBlock flags. Supported flags: RAM_SHARED, RAM_PMEM. +* @ram_flags: RamBlock flags. Supported flags: RAM_SHARED, RAM_PMEM, +* RAM_NORESERVE. * @fd: the fd to mmap. * @offset: offset within the file referenced by fd * @errp: pointer to Error*, to store an error if it happens.
include/exec/ram_addr.h +2 -1
@@ -104,7 +104,8 @@ long qemu_maxrampagesize(void);* Parameters: * @size: the size in bytes of the ram block * @mr: the memory region where the ram block is -* @ram_flags: RamBlock flags. Supported flags: RAM_SHARED, RAM_PMEM. +* @ram_flags: RamBlock flags. Supported flags: RAM_SHARED, RAM_PMEM, +* RAM_NORESERVE. * @mem_path or @fd: specify the backing file or device * @readonly: true to open @path for reading, false for read/write. * @errp: pointer to Error*, to store an error if it happens
include/qemu/mmap-alloc.h +2 -0
@@ -17,6 +17,7 @@ size_t qemu_mempath_getpagesize(const char *mem_path);* @readonly: true for a read-only mapping, false for read/write. * @shared: map has RAM_SHARED flag. * @is_pmem: map has RAM_PMEM flag. +* @noreserve: map has RAM_NORESERVE flag. * @map_offset: map starts at offset of map_offset from the start of fd * * Return:
@@ -29,6 +30,7 @@ void *qemu_ram_mmap(int fd,bool readonly, bool shared, bool is_pmem, + bool noreserve, off_t map_offset);void qemu_ram_munmap (int fd,void *ptr,size_t size);
include/qemu/osdep.h +2 -1
@@ -343,7 +343,8 @@ extern int daemon(int, int);int qemu_daemon (int nochdir,int noclose);void *qemu_try_memalign (size_t alignment,size_t size);void *qemu_memalign (size_t alignment,size_t size); -void *qemu_anon_ram_alloc (size_t size,uint64_t *align,bool shared); +void *qemu_anon_ram_alloc (size_t size,uint64_t *align,bool shared, +bool noreserve);void qemu_vfree (void *ptr);void qemu_anon_ram_free (void *ptr,size_t size);
migration/ram.c +1 -2
@@ -3322,8 +3322,7 @@ int colo_init_ram_cache(void) WITH_RCU_READ_LOCK_GUARD() {RAMBLOCK_FOREACH_NOT_IGNORED (block) { block->colo_cache =qemu_anon_ram_alloc (block->used_length, -NULL , - false); +NULL , false, false);if (!block->colo_cache) { error_report("%s: Can't alloc memory for COLO cache of block %s," "size 0x" RAM_ADDR_FMT, __func__, block->idstr,
softmmu/physmem.c +11 -5
@@ -1582,7 +1582,7 @@ static void *file_ram_alloc(RAMBlock *block, area =qemu_ram_mmap (fd, memory, block->mr->align, readonly, block->flags &RAM_SHARED , block->flags &RAM_PMEM , - offset); + block->flags &RAM_NORESERVE , offset);if (area ==MAP_FAILED ) {error_setg_errno (errp, errno,"unable to map backing store for guest RAM" );
@@ -1702,6 +1702,11 @@ bool qemu_ram_is_shared(RAMBlock *rb)return rb->flags &RAM_SHARED ; } +bool qemu_ram_is_noreserve (RAMBlock *rb) +{ +return rb->flags &RAM_NORESERVE ; +} +/* Note: Only set at the start of postcopy */ bool qemu_ram_is_uf_zeroable (RAMBlock *rb) {
@@ -1930,6 +1935,7 @@ static void dirty_memory_extend(ram_addr_t old_ram_size,static void ram_block_add (RAMBlock *new_block,Error **errp) {const bool shared =qemu_ram_is_shared (new_block); +const bool noreserve =qemu_ram_is_noreserve (new_block);RAMBlock *block;RAMBlock *last_block =NULL ;ram_addr_t old_ram_size, new_ram_size;
@@ -1952,7 +1958,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) }else { new_block->host =qemu_anon_ram_alloc (new_block->max_length, &new_block->mr->align, - shared); + shared, noreserve);if (!new_block->host) { error_setg_errno(errp, errno, "cannot set up guest memory '%s'",
@@ -2023,7 +2029,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,int64_t file_size, file_align;/* Just support these ram flags by now. */ -assert ((ram_flags & ~(RAM_SHARED |RAM_PMEM )) ==0 ); +assert ((ram_flags & ~(RAM_SHARED |RAM_PMEM |RAM_NORESERVE )) ==0 );if (xen_enabled ()) {error_setg (errp,"-mem-path not supported with Xen" );
@@ -2114,7 +2120,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,RAMBlock *new_block;Error *local_err =NULL ; -assert ((ram_flags & ~(RAM_SHARED |RAM_RESIZEABLE )) ==0 ); +assert ((ram_flags & ~(RAM_SHARED |RAM_RESIZEABLE |RAM_NORESERVE )) ==0 ); size =HOST_PAGE_ALIGN (size); max_size =HOST_PAGE_ALIGN (max_size);
@@ -2149,7 +2155,7 @@ RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,RAMBlock *qemu_ram_alloc (ram_addr_t size,uint32_t ram_flags,MemoryRegion *mr,Error **errp) { -assert ((ram_flags & ~RAM_SHARED ) ==0 ); +assert ((ram_flags & ~(RAM_SHARED |RAM_NORESERVE )) ==0 );return qemu_ram_alloc_internal (size, size,NULL ,NULL , ram_flags, mr, errp); }
util/mmap-alloc.c +7 -0
@@ -20,6 +20,7 @@#include "qemu/osdep.h" #include "qemu/mmap-alloc.h" #include "qemu/host-utils.h" +#include "qemu/error-report.h" #define HUGETLBFS_MAGIC 0x958458f6
@@ -176,12 +177,18 @@ void *qemu_ram_mmap(int fd,bool readonly, bool shared, bool is_pmem, + bool noreserve, off_t map_offset) { const size_t guard_pagesize = mmap_guard_pagesize(fd);size_t offset, total;void *ptr, *guardptr; +if (noreserve) { +error_report ("Skipping reservation of swap space is not supported" ); +return MAP_FAILED ; + } +/* * Note: this always allocates at least one extra page of virtual address * space, even if size is already aligned.
util/oslib-posix.c +4 -2
@@ -227,10 +227,12 @@ void *qemu_memalign(size_t alignment, size_t size) }/* alloc shared memory pages */ -void *qemu_anon_ram_alloc (size_t size,uint64_t *alignment,bool shared) +void *qemu_anon_ram_alloc (size_t size,uint64_t *alignment,bool shared, +bool noreserve) {size_t align =QEMU_VMALLOC_ALIGN ; -void *ptr =qemu_ram_mmap (-1 , size, align, false, shared, false,0 ); +void *ptr =qemu_ram_mmap (-1 , size, align, false, shared, false, noreserve, +0 );if (ptr ==MAP_FAILED ) {return NULL ;
util/oslib-win32.c +12 -1
@@ -39,6 +39,7 @@#include "trace.h" #include "qemu/sockets.h" #include "qemu/cutils.h" +#include "qemu/error-report.h" #include <malloc.h> /* this must come after including "trace.h" */
@@ -77,10 +78,20 @@ static int get_allocation_granularity(void)return system_info.dwAllocationGranularity; } -void *qemu_anon_ram_alloc (size_t size,uint64_t *align,bool shared) +void *qemu_anon_ram_alloc (size_t size,uint64_t *align,bool shared, +bool noreserve) {void *ptr; +if (noreserve) { +/* +* We need a MEM_COMMIT before accessing any memory in a MEM_RESERVE +* area; we cannot easily mimic POSIX MAP_NORESERVE semantics. +*/ +error_report ("Skipping reservation of swap space is not supported." ); +return NULL ; + } + ptr =VirtualAlloc (NULL , size,MEM_COMMIT ,PAGE_READWRITE );trace_qemu_anon_ram_alloc (size, ptr);
Signature · 1 line
2.29.2
Re: [PATCH v2 7/9] memory: introduce RAM_NORESERVE and wire it up in qemu_ram_mmap()
Quoted 21 lines · David Hildenbrand
Let's introduce
RAM_NORESERVE, allowing mmap'ing withMAP_NORESERVE. The new flag has the following semantics:RAM is mmap-ed with
MAP_NORESERVE. When set, reserving swap space (or huge pages on Linux) is skipped: will bail out if not supported. When not set, the OS might reserve swap space (or huge pages on Linux), depending on OS support.Allow passing it into:
-memory_region_init_ram_nomigrate()
-memory_region_init_resizeable_ram()
-memory_region_init_ram_from_file()... and teach
qemu_ram_mmap()andqemu_anon_ram_alloc()about the flag. Bail out if the flag is not supported, which is the case right now for both, POSIX and win32. We will add the POSIX mmap implementation next and allow specifyingRAM_NORESERVEvia memory backends.The target use case is virtio-mem, which dynamically exposes memory inside a large, sparse memory area to the VM.
Quoted 9 lines
Cc Juan Quintela <[email protected]>Cc Halil Pasic <[email protected]>Cc Cornelia Huck <[email protected]>Cc Christian Borntraeger <[email protected]>Cc Thomas Huth <[email protected]>Cc Stefan Weil <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Signature · 1 line
Peter Xu
PATCH v2 8/9 util/mmap-alloc: Support RAM_NORESERVE via MAP_NORESERVE
In reply to David HildenbrandLet's support RAM_NORESERVE via MAP_NORESERVE. At least on Linux, the flag has no effect on shared mappings - except for hugetlbfs.
Linux man page:
"MAP_NORESERVE: Do not reserve swap space for this mapping. When swap space is reserved, one has the guarantee that it is possible to modify the mapping. When swap space is not reserved one might get SIGSEGV upon a write if no physical memory is available. See also the discussion of the file /proc/sys/vm/overcommit_memory in proc(5). In kernels before 2.6, this flag had effect only for private writable mappings."
Note that the "guarantee" part is wrong with memory overcommit in Linux.
Also, in Linux hugetlbfs is treated differently - we configure reservation of huge pages from the pool, not reservation of swap space (huge pages cannot be swapped).
The rough behavior is [1]:
a) !Hugetlbfs:
1) Without MAP_NORESERVE *or* with memory overcommit under Linux disabled ("/proc/sys/vm/overcommit_memory == 2"), the following accounting/reservation happens:
For a file backed map
SHARED or READ-only - 0 cost (the file is the map not swap) PRIVATE WRITABLE - size of mapping per instance
For an anonymous or /dev/zero map
SHARED - size of mapping
PRIVATE READ-only - 0 cost (but of little use) PRIVATE WRITABLE - size of mapping per instance
2) With MAP_NORESERVE, no accounting/reservation happens.
b) Hugetlbfs:
1) Without MAP_NORESERVE, huge pages are reserved.
2) With MAP_NORESERVE, no huge pages are reserved.
Note: With "/proc/sys/vm/overcommit_memory == 0", we were already able to configure it for !hugetlbfs globally; this toggle now allows configuring it more fine-grained, not for the whole system.
The target use case is virtio-mem, which dynamically exposes memory inside a large, sparse memory area to the VM.
[1] https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
--- softmmu/physmem.c | 1 + util/mmap-alloc.c | 74 ++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 71 insertions(+), 4 deletions(-)
softmmu/physmem.c +1 -0
@@ -2224,6 +2224,7 @@ void qemu_ram_remap(ram_addr_t addr, ram_addr_t length)abort (); }else { flags =MAP_FIXED ; + flags |= (block->flags &RAM_NORESERVE ) ?MAP_NORESERVE :0 ;if (block->fd >=0 ) { flags |= (block->flags &RAM_SHARED ?MAP_SHARED :MAP_PRIVATE );
util/mmap-alloc.c +70 -4
110 lines not on this page · raw message
Signature · 1 line
2.29.2
Re: [PATCH v2 8/9] util/mmap-alloc: Support RAM_NORESERVE via MAP_NORESERVE
Quoted 3 lines · David Hildenbrand
util/mmap-alloc.c +3
-0+#define OVERCOMMIT_MEMORY_PATH "/proc/sys/vm/overcommit_memory" +static bool map_noreserve_effective (int fd,bool readonly,bool shared) +{
[...]
util/mmap-alloc.c void *qemu_ram_mmap(int fd, · line 254
Quoted 7 lines
util/mmap-alloc.c +1
-2@@ -184,8 +251,7 @@ void *qemu_ram_mmap(int fd,size_t offset, total;void *ptr, *guardptr; -if (noreserve) { -error_report ("Skipping reservation of swap space is not supported" ); +if (noreserve && !map_noreserve_effective (fd, shared, readonly)) {
Need to switch "shared" & "readonly"?
Quoted 2 lines
util/mmap-alloc.c +0
-0return MAP_FAILED ; }
Signature · 1 line
Peter Xu
Re: [PATCH v2 8/9] util/mmap-alloc: Support RAM_NORESERVE via MAP_NORESERVE
Quoted 11 lines · Peter Xu
Quoted 3 lines · David Hildenbrand
util/mmap-alloc.c +3
-0+#define OVERCOMMIT_MEMORY_PATH "/proc/sys/vm/overcommit_memory" +static bool map_noreserve_effective (int fd,bool readonly,bool shared) +{[...]
Quoted 4 lines
util/mmap-alloc.c +0
-0@@ -184,8 +251,7 @@ void *qemu_ram_mmap(int fd,size_t offset, total;void *ptr, *guardptr;
Quoted 5 lines
util/mmap-alloc.c void *qemu_ram_mmap(int fd, · line 254
Quoted 3 lines
util/mmap-alloc.c +1
-2-if (noreserve) { -error_report ("Skipping reservation of swap space is not supported" ); +if (noreserve && !map_noreserve_effective (fd, shared, readonly)) {Need to switch "shared" & "readonly"?
Indeed, interestingly it has the same effect (as we don't have anonymous read-only memory in QEMU :) )
(wouldn't have happened with flags ... hmm)
Thanks!
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v2 8/9] util/mmap-alloc: Support RAM_NORESERVE via MAP_NORESERVE
Quoted 17 lines · David Hildenbrand
Quoted 7 lines · Peter Xu
Quoted 3 lines · David Hildenbrand
util/mmap-alloc.c +3
-0+#define OVERCOMMIT_MEMORY_PATH "/proc/sys/vm/overcommit_memory" +static bool map_noreserve_effective (int fd,bool readonly,bool shared) +{[...]
Quoted 8 lines
util/mmap-alloc.c void *qemu_ram_mmap(int fd, · line 253
Quoted 6 lines
util/mmap-alloc.c +1
-2@@ -184,8 +251,7 @@ void *qemu_ram_mmap(int fd,size_t offset, total;void *ptr, *guardptr; -if (noreserve) { -error_report ("Skipping reservation of swap space is not supported" ); +if (noreserve && !map_noreserve_effective (fd, shared, readonly)) {Need to switch "shared" & "readonly"?
Quoted 2 lines
Indeed, interestingly it has the same effect (as we don't have anonymous read-only memory in QEMU :) )
But note there is still a "g_assert(!shared || fd >= 0);" inside.. :)
Quoted 2 lines
(wouldn't have happened with flags ... hmm)
Right.
Signature · 1 line
Peter Xu
Re: [PATCH v2 8/9] util/mmap-alloc: Support RAM_NORESERVE via MAP_NORESERVE
Quoted 21 lines · Peter Xu
Quoted 17 lines · David Hildenbrand
Quoted 7 lines · Peter Xu
Quoted 3 lines · David Hildenbrand
util/mmap-alloc.c +3
-0+#define OVERCOMMIT_MEMORY_PATH "/proc/sys/vm/overcommit_memory" +static bool map_noreserve_effective (int fd,bool readonly,bool shared) +{[...]
Quoted 8 lines
util/mmap-alloc.c void *qemu_ram_mmap(int fd, · line 253
Quoted 6 lines
util/mmap-alloc.c +1
-2@@ -184,8 +251,7 @@ void *qemu_ram_mmap(int fd,size_t offset, total;void *ptr, *guardptr; -if (noreserve) { -error_report ("Skipping reservation of swap space is not supported" ); +if (noreserve && !map_noreserve_effective (fd, shared, readonly)) {Need to switch "shared" & "readonly"?
Quoted 2 lines
Indeed, interestingly it has the same effect (as we don't have anonymous read-only memory in QEMU :) )
Quoted 7 lines
But note there is still a "
g_assert(!shared || fd >= 0);" inside.. :)Quoted 2 lines
(wouldn't have happened with flags ... hmm)
Right.
I'll probably go with
/* Map PROT_READ instead of PROT_READ|PROT_WRITE. */
#define QEMU_RAM_MMAP_READONLY (1 << 0)
/* Map MAP_SHARED instead of MAP_PRIVATE. */
#define QEMU_RAM_MMAP_SHARED (1 << 1)
/* Map MAP_SYNC|MAP_SHARED_VALIDATE if possible, fallback and warn otherwise. */
#define QEMU_RAM_MMAP_PMEM (1 << 2)
for qemu_ram_mmap(). qemu_anon_ram_alloc() will still have bools, but there it will at least be only two.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v2 8/9] util/mmap-alloc: Support RAM_NORESERVE via MAP_NORESERVE
In reply to Peter XuQuoted 18 lines · Peter Xu
Quoted 17 lines · David Hildenbrand
Quoted 15 lines · Peter Xu
Quoted 3 lines · David Hildenbrand
util/mmap-alloc.c +3
-0+#define OVERCOMMIT_MEMORY_PATH "/proc/sys/vm/overcommit_memory" +static bool map_noreserve_effective (int fd,bool readonly,bool shared) +{[...]
util/mmap-alloc.c void *qemu_ram_mmap(int fd, · line 253
Quoted 6 lines
util/mmap-alloc.c +1
-2@@ -184,8 +251,7 @@ void *qemu_ram_mmap(int fd,size_t offset, total;void *ptr, *guardptr; -if (noreserve) { -error_report ("Skipping reservation of swap space is not supported" ); +if (noreserve && !map_noreserve_effective (fd, shared, readonly)) {Need to switch "shared" & "readonly"?
Quoted 4 lines
Quoted 2 lines
Indeed, interestingly it has the same effect (as we don't have anonymous read-only memory in QEMU :) )
But note there is still a "
g_assert(!shared || fd >= 0);" inside.. :)
Aaaaaand, I just figured that we actually can create shared anonymous memory in QEMU, simply via
-object memory-backend-ram,share=on
Introduced in 06329ccecfa0 ("mem: add share parameter to memory-backend-ram"). That's also where we introduced the "shared" flag for qemu_anon_ram_alloc().
That commit mentions a use case for "RDMA devices in order to remap non-contiguous QEMU virtual addresses to a contiguous virtual address range.". I fail to understand why that requires sharing RAM with child processes.
Especially:
a) qemu_ram_is_shared() returned false before patch #1. RAM_SHARED is never set.
b) qemu_ram_remap() does not work as expected?
c) ram_discard_range() is broken with shared anonymous memory. Instead of MADV_DONTNEED we need MADV_REMOVE.
This looks like a partially broken feature and I wonder if there is an actual user.
@Marcel, can you clarify if there is an actual use case for shared anonymous memory in QEMU? I.e., if the original use case that required that change is valid? (and why it wasn't able to just use proper shmem)
Shared anonymous memory is weird.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v2 8/9] util/mmap-alloc: Support RAM_NORESERVE via MAP_NORESERVE
Hi David,
Quoted 51 lines · David Hildenbrand
Quoted 5 lines · Peter Xu
Quoted 4 lines · David Hildenbrand
Quoted 3 lines · Peter Xu
Review
Quoted 2 lines · David Hildenbrand
(quoted diff) +2
-0+#define OVERCOMMIT_MEMORY_PATH "/proc/sys/vm/overcommit_memory" +static bool map_noreserve_effective(int fd, bool readonly, boolshared)
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
+{
[...]
util/mmap-alloc.c void *qemu_ram_mmap(int fd, · line 187
Quoted 5 lines
util/mmap-alloc.c +0
-2@@ -184,8 +251,7 @@ void *qemu_ram_mmap(int fd,size_t offset, total;void *ptr, *guardptr; -if (noreserve) { - error_report("Skipping reservation of swap space is notsupported");
Quoted 1 line
Quoted 1 line
Quoted 1 line
Review void *qemu_ram_mmap(int fd, · line 253
Quoted 1 line
(quoted diff) +1
-0+ if (noreserve && !map_noreserve_effective(fd, shared, readonly)){
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Need to switch "shared" & "readonly"?
Indeed, interestingly it has the same effect (as we don't have anonymous read-only memory in QEMU :) )
But note there is still a "
g_assert(!shared || fd >= 0);" inside.. :)Aaaaaand, I just figured that we actually can create shared anonymous memory in QEMU, simply via
-object memory-backend-ram,share=on
Introduced in
06329ccecfa0("mem: add share parameter to memory-backend-ram"). That's also where we introduced the "shared" flag forqemu_anon_ram_alloc().That commit mentions a use case for "RDMA devices in order to remap non-contiguous QEMU virtual addresses to a contiguous virtual address range.". I fail to understand why that requires sharing RAM with child processes.
Especially:
a)
qemu_ram_is_shared()returned false before patch #1.RAM_SHAREDis never set.b)
qemu_ram_remap()does not work as expected?c)
ram_discard_range()is broken with shared anonymous memory. Instead ofMADV_DONTNEEDwe needMADV_REMOVE.
Quoted 8 lines
This looks like a partially broken feature and I wonder if there is an actual user.
@Marcel, can you clarify if there is an actual use case for shared anonymous memory in QEMU? I.e., if the original use case that required that change is valid? (and why it wasn't able to just use proper shmem)
As you correctly stated, the PVRDMA device requires remapping of non-contiguous QEMU
virtual addresses to a contiguous virtual address range.
In order to do so it calls
mremap (... , MREMAP_MAYMOVE | MREMAP_FIXED, ...)
The above call succeeds only if the memory is marked as "shared".
Quoted 2 lines
Shared anonymous memory is weird.
In this case it is not about sharing the memory between different processes, but about being
able to remap it.
Thanks,
Marcel
Quoted 7 lines
--
Thanks,David / dhildenb
Re: [PATCH v2 8/9] util/mmap-alloc: Support RAM_NORESERVE via MAP_NORESERVE
Quoted 64 lines · Marcel Apfelbaum
Hi David,
On Sun, Mar 7, 2021 at 3:18 PM David Hildenbrand <[email protected] <mailto:[email protected]>> wrote:
On 05.03.21 16:51, Peter Xu wrote:
> On Fri, Mar 05, 2021 at 04:44:36PM +0100, David Hildenbrand wrote:
>> On 05.03.21 16:42, Peter Xu wrote:
>>> On Fri, Mar 05, 2021 at 11:16:33AM +0100, David Hildenbrand wrote:
>>>> +#defineOVERCOMMIT_MEMORY_PATH"/proc/sys/vm/overcommit_memory" >>>> +static bool map_noreserve_effective(int fd, bool readonly,
bool shared)
>>>> +{
>>>
>>> [...]
>>>
>>>> @@ -184,8 +251,7 @@ void *qemu_ram_mmap(int fd,
>>>>size_toffset, total;
>>>> void *ptr, *guardptr;
>>>> - if (noreserve) {
>>>> - error_report("Skipping reservation of swap space is
not supported");
>>>> + if (noreserve && !map_noreserve_effective(fd, shared,
readonly)) {
>>>
>>> Need to switch "shared" & "readonly"?
>>
>> Indeed, interestingly it has the same effect (as we don't have
anonymous
>> read-only memory in QEMU :) )
>
> But note there is still a "g_assert(!shared || fd >= 0);" inside.. :)Aaaaaand, I just figured that we actually can create shared anonymous memory in QEMU, simply via
-object memory-backend-ram,share=on
Introduced in
06329ccecfa0("mem: add share parameter to memory-backend-ram"). That's also where we introduced the "shared" flag forqemu_anon_ram_alloc().That commit mentions a use case for "RDMA devices in order to remap non-contiguous QEMU virtual addresses to a contiguous virtual address range.". I fail to understand why that requires sharing RAM with child processes.
Especially:
a)
qemu_ram_is_shared()returned false before patch #1.RAM_SHAREDis never set.b)
qemu_ram_remap()does not work as expected?c)
ram_discard_range()is broken with shared anonymous memory. Instead ofMADV_DONTNEEDwe needMADV_REMOVE.This looks like a partially broken feature and I wonder if there is an actual user.
@Marcel, can you clarify if there is an actual use case for shared anonymous memory in QEMU? I.e., if the original use case that required that change is valid? (and why it wasn't able to just use proper shmem)
Quoted 5 lines
As you correctly stated, the PVRDMA device requires remapping of non-contiguous QEMU virtual addresses to a contiguous virtual address range.
In order to do so it calls
mremap (... ,MREMAP_MAYMOVE|MREMAP_FIXED, ...)
Thanks - I was missing who remaps and how (for a second I thought in another forked process).
docs/pvrdma.txt seems to describe the situation. Having to use anonymous shared memory is a bit unfortunate.
I yet haven't figured out how it is valid to remap parts of RAMBlocks to other locations via MREMAP_MAYMOVE. This sounds to me like we are punching holes into RAMBlocks - that can't be right.
Or maybe we are just shuffling around pages within a RAMBlock such that we don't actually punch holes?
Or does that happen when the source VM is stopped and won't ever run again?
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v2 8/9] util/mmap-alloc: Support RAM_NORESERVE via MAP_NORESERVE
Hi David,
Quoted 88 lines · David Hildenbrand
Quoted 9 lines · Marcel Apfelbaum
Hi David,
On Sun, Mar 7, 2021 at 3:18 PM David Hildenbrand <[email protected] <mailto:[email protected]>> wrote:
On 05.03.21 16:51, Peter Xu wrote: > On Fri, Mar 05, 2021 at 04:44:36PM +0100, David Hildenbrand wrote: >> On 05.03.21 16:42, Peter Xu wrote: >>> On Fri, Mar 05, 2021 at 11:16:33AM +0100, David HildenbrandQuoted 1 line · wrote:
>>>> +#define
OVERCOMMIT_MEMORY_PATH"
/proc/sys/vm/overcommit_memory"Quoted 14 lines
>>>> +static bool map_noreserve_effective(int fd, bool readonly, bool shared) >>>> +{ >>> >>> [...] >>> >>>> @@ -184,8 +251,7 @@ void *qemu_ram_mmap(int fd, >>>> size_t offset, total; >>>> void *ptr, *guardptr; >>>> - if (noreserve) { >>>> - error_report("Skipping reservation of swap space is not supported"); >>>> + if (noreserve && !map_noreserve_effective(fd, shared, readonly)) {Quoted 8 lines
>>> >>> Need to switch "shared" & "readonly"? >> >> Indeed, interestingly it has the same effect (as we don't have anonymous >> read-only memory in QEMU :) ) > > But note there is still a "g_assert(!shared || fd >= 0);"inside.. :)
Quoted 8 lines
Aaaaaand, I just figured that we actually can create shared anonymous memory in QEMU, simply via
-object memory-backend-ram,share=on
Introduced in
06329ccecfa0("mem: add share parameter to memory-backend-ram"). That's also where we introduced the "shared"flag
Quoted 5 lines
for
qemu_anon_ram_alloc().That commit mentions a use case for "RDMA devices in order to remap non-contiguous QEMU virtual addresses to a contiguous virtual address range.". I fail to understand why that requires sharing RAM with
child
Quoted 10 lines
processes.
Especially:
a)
qemu_ram_is_shared()returned false before patch #1.RAM_SHAREDis never set.b)
qemu_ram_remap()does not work as expected?c)
ram_discard_range()is broken with shared anonymous memory.Instead
Quoted 3 lines
of
MADV_DONTNEEDwe needMADV_REMOVE.This looks like a partially broken feature and I wonder if there is
an
Quoted 4 lines
actual user.
@Marcel, can you clarify if there is an actual use case for shared anonymous memory in QEMU? I.e., if the original use case that
required
Quoted 1 line
that change is valid? (and why it wasn't able to just use proper
shmem)
Quoted 8 lines
As you correctly stated, the PVRDMA device requires remapping of non-contiguous QEMU
virtual addresses to a contiguous virtual address range.In order to do so it calls
mremap (... , MREMAP_MAYMOVE | MREMAP_FIXED, ...)Thanks - I was missing who remaps and how (for a second I thought in another forked process).
docs/pvrdma.txtseems to describe the situation. Having to use anonymous shared memory is a bit unfortunate.
Quoted 8 lines
I yet haven't figured out how it is valid to remap parts of RAMBlocks to other locations via
MREMAP_MAYMOVE. This sounds to me like we are punching holes into RAMBlocks - that can't be right.Or maybe we are just shuffling around pages within a RAMBlock such that we don't actually punch holes?
Indeed, we are adding a new mapping , but we leave the previous one in place.
The VM will continue to work with the "original" RAM while the host RDMA subsystem
will work with the re-mapped one.
Thanks,
Marcel
Quoted 9 lines
Or does that happen when the source VM is stopped and won't ever run again?
--
Thanks,David / dhildenb
PATCH v2 9/9 hostmem: Wire up RAM_NORESERVE via "reserve" property
In reply to David HildenbrandLet's provide a way to control the use of RAM_NORESERVE via memory backends using the "reserve" property which defaults to true (old behavior).
Only POSIX supports setting the flag (and Linux support is checked at runtime, depending on the setting of "/proc/sys/vm/overcommit_memory"). Windows will bail out.
The target use case is virtio-mem, which dynamically exposes memory inside a large, sparse memory area to the VM. This essentially allows avoiding to set "/proc/sys/vm/overcommit_memory == 0") when using virtio-mem and also supporting hugetlbfs in the future.
--- backends/hostmem-file.c | 11 ++++++----- backends/hostmem-memfd.c | 1 + backends/hostmem-ram.c | 1 + backends/hostmem.c | 33 +++++++++++++++++++++++++++++++++ include/sysemu/hostmem.h | 2 +- 5 files changed, 42 insertions(+), 6 deletions(-)
backends/hostmem-file.c +6 -5
@@ -40,6 +40,7 @@ file_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) object_get_typename(OBJECT(backend)));#else HostMemoryBackendFile *fb =MEMORY_BACKEND_FILE (backend); +uint32_t ram_flags;gchar *name;if (!backend->size) {
@@ -52,11 +53,11 @@ file_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) } name =host_memory_backend_get_name (backend); -memory_region_init_ram_from_file (&backend->mr,OBJECT (backend), - name, - backend->size, fb->align, - (backend->share ?RAM_SHARED :0 ) | - (fb->is_pmem ?RAM_PMEM :0 ), + ram_flags = backend->share ?RAM_SHARED :0 ; + ram_flags |= backend->reserve ?0 :RAM_NORESERVE ; + ram_flags |= fb->is_pmem ?RAM_PMEM :0 ; +memory_region_init_ram_from_file (&backend->mr,OBJECT (backend), name, + backend->size, fb->align, ram_flags, fb->mem_path, fb->readonly, errp);g_free (name);#endif
backends/hostmem-memfd.c +1 -0
@@ -55,6 +55,7 @@ memfd_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) name =host_memory_backend_get_name (backend); ram_flags = backend->share ?RAM_SHARED :0 ; + ram_flags |= backend->reserve ?0 :RAM_NORESERVE ;memory_region_init_ram_from_fd (&backend->mr,OBJECT (backend), name, backend->size, ram_flags, fd,0 , errp);g_free (name);
backends/hostmem-ram.c +1 -0
@@ -29,6 +29,7 @@ ram_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) name =host_memory_backend_get_name (backend); ram_flags = backend->share ?RAM_SHARED :0 ; + ram_flags |= backend->reserve ?0 :RAM_NORESERVE ;memory_region_init_ram_flags_nomigrate (&backend->mr,OBJECT (backend), name, backend->size, ram_flags, errp);g_free (name);
backends/hostmem.c +33 -0
@@ -217,6 +217,11 @@ static void host_memory_backend_set_prealloc(Object *obj, bool value,Error *local_err =NULL ;HostMemoryBackend *backend =MEMORY_BACKEND (obj); +if (!backend->reserve && value) { +error_setg (errp,"'prealloc=on' and 'reserve=off' are incompatible" ); +return ; + } +if (!host_memory_backend_mr_inited (backend)) { backend->prealloc = value;return ;
@@ -268,6 +273,7 @@ static void host_memory_backend_init(Object *obj)/* TODO: convert access to globals to compat properties */ backend->merge =machine_mem_merge (machine); backend->dump =machine_dump_guest_core (machine); + backend->reserve = true; backend->prealloc_threads =1 ; }
@@ -426,6 +432,28 @@ static void host_memory_backend_set_share(Object *o, bool value, Error **errp) backend->share = value; } +static bool host_memory_backend_get_reserve (Object *o,Error **errp) +{ +HostMemoryBackend *backend =MEMORY_BACKEND (o); + +return backend->reserve; +} + +static void host_memory_backend_set_reserve (Object *o,bool value,Error **errp) +{ +HostMemoryBackend *backend =MEMORY_BACKEND (o); + +if (host_memory_backend_mr_inited (backend)) { +error_setg (errp,"cannot change property value" ); +return ; + } +if (backend->prealloc && !value) { +error_setg (errp,"'prealloc=on' and 'reserve=off' are incompatible" ); +return ; + } + backend->reserve = value; +} +static bool host_memory_backend_get_use_canonical_path (Object *obj,Error **errp) {
@@ -494,6 +522,11 @@ host_memory_backend_class_init(ObjectClass *oc, void *data) host_memory_backend_get_share, host_memory_backend_set_share);object_class_property_set_description (oc,"share" ,"Mark the memory as private to QEMU or shared" ); +object_class_property_add_bool (oc,"reserve" , + host_memory_backend_get_reserve, host_memory_backend_set_reserve); +object_class_property_set_description (oc,"reserve" , +"Reserve swap space (or huge pages under Linux) for the whole memory" +" backend, if supported by the OS." );/* * Do not delete/rename option. This option must be considered stable * (as if it didn't have the 'x-' prefix including deprecation period) as
include/sysemu/hostmem.h +1 -1
@@ -64,7 +64,7 @@ struct HostMemoryBackend {/* protected */ uint64_t size;bool merge, dump, use_canonical_path; -bool prealloc, is_mapped, share; +bool prealloc, is_mapped, share, reserve;uint32_t prealloc_threads;DECLARE_BITMAP (host_nodes,MAX_NODES +1 );HostMemPolicy policy;
Signature · 1 line
2.29.2
Re: [PATCH v2 9/9] hostmem: Wire up RAM_NORESERVE via "reserve" property
Quoted 8 lines · David Hildenbrand
Let's provide a way to control the use of
RAM_NORESERVEvia memory backends using the "reserve" property which defaults to true (old behavior).Only POSIX supports setting the flag (and Linux support is checked at runtime, depending on the setting of "
/proc/sys/vm/overcommit_memory"). Windows will bail out.
Quoted 7 lines
The target use case is virtio-mem, which dynamically exposes memory inside a large, sparse memory area to the VM. This essentially allows avoiding to set "
/proc/sys/vm/overcommit_memory== 0") when using virtio-mem and also supporting hugetlbfs in the future.Reviewed-by Peter Xu <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Paolo, will the whole series get through your memory API queue?
Also, do you think we should move backends/hostmem* under the "memory API" section in MAINTAINERS? Most memory backend changes also involve changes in the memory API.
Signature · 1 line
Eduardo