Postcopy implementation
v9 cover Postcopy implementation
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tag
Testing status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)
This work has been partially funded by the EU Orbit project:
see http://www.orbitproject.eu/about/
v9
Almost all of the changes are changes from review comments, and most are minor.
The following are new patches, mostly split out from earlier patches (The exception being adding userfaultfd.h header back in - which was in v5
but we took out, but is needed again due to 1842bdfd)
04/56 Move page_size_init earlier
08/56 qemu_ram_block_by_name
12/56 Factor out host_from_stream_offset call and check 15/56 Add Linux userfaultfd.h header back 21/56 migration_is_setup_or_active
30/56 migration_completion: Take current state 34/56 Maintain unsentmap
The previous patches 03,10,13,36/54 went in upstream already.
Fix for assert using hotplug (Thanks Bharata for spotting that) Fix for migrate_cancel a second time
Rework for migration_bitmap_rcu after Denis's deadlock fix
Rework ram_load into a separate postcopy loop
The 'sentmap' is now an 'unsentmap' - this saves a complement step at the end
The unsentmap creation is now split into a separate patch
The unsentmap is now stored in the RCU structure (although it can't
really resize during the migrate)
Fix for block migration
still not a suggested combination.
Minor:
trace_savevm_send_open_return_path added
split migration_is_active function into separate patch and made
migration_is_setup_or_active
move file reads into migrate_handle_advise and migrate_handle_packaged use of MIN in send-packaged-chunk
migration_thread_started -> migration_thread_running
updated qemu_get_buffer_in_place with Juan's version (and size_t'ified it) split the host_from_stream_offset change into a separate patch
'ram_load: Factor out host_from_stream_offset call and check'
split out ram_find_block_by_id into a separate patch and now
called qemu_ram_block_by_name
postcopy_discard_send_range etc now take start/length rather than start/end
(also added another trace)
split ram_save_host_page into ram_save_host_page/ram_save_target_page split host page cleanup function into a core that handles both passes
Dave
Dr. David Alan Gilbert (56):
Add postcopy documentation
Provide runtime Target page information
Move configuration section writing
Move page_size_init earlier
qemu_ram_block_from_host
qemu_ram_block_by_name
Rename mis->file to from_src_file
Add qemu_get_buffer_in_place to avoid copies some of the time Add wrapper for setting blocking status on a QEMUFile Add QEMU_MADV_NOHUGEPAGE
ram_debug_dump_bitmap: Dump a migration bitmap as text
ram_load: Factor out host_from_stream_offset call and check
migrate_init: Call from savevm
Rename save_live_complete to save_live_complete_precopy Add Linux userfaultfd.h header
Return path: Open a return path on QEMUFile for sockets Return path: socket_writev_buffer: Block even on non-blocking fd's Migration commands
Return path: Control commands
Return path: Send responses from destination to source migration_is_setup_or_active
Return path: Source handling of return path
Rework loadvm path for subloops
Add migration-capability boolean for postcopy-ram.
Add wrappers and handlers for sending/receiving the postcopy-ram
migration messages.
MIG_CMD_PACKAGED: Send a packaged chunk of migration stream Modify save_live_pending for postcopy
postcopy: OS support test
migrate_start_postcopy: Command to trigger transition to postcopy
migration_completion: Take current state MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state Avoid sending vmdescription during postcopy
Add qemu_savevm_state_complete_postcopy
Postcopy: Maintain unsentmap
Postcopy: Calculate discard
postcopy: Incoming initialisation
postcopy: ram_enable_notify to switch on userfault
Postcopy: Postcopy startup in migration thread
Postcopy: End of iteration
Page request: Add MIG_RP_MSG_REQ_PAGES reverse command Page request: Process incoming page request
Page request: Consume pages off the post-copy queue postcopy_ram.c: place_page and helpers
Postcopy: Use helpers to map pages during migration
postcopy: Check order of received target pages
Don't sync dirty bitmaps in postcopy
Don't iterate on precopy-only devices during postcopy Host page!=target page: Cleanup bitmaps
Round up RAMBlock sizes to host page sizes
Postcopy; Handle userfault requests
Start up a postcopy/listener thread ready for incoming page data
postcopy: Wire up loadvm_postcopy_handle_ commands
Postcopy: Mark nohugepage before discard
End of migration for postcopy
Disable mlock around incoming postcopy
Inhibit ballooning during postcopy
balloon.c | 11 + docs/migration.txt | 191 ++++++++ exec.c | 92 +++- hmp-commands.hx | 15 + hmp.c | 7 + hmp.h | 1 + hw/ppc/spapr.c | 2 +- hw/virtio/virtio-balloon.c | 4 +- include/exec/cpu-common.h | 4 + include/exec/exec-all.h | 1 - include/exec/ram_addr.h | 2 - include/migration/migration.h | 121 ++++- include/migration/postcopy-ram.h | 99 ++++ include/migration/qemu-file.h | 10 + include/migration/vmstate.h | 8 +- include/qemu-common.h | 1 + include/qemu/osdep.h | 9 + include/qemu/typedefs.h | 3 + include/sysemu/balloon.h | 2 + include/sysemu/sysemu.h | 46 +- kvm-all.c | 1 - linux-headers/linux/userfaultfd.h | 167 +++++++ migration/Makefile.objs | 2 +- migration/block.c | 9 +- migration/migration.c | 723 +++++++++++++++++++++++++-- migration/postcopy-ram.c | 760 +++++++++++++++++++++++++++++ migration/qemu-file-unix.c | 111 ++++- migration/qemu-file.c | 64 +++ migration/ram.c | 997 +++++++++++++++++++++++++++++++++++--- migration/savevm.c | 823 +++++++++++++++++++++++++++---- qapi-schema.json | 18 +- qmp-commands.hx | 19 + qtest.c | 1 - trace-events | 84 +++- vl.c | 1 + 35 files changed, 4159 insertions(+), 250 deletions(-) create mode 100644 include/migration/postcopy-ram.h create mode 100644 linux-headers/linux/userfaultfd.h create mode 100644 migration/postcopy-ram.c
Signature · 1 line
2.5.0
PATCH v9 1/56 Add postcopy documentation
From: "Dr. David Alan Gilbert" <[email protected]>
--- docs/migration.txt | 191 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+)
docs/migration.txt +191 -0
@@ -291,3 +291,194 @@ save/send this state when we are in the middle of a pio operation (that is what ide_drive_pio_state_needed() checks). If DRQ_STAT is not enabled, the values on that fields are garbage and don't need to be sent. + += Return path = + +In most migration scenarios there is only a single data path that runs +from the source VM to the destination, typically along a single fd (although +possibly with another fd or similar for some fast way of throwing pages across). + +However, some uses need two way communication; in particular the Postcopy +destination needs to be able to request pages on demand from the source. + +For these scenarios there is a 'return path' from the destination to the source; +qemu_file_get_return_path(QEMUFile* fwdpath) gives the QEMUFile* for the return +path. + + Source side + Forward path - written by migration thread + Return path - opened by main thread, read by return-path thread + + Destination side + Forward path - read by main thread + Return path - opened by main thread, written by main thread AND postcopy + thread (protected by rp_mutex) + += Postcopy = +'Postcopy' migration is a way to deal with migrations that refuse to converge +(or take too long to converge) its plus side is that there is an upper bound on +the amount of migration traffic and time it takes, the down side is that during +the postcopy phase, a failure of *either* side or the network connection causes +the guest to be lost. + +In postcopy the destination CPUs are started before all the memory has been +transferred, and accesses to pages that are yet to be transferred cause +a fault that's translated by QEMU into a request to the source QEMU. + +Postcopy can be combined with precopy (i.e. normal migration) so that if precopy +doesn't finish in a given time the switch is made to postcopy. + +=== Enabling postcopy === + +To enable postcopy, issue this command on the monitor prior to the +start of migration: + +migrate_set_capability x-postcopy-ram on + +The normal commands are then used to start a migration, which is still +started in precopy mode. Issuing: + +migrate_start_postcopy + +will now cause the transition from precopy to postcopy. +It can be issued immediately after migration is started or any +time later on. Issuing it after the end of a migration is harmless. + +Note: During the postcopy phase, the bandwidth limits set using +migrate_set_speed is ignored (to avoid delaying requested pages that +the destination is waiting for). + +=== Postcopy device transfer === + +Loading of device data may cause the device emulation to access guest RAM +that may trigger faults that have to be resolved by the source, as such +the migration stream has to be able to respond with page data *during* the +device load, and hence the device data has to be read from the stream completely +before the device load begins to free the stream up. This is achieved by +'packaging' the device data into a blob that's read in one go. + +Source behaviour + +Until postcopy is entered the migration stream is identical to normal +precopy, except for the addition of a 'postcopy advise' command at +the beginning, to tell the destination that postcopy might happen. +When postcopy starts the source sends the page discard data and then +forms the 'package' containing: + + Command: 'postcopy listen' + The device state + A series of sections, identical to the precopy streams device state stream + containing everything except postcopiable devices (i.e. RAM) + Command: 'postcopy run' + +The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the +contents are formatted in the same way as the main migration stream. + +During postcopy the source scans the list of dirty pages and sends them +to the destination without being requested (in much the same way as precopy), +however when a page request is received from the destination, the dirty page +scanning restarts from the requested location. This causes requested pages +to be sent quickly, and also causes pages directly after the requested page +to be sent quickly in the hope that those pages are likely to be used +by the destination soon. + +Destination behaviour + +Initially the destination looks the same as precopy, with a single thread +reading the migration stream; the 'postcopy advise' and 'discard' commands +are processed to change the way RAM is managed, but don't affect the stream +processing. + +------------------------------------------------------------------------------ + 1 2 3 4 5 6 7 +main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) +thread | | + | (page request) + | \___ + v \ +listen thread: --- page -- page -- page -- page -- page -- + + a b c +------------------------------------------------------------------------------ + +On receipt of CMD_PACKAGED (1) + All the data associated with the package - the ( ... ) section in the +diagram - is read into memory (into a QEMUSizedBuffer), and the main thread +recurses into qemu_loadvm_state_main to process the contents of the package (2) +which contains commands (3,6) and devices (4...) + +On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package) +a new thread (a) is started that takes over servicing the migration stream, +while the main thread carries on loading the package. It loads normal +background page data (b) but if during a device load a fault happens (5) the +returned page (c) is loaded by the listen thread allowing the main threads +device load to carry on. + +The last thing in the CMD_PACKAGED is a 'RUN' command (6) letting the destination +CPUs start running. +At the end of the CMD_PACKAGED (7) the main thread returns to normal running behaviour +and is no longer used by migration, while the listen thread carries +on servicing page data until the end of migration. + +=== Postcopy states === + +Postcopy moves through a series of states (see postcopy_state) from +ADVISE->DISCARD->LISTEN->RUNNING->END + + Advise: Set at the start of migration if postcopy is enabled, even + if it hasn't had the start command; here the destination + checks that its OS has the support needed for postcopy, and performs + setup to ensure the RAM mappings are suitable for later postcopy. + The destination will fail early in migration at this point if the + required OS support is not present. + (Triggered by reception of POSTCOPY_ADVISE command) + + Discard: Entered on receipt of the first 'discard' command; prior to + the first Discard being performed, hugepages are switched off + (using madvise) to ensure that no new huge pages are created + during the postcopy phase, and to cause any huge pages that + have discards on them to be broken. + + Listen: The first command in the package, POSTCOPY_LISTEN, switches + the destination state to Listen, and starts a new thread + (the 'listen thread') which takes over the job of receiving + pages off the migration stream, while the main thread carries + on processing the blob. With this thread able to process page + reception, the destination now 'sensitises' the RAM to detect + any access to missing pages (on Linux using the 'userfault' + system). + + Running: POSTCOPY_RUN causes the destination to synchronise all + state and start the CPUs and IO devices running. The main + thread now finishes processing the migration package and + now carries on as it would for normal precopy migration + (although it can't do the cleanup it would do as it + finishes a normal migration). + + End: The listen thread can now quit, and perform the cleanup of migration + state, the migration is now complete. + +=== Source side page maps === + +The source side keeps two bitmaps during postcopy; 'the migration bitmap' +and 'unsent map'. The 'migration bitmap' is basically the same as in +the precopy case, and holds a bit to indicate that page is 'dirty' - +i.e. needs sending. During the precopy phase this is updated as the CPU +dirties pages, however during postcopy the CPUs are stopped and nothing +should dirty anything any more. + +The 'unsent map' is used for the transition to postcopy. It is a bitmap that +has a bit cleared whenever a page is sent to the destination, however during +the transition to postcopy mode it is combined with the migration bitmap +to form a set of pages that: + a) Have been sent but then redirtied (which must be discarded) + b) Have not yet been sent - which also must be discarded to cause any + transparent huge pages built during precopy to be broken. + +Note that the contents of the unsentmap are sacrificed during the calculation +of the discard set and thus aren't valid once in postcopy. The dirtymap +is still valid and is used to ensure that no page is sent more than once. Any +request for a page that has already been sent is ignored. Duplicate requests +such as this can happen as a page is sent at about the same time the +destination accesses it. +
Signature · 1 line
2.5.0
PATCH v9 2/56 Provide runtime Target page information
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The migration code generally is built target-independent, however there are a few places where knowing the target page size would avoid artificially moving stuff into migration/ram.c.
Provide 'qemu_target_page_bits()' that returns TARGET_PAGE_BITS to other bits of code so that they can stay target-independent.
--- exec.c | 10 ++++++++++ include/sysemu/sysemu.h | 1 + 2 files changed, 11 insertions(+)
exec.c +10 -0
@@ -3500,6 +3500,16 @@ int cpu_memory_rw_debug(CPUState *cpu, target_ulong addr, }return 0 ; } + +/* +* Allows code that needs to deal with migration bitmaps etc to still be built +* target independent. +*/ +size_t qemu_target_page_bits (void ) +{ +return TARGET_PAGE_BITS ; +} +#endif /*
include/sysemu/sysemu.h +1 -0
@@ -70,6 +70,7 @@ void qemu_system_killed(int signal, pid_t pid);void qemu_devices_reset (void );void qemu_system_reset (bool report);void qemu_system_guest_panicked (void ); +size_t qemu_target_page_bits (void );void qemu_add_exit_notifier (Notifier *notify);void qemu_remove_exit_notifier (Notifier *notify);
Signature · 1 line
2.5.0
PATCH v9 3/56 Move configuration section writing
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The vmstate_configuration is currently written
in 'qemu_savevm_state_begin', move it to
'qemu_savevm_state_header' since it's got a hard
requirement that it must be the 1st thing after
the header.
(In postcopy some 'command' sections get sent
early before the saving of the main sections
and hence before qemu_savevm_state_begin).
--- migration/savevm.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
migration/savevm.c +6 -5
@@ -713,6 +713,12 @@ void qemu_savevm_state_header(QEMUFile *f)trace_savevm_state_header ();qemu_put_be32 (f,QEMU_VM_FILE_MAGIC );qemu_put_be32 (f,QEMU_VM_FILE_VERSION ); + +if (!savevm_state.skip_configuration) { +qemu_put_byte (f,QEMU_VM_CONFIGURATION ); +vmstate_save_state (f, &vmstate_configuration, &savevm_state,0 ); + } + } void qemu_savevm_state_begin(QEMUFile *f,
@@ -729,11 +735,6 @@ void qemu_savevm_state_begin(QEMUFile *f, se->ops->set_params (params, se->opaque); } -if (!savevm_state.skip_configuration) { -qemu_put_byte (f,QEMU_VM_CONFIGURATION ); -vmstate_save_state (f, &vmstate_configuration, &savevm_state,0 ); - } - QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {if (!se->ops || !se->ops->save_live_setup) {continue ;
Signature · 1 line
2.5.0
PATCH v9 4/56 Move page_size_init earlier
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The HOST_PAGE_ALIGN macros don't work until the page size variables have been set up; later in postcopy I use those macros in the RAM code, and it can be triggered using -object.
Fix this by initialising page_size_init() earlier - it's currently initialised inside the accelerators, move it up into vl.c.
--- include/exec/exec-all.h | 1 - include/qemu-common.h | 1 + kvm-all.c | 1 - qtest.c | 1 - vl.c | 1 + 5 files changed, 2 insertions(+), 3 deletions(-)
include/exec/exec-all.h +0 -1
@@ -72,7 +72,6 @@ void restore_state_to_opc(CPUArchState *env, struct TranslationBlock *tb,void cpu_gen_init (void );bool cpu_restore_state (CPUState *cpu,uintptr_t searched_pc); -void page_size_init (void ); void QEMU_NORETURN cpu_resume_from_signal(CPUState *cpu, void *puc); void QEMU_NORETURN cpu_io_recompile(CPUState *cpu, uintptr_t retaddr);
include/qemu-common.h +1 -0
@@ -499,5 +499,6 @@ size_t buffer_find_nonzero_offset(const void *buf, size_t len);int parse_debug_env (const char *name,int max,int initial);const char *qemu_ether_ntoa (const MACAddr *mac); +void page_size_init (void );#endif
kvm-all.c +0 -1
@@ -1461,7 +1461,6 @@ static int kvm_init(MachineState *ms)* page size for the system though. */ assert (TARGET_PAGE_SIZE <=getpagesize ()); -page_size_init (); s->sigmask_len =8 ;
qtest.c +0 -1
@@ -657,7 +657,6 @@ void qtest_init(const char *qtest_chrdev, const char *qtest_log, Error **errp) inbuf =g_string_new ("" ); qtest_chr = chr; -page_size_init (); }bool qtest_driver (void )
vl.c +1 -0
@@ -4254,6 +4254,7 @@ int main(int argc, char **argv, char **envp)exit (1 ); } +page_size_init ();socket_init (); if (qemu_opts_foreach(qemu_find_opts("object"),
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 04/56] Move page_size_init earlier
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The
HOST_PAGE_ALIGNmacros don't work until the page size variables have been set up; later in postcopy I use those macros in the RAM code, and it can be triggered using -object.Fix this by initialising
page_size_init()earlier - it's currently initialised inside the accelerators, move it up intovl.c.Signed-off-by Dr. David Alan Gilbert <[email protected]>
PATCH v9 5/56 qemu_ram_block_from_host
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy sends RAMBlock names and offsets over the wire (since it can't rely on the order of ramaddr being the same), and it starts out with HVA fault addresses from the kernel.
qemu_ram_block_from_host translates a HVA into a RAMBlock, an offset in the RAMBlock and the global ram_addr_t value.
Rewrite qemu_ram_addr_from_host to use qemu_ram_block_from_host.
Provide qemu_ram_get_idstr since its the actual name text sent on the wire.
--- exec.c | 54 +++++++++++++++++++++++++++++++++++++++-------- include/exec/cpu-common.h | 3 +++ include/exec/ram_addr.h | 2 -- include/qemu/typedefs.h | 1 + 4 files changed, 49 insertions(+), 11 deletions(-)
exec.c +45 -9
@@ -1375,6 +1375,11 @@ static RAMBlock *find_ram_block(ram_addr_t addr)return NULL ; } +const char *qemu_ram_get_idstr (RAMBlock *rb) +{ +return rb->idstr; +} +/* Called with iothread lock held. */ void qemu_ram_set_idstr (ram_addr_t addr,const char *name,DeviceState *dev) {
@@ -1875,8 +1880,16 @@ static void *qemu_ram_ptr_length(ram_addr_t addr, hwaddr *size) } } -/* Some of the softmmu routines need to translate from a host pointer -* (typically a TLB entry) back to a ram offset. +/* +* Translates a host ptr back to a RAMBlock, a ram_addr and an offset +* in that RAMBlock. +* +* ptr: Host pointer to look up +* round_offset: If true round the result offset down to a page boundary +* *ram_addr: set to result ram_addr +* *offset: set to result offset within the RAMBlock +* +* Returns: RAMBlock (or NULL if not found) * * By the time this function returns, the returned pointer is not protected * by RCU anymore. If the caller is not within an RCU critical section and
@@ -1884,18 +1897,22 @@ static void *qemu_ram_ptr_length(ram_addr_t addr, hwaddr *size)* pointer, such as a reference to the region that includes the incoming * ram_addr_t. */ -MemoryRegion *qemu_ram_addr_from_host (void *ptr,ram_addr_t *ram_addr) +RAMBlock *qemu_ram_block_from_host (void *ptr,bool round_offset, +ram_addr_t *ram_addr, +ram_addr_t *offset) {RAMBlock *block;uint8_t *host = ptr; -MemoryRegion *mr;if (xen_enabled ()) {rcu_read_lock (); *ram_addr =xen_ram_addr_from_mapcache (ptr); - mr =qemu_get_ram_block (*ram_addr)->mr; + block =qemu_get_ram_block (*ram_addr); +if (block) { + *offset = (host - block->host); + }rcu_read_unlock (); -return mr; +return block; }rcu_read_lock ();
@@ -1918,10 +1935,29 @@ MemoryRegion *qemu_ram_addr_from_host(void *ptr, ram_addr_t *ram_addr)return NULL ; found: - *ram_addr = block->offset + (host - block->host); - mr = block->mr; + *offset = (host - block->host); +if (round_offset) { + *offset &=TARGET_PAGE_MASK ; + } + *ram_addr = block->offset + *offset;rcu_read_unlock (); -return mr; +return block; +} + +/* Some of the softmmu routines need to translate from a host pointer +(typically a TLB entry) back to a ram offset. */ +MemoryRegion *qemu_ram_addr_from_host (void *ptr,ram_addr_t *ram_addr) +{ +RAMBlock *block; +ram_addr_t offset;/* Not used */ + + block =qemu_ram_block_from_host (ptr, false, ram_addr, &offset); + +if (!block) { +return NULL ; + } + +return block->mr; } static void notdirty_mem_write(void *opaque, hwaddr ram_addr,
include/exec/cpu-common.h +3 -0
@@ -64,8 +64,11 @@ typedef uint32_t CPUReadMemoryFunc(void *opaque, hwaddr addr);void qemu_ram_remap (ram_addr_t addr,ram_addr_t length);/* This should not be used by devices. */ MemoryRegion *qemu_ram_addr_from_host (void *ptr,ram_addr_t *ram_addr); +RAMBlock *qemu_ram_block_from_host (void *ptr,bool round_offset, +ram_addr_t *ram_addr,ram_addr_t *offset);void qemu_ram_set_idstr (ram_addr_t addr,const char *name,DeviceState *dev);void qemu_ram_unset_idstr (ram_addr_t addr); +const char *qemu_ram_get_idstr (RAMBlock *rb);void cpu_physical_memory_rw (hwaddr addr,uint8_t *buf,int len,int is_write);
include/exec/ram_addr.h +0 -2
@@ -22,8 +22,6 @@ #ifndef CONFIG_USER_ONLY #include "hw/xen/xen.h" -typedef struct RAMBlock RAMBlock; - struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr;
include/qemu/typedefs.h +1 -0
@@ -78,6 +78,7 @@ typedef struct QEMUSizedBuffer QEMUSizedBuffer;typedef struct QEMUTimerListGroup QEMUTimerListGroup ;typedef struct QEMUTimer QEMUTimer ;typedef struct Range Range ; +typedef struct RAMBlock RAMBlock ;typedef struct SerialState SerialState ;typedef struct SHPCDevice SHPCDevice ;typedef struct SMBusDevice SMBusDevice ;
Signature · 1 line
2.5.0
PATCH v9 6/56 qemu_ram_block_by_name
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add a function to find a RAMBlock by name; use it in two of the places that already open code that loop; we've got another use later in postcopy.
--- exec.c | 20 ++++++++++++++++++++ include/exec/cpu-common.h | 1 + migration/ram.c | 35 +++++++++++++++-------------------- 3 files changed, 36 insertions(+), 20 deletions(-)
exec.c +20 -0
@@ -1944,6 +1944,26 @@ found:return block; } +/* +* Finds the named RAMBlock +* +* name: The name of RAMBlock to find +* +* Returns: RAMBlock (or NULL if not found) +*/ +RAMBlock *qemu_ram_block_by_name (const char *name) +{ +RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { +if (!strcmp (name, block->idstr)) { +return block; + } + } + +return NULL ; +} +/* Some of the softmmu routines need to translate from a host pointer (typically a TLB entry) back to a ram offset. */ MemoryRegion *qemu_ram_addr_from_host (void *ptr,ram_addr_t *ram_addr)
include/exec/cpu-common.h +1 -0
@@ -64,6 +64,7 @@ typedef uint32_t CPUReadMemoryFunc(void *opaque, hwaddr addr);void qemu_ram_remap (ram_addr_t addr,ram_addr_t length);/* This should not be used by devices. */ MemoryRegion *qemu_ram_addr_from_host (void *ptr,ram_addr_t *ram_addr); +RAMBlock *qemu_ram_block_by_name (const char *name);RAMBlock *qemu_ram_block_from_host (void *ptr,bool round_offset,ram_addr_t *ram_addr,ram_addr_t *offset);void qemu_ram_set_idstr (ram_addr_t addr,const char *name,DeviceState *dev);
migration/ram.c +15 -20
@@ -1420,14 +1420,12 @@ static inline void *host_from_stream_offset(QEMUFile *f,qemu_get_buffer (f, (uint8_t *)id, len); id[len] =0 ; - QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { -if (!strncmp (id, block->idstr,sizeof (id)) && - block->max_length > offset) { -return block->host + offset; - } + block =qemu_ram_block_by_name (id); +if (block && block->max_length > offset) { +return block->host + offset; } -error_report ("Can't find block %s!" , id); +error_report ("Can't find block %s" , id);return NULL ; }
@@ -1576,23 +1574,20 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) id[len] =0 ; length =qemu_get_be64 (f); - QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { -if (!strncmp (id, block->idstr,sizeof (id))) { -if (length != block->used_length) { -Error *local_err =NULL ; + block =qemu_ram_block_by_name (id); +if (block) { +if (length != block->used_length) { +Error *local_err =NULL ; - ret =qemu_ram_resize (block->offset, length, &local_err); -if (local_err) { -error_report_err (local_err); - } + ret =qemu_ram_resize (block->offset, length, + &local_err); +if (local_err) { +error_report_err (local_err); } -ram_control_load_hook (f,RAM_CONTROL_BLOCK_REG , - block->idstr); -break ; } - } - -if (!block) { +ram_control_load_hook (f,RAM_CONTROL_BLOCK_REG , + block->idstr); + }else {error_report ("Unknown ramblock \"%s\", cannot " "accept migration" , id); ret = -EINVAL ;
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 06/56] qemu_ram_block_by_name
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add a function to find a RAMBlock by name; use it in two of the places that already open code that loop; we've got another use later in postcopy.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
migration/ram.c static inline void *host_from_stream_offset(QEMUFile *f, · line 1428
Quoted 2 lines
migration/ram.c +1
-1-error_report ("Can't find block %s!" , id); +error_report ("Can't find block %s" , id);
No bangs on error messages? You are a boring man O:-)
PATCH v9 7/56 Rename mis->file to from_src_file
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
'file' becomes confusing when you have flows in each direction; rename to make it clear.
This leaves just the main forward direction ms->file, which is used in a lot of places and is probably not worth renaming given the churn.
--- include/migration/migration.h | 2 +- migration/migration.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
include/migration/migration.h +1 -1
@@ -48,7 +48,7 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;/* State for the incoming migration */ struct MigrationIncomingState { -QEMUFile *file; +QEMUFile *from_src_file;/* See savevm.c */ LoadStateEntry_Head loadvm_handlers;
migration/migration.c +1 -1
@@ -95,7 +95,7 @@ MigrationIncomingState *migration_incoming_get_current(void)MigrationIncomingState *migration_incoming_state_new (QEMUFile * f) { mis_current =g_new0 (MigrationIncomingState,1 ); - mis_current->file = f; + mis_current->from_src_file = f;QLIST_INIT (&mis_current->loadvm_handlers);return mis_current;
Signature · 1 line
2.5.0
PATCH v9 8/56 Add qemu_get_buffer_in_place to avoid copies some of the time
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
qemu_get_buffer always copies the data it reads to a users buffer, however in many cases the file buffer inside qemu_file could be given back to the caller, avoiding the copy. This isn't always possible depending on the size and alignment of the data.
Thus 'qemu_get_buffer_in_place' either copies the data to a supplied buffer or updates a pointer to the internal buffer if convenient.
--- include/migration/qemu-file.h | 2 ++ migration/qemu-file.c | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
include/migration/qemu-file.h +2 -0
@@ -163,9 +163,11 @@ void qemu_put_be32(QEMUFile *f, unsigned int v);void qemu_put_be64 (QEMUFile *f,uint64_t v);size_t qemu_peek_buffer (QEMUFile *f,uint8_t **buf,size_t size,size_t offset);size_t qemu_get_buffer (QEMUFile *f,uint8_t *buf,size_t size); +size_t qemu_get_buffer_in_place (QEMUFile *f,uint8_t **buf,size_t size);ssize_t qemu_put_compression_data (QEMUFile *f,const uint8_t *p,size_t size,int level);int qemu_put_qemu_file (QEMUFile *f_des,QEMUFile *f_src); +/* * Note that you can only peek continuous bytes from where the current pointer * is; you aren't guaranteed to be able to peak to +n bytes unless you've
migration/qemu-file.c +37 -0
@@ -434,6 +434,43 @@ size_t qemu_get_buffer(QEMUFile *f, uint8_t *buf, size_t size) }/* +* Read 'size' bytes of data from the file. +* 'size' can be larger than the internal buffer. +* +* The data: +* may be held on an internal buffer (in which case *buf is updated +* to point to it) that is valid until the next qemu_file operation. +* OR +* will be copied to the *buf that was passed in. +* +* The code tries to avoid the copy if possible. +* +* It will return size bytes unless there was an error, in which case it will +* return as many as it managed to read (assuming blocking fd's which +* all current QEMUFile are) +* +* Note: Since **buf may get changed, the caller should take care to +* keep a pointer to the original buffer if it needs to deallocate it. +*/ +size_t qemu_get_buffer_in_place (QEMUFile *f,uint8_t **buf,size_t size) +{ +if (size <IO_BUF_SIZE ) { +size_t res; +uint8_t *src; + + res =qemu_peek_buffer (f, &src, size,0 ); + +if (res == size) { +qemu_file_skip (f, res); + *buf = src; +return res; + } + } + +return qemu_get_buffer (f, *buf, size); +} + +/* * Peeks a single byte from the buffer; this isn't guaranteed to work if * offset leaves a gap after the previous read/peeked data. */
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 08/56] Add qemu_get_buffer_in_place to avoid copies some of the time
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
qemu_get_bufferalways copies the data it reads to a users buffer, however in many cases the file buffer insideqemu_filecould be given back to the caller, avoiding the copy. This isn't always possible depending on the size and alignment of the data.Thus '
qemu_get_buffer_in_place' either copies the data to a supplied buffer or updates a pointer to the internal buffer if convenient.Signed-off-by Dr. David Alan Gilbert <[email protected]>
PATCH v9 9/56 Add wrapper for setting blocking status on a QEMUFile
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add a wrapper to change the blocking status on a QEMUFile rather than having to use qemu_set_block(qemu_get_fd(f)); it seems best to avoid exposing the fd since not all QEMUFile's really have one. With this wrapper we could move the implementation down to be different on different transports.
--- include/migration/qemu-file.h | 1 + migration/qemu-file.c | 15 +++++++++++++++ 2 files changed, 16 insertions(+)
include/migration/qemu-file.h +1 -0
@@ -197,6 +197,7 @@ int qemu_file_get_error(QEMUFile *f);void qemu_file_set_error (QEMUFile *f,int ret);int qemu_file_shutdown (QEMUFile *f);void qemu_fflush (QEMUFile *f); +void qemu_file_set_blocking (QEMUFile *f,bool block);static inline void qemu_put_be64s (QEMUFile *f,const uint64_t *pv) {
migration/qemu-file.c +15 -0
@@ -648,3 +648,18 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])return res == len ? res :0 ; } + +/* +* Set the blocking state of the QEMUFile. +* Note: On some transports the OS only keeps a single blocking state for +* both directions, and thus changing the blocking on the main +* QEMUFile can also affect the return path. +*/ +void qemu_file_set_blocking (QEMUFile *f,bool block) +{ +if (block) { +qemu_set_block (qemu_get_fd (f)); + }else { +qemu_set_nonblock (qemu_get_fd (f)); + } +}
Signature · 1 line
2.5.0
PATCH v9 10/56 Add QEMU_MADV_NOHUGEPAGE
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add QEMU_MADV_NOHUGEPAGE as an OS-independent version of MADV_NOHUGEPAGE.
We include sys/mman.h before making the test to ensure that we pick up the system defines.
--- include/qemu/osdep.h | 9 +++++++++ 1 file changed, 9 insertions(+)
include/qemu/osdep.h +9 -0
@@ -139,6 +139,8 @@ void qemu_anon_ram_free(void *ptr, size_t size);#if defined(CONFIG_MADVISE ) +#include <sys/mman.h> +#define QEMU_MADV_WILLNEED MADV_WILLNEED#define QEMU_MADV_DONTNEED MADV_DONTNEED#ifdef MADV_DONTFORK
@@ -171,6 +173,11 @@ void qemu_anon_ram_free(void *ptr, size_t size);#else #define QEMU_MADV_HUGEPAGE QEMU_MADV_INVALID#endif +#ifdef MADV_NOHUGEPAGE +#define QEMU_MADV_NOHUGEPAGE MADV_NOHUGEPAGE +#else +#define QEMU_MADV_NOHUGEPAGE QEMU_MADV_INVALID +#endif #elif defined(CONFIG_POSIX_MADVISE)
@@ -182,6 +189,7 @@ void qemu_anon_ram_free(void *ptr, size_t size);#define QEMU_MADV_DODUMP QEMU_MADV_INVALID#define QEMU_MADV_DONTDUMP QEMU_MADV_INVALID#define QEMU_MADV_HUGEPAGE QEMU_MADV_INVALID +#define QEMU_MADV_NOHUGEPAGE QEMU_MADV_INVALID#else /* no-op */
@@ -193,6 +201,7 @@ void qemu_anon_ram_free(void *ptr, size_t size);#define QEMU_MADV_DODUMP QEMU_MADV_INVALID#define QEMU_MADV_DONTDUMP QEMU_MADV_INVALID#define QEMU_MADV_HUGEPAGE QEMU_MADV_INVALID +#define QEMU_MADV_NOHUGEPAGE QEMU_MADV_INVALID#endif
Signature · 1 line
2.5.0
PATCH v9 11/56 ram_debug_dump_bitmap: Dump a migration bitmap as text
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Useful for debugging the migration bitmap and other bitmaps of the same format (including the sentmap in postcopy).
The bitmap is printed to stderr.
Lines that are all the expected value are excluded so the output can be quite compact for many bitmaps.
--- include/migration/migration.h | 1 + migration/ram.c | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+)
include/migration/migration.h +1 -0
@@ -145,6 +145,7 @@ uint64_t xbzrle_mig_pages_cache_miss(void);double xbzrle_mig_cache_miss_rate (void );void ram_handle_compressed (void *host,uint8_t ch,uint64_t size); +void ram_debug_dump_bitmap (unsigned long *todump,bool expected);/** * @migrate_add_blocker - prevent migration from proceeding
migration/ram.c +39 -0
@@ -1160,6 +1160,45 @@ void migration_bitmap_extend(ram_addr_t old, ram_addr_t new) } } +/* +* 'expected' is the value you expect the bitmap mostly to be full +* of; it won't bother printing lines that are all this value. +* If 'todump' is null the migration bitmap is dumped. +*/ +void ram_debug_dump_bitmap (unsigned long *todump,bool expected) +{ +int64_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +int64_t cur; +int64_t linelen =128 ; +char linebuf[129 ]; + +if (!todump) { + todump =atomic_rcu_read (&migration_bitmap_rcu)->bmap; + } + +for (cur =0 ; cur < ram_pages; cur += linelen) { +int64_t curb; +bool found = false; +/* +* Last line; catch the case where the line length +* is longer than remaining ram +*/ +if (cur + linelen > ram_pages) { + linelen = ram_pages - cur; + } +for (curb =0 ; curb < linelen; curb++) { +bool thisbit =test_bit (cur + curb, todump); + linebuf[curb] = thisbit ?'1' :'.' ; + found = found || (thisbit != expected); + } +if (found) { + linebuf[curb] ='\0' ; +fprintf (stderr,"0x%08" PRIx64" : %s\n" , cur, linebuf); + } + } +} +/* Each of ram_save_setup, ram_save_iterate and ram_save_complete has * long-running RCU critical section. When rcu-reclaims in the code * start to become numerous it will be necessary to reduce the
Signature · 1 line
2.5.0
PATCH v9 12/56 ram_load: Factor out host_from_stream_offset call and check
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The main RAM load loop has a call to host_from_stream_offset for each page type that actually loads data with the same test; factor it out before the switch.
The host = NULL is to silence a bogus gcc warning of
an unitialised in the RAM_SAVE_COMPRESS_PAGE case, it doesn't seem to realise that host is always initialised by the if at the top in the cases the switch takes.
--- migration/ram.c | 41 +++++++++++++++-------------------------- 1 file changed, 15 insertions(+), 26 deletions(-)
migration/ram.c +15 -26
@@ -1592,13 +1592,23 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)rcu_read_lock ();while (!ret && !(flags &RAM_SAVE_FLAG_EOS )) {ram_addr_t addr, total_ram_bytes; -void *host; +void *host =NULL ;uint8_t ch; addr =qemu_get_be64 (f); flags = addr & ~TARGET_PAGE_MASK ; addr &=TARGET_PAGE_MASK ; +if (flags & (RAM_SAVE_FLAG_COMPRESS |RAM_SAVE_FLAG_PAGE | +RAM_SAVE_FLAG_COMPRESS_PAGE |RAM_SAVE_FLAG_XBZRLE )) { + host =host_from_stream_offset (f, addr, flags); +if (!host) { +error_report ("Illegal RAM offset " RAM_ADDR_FMT , addr); + ret = -EINVAL ; +break ; + } + } +switch (flags & ~RAM_SAVE_FLAG_CONTINUE ) {case RAM_SAVE_FLAG_MEM_SIZE :/* Synchronize RAM block list */
@@ -1635,33 +1645,17 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) total_ram_bytes -= length; }break ; +case RAM_SAVE_FLAG_COMPRESS : - host =host_from_stream_offset (f, addr, flags); -if (!host) { -error_report ("Illegal RAM offset " RAM_ADDR_FMT , addr); - ret = -EINVAL ; -break ; - } ch =qemu_get_byte (f);ram_handle_compressed (host, ch,TARGET_PAGE_SIZE );break ; +case RAM_SAVE_FLAG_PAGE : - host =host_from_stream_offset (f, addr, flags); -if (!host) { -error_report ("Illegal RAM offset " RAM_ADDR_FMT , addr); - ret = -EINVAL ; -break ; - }qemu_get_buffer (f, host,TARGET_PAGE_SIZE );break ; -case RAM_SAVE_FLAG_COMPRESS_PAGE : - host =host_from_stream_offset (f, addr, flags); -if (!host) { -error_report ("Invalid RAM offset " RAM_ADDR_FMT , addr); - ret = -EINVAL ; -break ; - } +case RAM_SAVE_FLAG_COMPRESS_PAGE : len =qemu_get_be32 (f);if (len <0 || len >compressBound (TARGET_PAGE_SIZE )) {error_report ("Invalid compressed data length: %d" , len);
@@ -1671,13 +1665,8 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)qemu_get_buffer (f, compressed_data_buf, len);decompress_data_with_multi_threads (compressed_data_buf, host, len);break ; +case RAM_SAVE_FLAG_XBZRLE : - host =host_from_stream_offset (f, addr, flags); -if (!host) { -error_report ("Illegal RAM offset " RAM_ADDR_FMT , addr); - ret = -EINVAL ; -break ; - }if (load_xbzrle (f, addr, host) <0 ) {error_report ("Failed to decompress XBZRLE page at " RAM_ADDR_FMT , addr);
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 12/56] ram_load: Factor out host_from_stream_offset call and check
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
The main RAM load loop has a call to
host_from_stream_offsetfor each page type that actually loads data with the same test; factor it out before the switch.
Quoted 6 lines
The host = NULL is to silence a bogus gcc warning of
an unitialised in theRAM_SAVE_COMPRESS_PAGEcase, it doesn't seem to realise that host is always initialised by the if at the top in the cases the switch takes.Signed-off-by Dr. David Alan Gilbert <[email protected]>
PATCH v9 13/56 migrate_init: Call from savevm
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Suspend to file is very much like a migrate, and it makes life easier if we have the Migration state available, so initialise it in the savevm.c code for suspending.
--- include/migration/migration.h | 4 +--- include/qemu/typedefs.h | 1 + migration/migration.c | 2 +- migration/savevm.c | 2 ++ 4 files changed, 5 insertions(+), 4 deletions(-)
include/migration/migration.h +1 -3
@@ -42,10 +42,7 @@ struct MigrationParams {bool shared; }; -typedef struct MigrationState MigrationState ; - typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head; -/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *from_src_file;
@@ -116,6 +113,7 @@ int migrate_fd_close(MigrationState *s);void add_migration_state_change_notifier (Notifier *notify);void remove_migration_state_change_notifier (Notifier *notify); +MigrationState *migrate_init (const MigrationParams *params);bool migration_in_setup (MigrationState *);bool migration_has_finished (MigrationState *);bool migration_has_failed (MigrationState *);
include/qemu/typedefs.h +1 -0
@@ -43,6 +43,7 @@ typedef struct MemoryRegion MemoryRegion;typedef struct MemoryRegionSection MemoryRegionSection ;typedef struct MigrationIncomingState MigrationIncomingState ;typedef struct MigrationParams MigrationParams ; +typedef struct MigrationState MigrationState ;typedef struct Monitor Monitor ;typedef struct MouseTransformInfo MouseTransformInfo ;typedef struct MSIMessage MSIMessage ;
migration/migration.c +1 -1
@@ -682,7 +682,7 @@ bool migration_has_failed(MigrationState *s) s->state == MIGRATION_STATUS_FAILED); } -static MigrationState *migrate_init (const MigrationParams *params) +MigrationState *migrate_init (const MigrationParams *params) {MigrationState *s =migrate_get_current ();int64_t bandwidth_limit = s->bandwidth_limit;
migration/savevm.c +2 -0
@@ -922,6 +922,8 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) .blk = 0, .shared = 0 }; +MigrationState *ms =migrate_init (¶ms); + ms->file = f;if (qemu_savevm_state_blocked (errp)) {return -EINVAL ;
Signature · 1 line
2.5.0
PATCH v9 14/56 Rename save_live_complete to save_live_complete_precopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
In postcopy we're going to need to perform the complete phase for postcopiable devices at a different point, start out by renaming all of the 'complete's to make the difference obvious.
--- hw/ppc/spapr.c | 2 +- include/migration/vmstate.h | 2 +- include/sysemu/sysemu.h | 2 +- migration/block.c | 2 +- migration/migration.c | 2 +- migration/ram.c | 2 +- migration/savevm.c | 10 +++++----- trace-events | 2 +- 8 files changed, 12 insertions(+), 12 deletions(-)
hw/ppc/spapr.c +1 -1
@@ -1588,7 +1588,7 @@ static int htab_load(QEMUFile *f, void *opaque, int version_id)static SaveVMHandlers savevm_htab_handlers = { .save_live_setup = htab_save_setup, .save_live_iterate = htab_save_iterate, - .save_live_complete = htab_save_complete, + .save_live_complete_precopy = htab_save_complete, .load_state = htab_load, };
include/migration/vmstate.h +1 -1
@@ -40,7 +40,7 @@ typedef struct SaveVMHandlers {SaveStateHandler *save_state;void (*cleanup)(void *opaque); -int (*save_live_complete)(QEMUFile *f,void *opaque); +int (*save_live_complete_precopy)(QEMUFile *f,void *opaque);/* This runs both outside and inside the iothread lock. */ bool (*is_active)(void *opaque);
include/sysemu/sysemu.h +1 -1
@@ -89,8 +89,8 @@ void qemu_savevm_state_begin(QEMUFile *f, const MigrationParams *params);void qemu_savevm_state_header (QEMUFile *f);int qemu_savevm_state_iterate (QEMUFile *f); -void qemu_savevm_state_complete (QEMUFile *f);void qemu_savevm_state_cleanup (void ); +void qemu_savevm_state_complete_precopy (QEMUFile *f);uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size);int qemu_loadvm_state (QEMUFile *f);
migration/block.c +1 -1
@@ -876,7 +876,7 @@ static SaveVMHandlers savevm_block_handlers = { .set_params = block_set_params, .save_live_setup = block_save_setup, .save_live_iterate = block_save_iterate, - .save_live_complete = block_save_complete, + .save_live_complete_precopy = block_save_complete, .save_live_pending = block_save_pending, .load_state = block_load, .cleanup = block_migration_cleanup,
migration/migration.c +1 -1
@@ -994,7 +994,7 @@ static void migration_completion(MigrationState *s, bool *old_vm_running, ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE );if (ret >=0 ) {qemu_file_set_rate_limit (s->file,INT64_MAX ); -qemu_savevm_state_complete (s->file); +qemu_savevm_state_complete_precopy (s->file); } }qemu_mutex_unlock_iothread ();
migration/ram.c +1 -1
@@ -1700,7 +1700,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)static SaveVMHandlers savevm_ram_handlers = { .save_live_setup = ram_save_setup, .save_live_iterate = ram_save_iterate, - .save_live_complete = ram_save_complete, + .save_live_complete_precopy = ram_save_complete, .save_live_pending = ram_save_pending, .load_state = ram_load, .cleanup = ram_migration_cleanup,
migration/savevm.c +5 -5
@@ -807,19 +807,19 @@ static bool should_send_vmdesc(void)return !machine->suppress_vmdesc; } -void qemu_savevm_state_complete (QEMUFile *f) +void qemu_savevm_state_complete_precopy (QEMUFile *f) {QJSON *vmdesc;int vmdesc_len;SaveStateEntry *se;int ret; -trace_savevm_state_complete (); +trace_savevm_state_complete_precopy ();cpu_synchronize_all_states (); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { -if (!se->ops || !se->ops->save_live_complete) { +if (!se->ops || !se->ops->save_live_complete_precopy) {continue ; }if (se->ops && se->ops->is_active) {
@@ -831,7 +831,7 @@ void qemu_savevm_state_complete(QEMUFile *f)save_section_header (f, se,QEMU_VM_SECTION_END ); - ret = se->ops->save_live_complete (f, se->opaque); + ret = se->ops->save_live_complete_precopy (f, se->opaque);trace_savevm_section_end (se->idstr, se->section_id, ret);save_section_footer (f, se);if (ret <0 ) {
@@ -942,7 +942,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) ret =qemu_file_get_error (f);if (ret ==0 ) { -qemu_savevm_state_complete (f); +qemu_savevm_state_complete_precopy (f); ret =qemu_file_get_error (f); }if (ret !=0 ) {
trace-events +1 -1
@@ -1210,8 +1210,8 @@ savevm_section_skip(const char *id, unsigned int section_id) "%s, section_id %u" savevm_state_begin(void) "" savevm_state_header(void) "" savevm_state_iterate(void) "" -savevm_state_complete(void) "" savevm_state_cleanup(void) "" +savevm_state_complete_precopy(void) "" vmstate_save(const char *idstr, const char *vmsd_name) "%s, %s" vmstate_load(const char *idstr, const char *vmsd_name) "%s, %s" qemu_announce_self_iter(const char *mac) "%s"
Signature · 1 line
2.5.0
PATCH v9 15/56 Add Linux userfaultfd.h header
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy uses the userfaultfd.h feature in the Linux kernel; include the header.
(In early versions of the patch series we had this, and then we dropped this by only including it if the kernel headers defined the syscall number; however 1842bdfd added the syscall definition to our headers, which means we can't tell if the kernel has it or not)
--- linux-headers/linux/userfaultfd.h | 167 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100644 linux-headers/linux/userfaultfd.h
linux-headers/linux/userfaultfd.h new file +167 -0
@@ -0,0 +1,167 @@ +/* +* include/linux/userfaultfd.h +* +* Copyright (C) 2007 Davide Libenzi <[email protected]> +* Copyright (C) 2015 Red Hat, Inc. +* +*/ + +#ifndef _LINUX_USERFAULTFD_H +#define _LINUX_USERFAULTFD_H + +#include <linux/types.h> + +#define UFFD_API ((__u64)0xAA) +/* +* After implementing the respective features it will become: +* #define UFFD_API_FEATURES (UFFD_FEATURE_PAGEFAULT_FLAG_WP | \ +* UFFD_FEATURE_EVENT_FORK) +*/ +#define UFFD_API_FEATURES (0) +#define UFFD_API_IOCTLS \ + ((__u64)1 << _UFFDIO_REGISTER | \ + (__u64)1 << _UFFDIO_UNREGISTER | \ + (__u64)1 << _UFFDIO_API) +#define UFFD_API_RANGE_IOCTLS \ + ((__u64)1 << _UFFDIO_WAKE | \ + (__u64)1 << _UFFDIO_COPY | \ + (__u64)1 << _UFFDIO_ZEROPAGE) + +/* +* Valid ioctl command number range with this API is from 0x00 to +* 0x3F. UFFDIO_API is the fixed number, everything else can be +* changed by implementing a different UFFD_API. If sticking to the +* same UFFD_API more ioctl can be added and userland will be aware of +* which ioctl the running kernel implements through the ioctl command +* bitmask written by the UFFDIO_API. +*/ +#define _UFFDIO_REGISTER (0x00) +#define _UFFDIO_UNREGISTER (0x01) +#define _UFFDIO_WAKE (0x02) +#define _UFFDIO_COPY (0x03) +#define _UFFDIO_ZEROPAGE (0x04) +#define _UFFDIO_API (0x3F) + +/* userfaultfd ioctl ids */ +#define UFFDIO 0xAA +#define UFFDIO_API _IOWR(UFFDIO, _UFFDIO_API, \ + struct uffdio_api) +#define UFFDIO_REGISTER _IOWR(UFFDIO, _UFFDIO_REGISTER, \ + struct uffdio_register) +#define UFFDIO_UNREGISTER _IOR(UFFDIO, _UFFDIO_UNREGISTER, \ + struct uffdio_range) +#define UFFDIO_WAKE _IOR(UFFDIO, _UFFDIO_WAKE, \ + struct uffdio_range) +#define UFFDIO_COPY _IOWR(UFFDIO, _UFFDIO_COPY, \ + struct uffdio_copy) +#define UFFDIO_ZEROPAGE _IOWR(UFFDIO, _UFFDIO_ZEROPAGE, \ + struct uffdio_zeropage) + +/* read() structure */ +struct uffd_msg { +__u8 event; + +__u8 reserved1; +__u16 reserved2; +__u32 reserved3; + +union { +struct { +__u64 flags; +__u64 address; + } pagefault; + +struct { +/* unused reserved fields */ +__u64 reserved1; +__u64 reserved2; +__u64 reserved3; + } reserved; + } arg; +} __packed; + +/* +* Start at 0x12 and not at 0 to be more strict against bugs. +*/ +#define UFFD_EVENT_PAGEFAULT 0x12 +#if 0 /* not available yet */ +#define UFFD_EVENT_FORK 0x13 +#endif + +/* flags for UFFD_EVENT_PAGEFAULT */ +#define UFFD_PAGEFAULT_FLAG_WRITE (1<<0)/* If this was a write fault */ +#define UFFD_PAGEFAULT_FLAG_WP (1<<1)/* If reason is VM_UFFD_WP */ + +struct uffdio_api { +/* userland asks for an API number and the features to enable */ +__u64 api; +/* +* Kernel answers below with the all available features for +* the API, this notifies userland of which events and/or +* which flags for each event are enabled in the current +* kernel. +* +* Note: UFFD_EVENT_PAGEFAULT and UFFD_PAGEFAULT_FLAG_WRITE +* are to be considered implicitly always enabled in all kernels as +* long as the uffdio_api.api requested matches UFFD_API. +*/ +#if 0 /* not available yet */ +#define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) +#define UFFD_FEATURE_EVENT_FORK (1<<1) +#endif +__u64 features; + +__u64 ioctls; +}; + +struct uffdio_range { +__u64 start; +__u64 len; +}; + +struct uffdio_register { +struct uffdio_range range; +#define UFFDIO_REGISTER_MODE_MISSING ((__u64)1<<0) +#define UFFDIO_REGISTER_MODE_WP ((__u64)1<<1) +__u64 mode; + +/* +* kernel answers which ioctl commands are available for the +* range, keep at the end as the last 8 bytes aren't read. +*/ +__u64 ioctls; +}; + +struct uffdio_copy { +__u64 dst; +__u64 src; +__u64 len; +/* +* There will be a wrprotection flag later that allows to map +* pages wrprotected on the fly. And such a flag will be +* available if the wrprotection ioctl are implemented for the +* range according to the uffdio_register.ioctls. +*/ +#define UFFDIO_COPY_MODE_DONTWAKE ((__u64)1<<0) +__u64 mode; + +/* +* "copy" is written by the ioctl and must be at the end: the +* copy_from_user will not read the last 8 bytes. +*/ +__s64 copy; +}; + +struct uffdio_zeropage { +struct uffdio_range range; +#define UFFDIO_ZEROPAGE_MODE_DONTWAKE ((__u64)1<<0) +__u64 mode; + +/* +* "zeropage" is written by the ioctl and must be at the end: +* the copy_from_user will not read the last 8 bytes. +*/ +__s64 zeropage; +}; + +#endif /* _LINUX_USERFAULTFD_H */
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 15/56] Add Linux userfaultfd.h header
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy uses the
userfaultfd.hfeature in the Linux kernel; include the header.(In early versions of the patch series we had this, and then we dropped this by only including it if the kernel headers defined the syscall number; however
1842bdfdadded the syscall definition to our headers, which means we can't tell if the kernel has it or not)Signed-off-by Dr. David Alan Gilbert <[email protected]>
PATCH v9 16/56 Return path: Open a return path on QEMUFile for sockets
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'.
Wire it up for 'socket' QEMUFile's.
--- include/migration/qemu-file.h | 7 +++++ migration/qemu-file-unix.c | 69 +++++++++++++++++++++++++++++++++++++------ migration/qemu-file.c | 12 ++++++++ 3 files changed, 79 insertions(+), 9 deletions(-)
include/migration/qemu-file.h +7 -0
@@ -89,6 +89,11 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque, uint64_t *bytes_sent);/* +* Return a QEMUFile for comms in the opposite direction +*/ +typedef QEMUFile *(QEMURetPathFunc )(void *opaque); + +/* * Stop any read or write (depending on flags) on the underlying * transport on the QEMUFile. * Existing blocking reads/writes must be woken
@@ -106,6 +111,7 @@ typedef struct QEMUFileOps {QEMURamHookFunc *after_ram_iterate;QEMURamHookFunc *hook_ram_load;QEMURamSaveFunc *save_page; +QEMURetPathFunc *get_return_path;QEMUFileShutdownFunc *shut_down; } QEMUFileOps;
@@ -196,6 +202,7 @@ int64_t qemu_file_get_rate_limit(QEMUFile *f);int qemu_file_get_error (QEMUFile *f);void qemu_file_set_error (QEMUFile *f,int ret);int qemu_file_shutdown (QEMUFile *f); +QEMUFile *qemu_file_get_return_path (QEMUFile *f);void qemu_fflush (QEMUFile *f);void qemu_file_set_blocking (QEMUFile *f,bool block);
migration/qemu-file-unix.c +60 -9
84 lines not on this page · raw message
migration/qemu-file.c +12 -0
@@ -44,6 +44,18 @@ int qemu_file_shutdown(QEMUFile *f)return f->ops->shut_down (f->opaque, true, true); } +/* +* Result: QEMUFile* for a 'return path' for comms in the opposite direction +* NULL if not available +*/ +QEMUFile *qemu_file_get_return_path (QEMUFile *f) +{ +if (!f->ops->get_return_path) { +return NULL ; + } +return f->ops->get_return_path (f->opaque); +} +bool qemu_file_mode_is_not_valid (const char *mode) { if (mode == NULL ||
Signature · 1 line
2.5.0
PATCH v9 17/56 Return path: socket_writev_buffer: Block even on non-blocking fd's
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The destination sets the fd to non-blocking on incoming migrations; this also affects the return path from the destination, and thus we need to make sure we can safely write to the return path.
--- migration/qemu-file-unix.c | 42 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 37 insertions(+), 5 deletions(-)
migration/qemu-file-unix.c +37 -5
@@ -22,6 +22,7 @@* THE SOFTWARE. */ #include "qemu-common.h" +#include "qemu/error-report.h" #include "qemu/iov.h" #include "qemu/sockets.h" #include "qemu/coroutine.h"
@@ -39,12 +40,43 @@ static ssize_t socket_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,QEMUFileSocket *s = opaque;ssize_t len;ssize_t size =iov_size (iov, iovcnt); +ssize_t offset =0 ; +int err; - len =iov_send (s->fd, iov, iovcnt,0 , size); -if (len < size) { - len = -socket_error (); - } -return len; +while (size >0 ) { + len =iov_send (s->fd, iov, iovcnt, offset, size); + +if (len >0 ) { + size -= len; + offset += len; + } + +if (size >0 ) { + err =socket_error (); + +if (err !=EAGAIN && err !=EWOULDBLOCK ) { +error_report ("socket_writev_buffer: Got err=%d for (%zd/%zd)" , + err, size, len); +/* +* If I've already sent some but only just got the error, I +* could return the amount validly sent so far and wait for the +* next call to report the error, but I'd rather flag the error +* immediately. +*/ +return -err; + } + +/* Emulate blocking */ +GPollFD pfd; + + pfd.fd = s->fd; + pfd.events =G_IO_OUT |G_IO_ERR ; + pfd.revents =0 ; +g_poll (&pfd,1 /* 1 fd */ ,-1 /* no timeout */ ); + } + } + +return offset; }static int socket_get_fd (void *opaque)
Signature · 1 line
2.5.0
PATCH v9 18/56 Migration commands
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process.
For use in postcopy.
--- include/migration/migration.h | 1 + include/sysemu/sysemu.h | 8 +++++ migration/savevm.c | 70 +++++++++++++++++++++++++++++++++++++++++++ trace-events | 2 ++ 4 files changed, 81 insertions(+)
include/migration/migration.h +1 -0
@@ -35,6 +35,7 @@#define QEMU_VM_SUBSECTION 0x05#define QEMU_VM_VMDESCRIPTION 0x06#define QEMU_VM_CONFIGURATION 0x07 +#define QEMU_VM_COMMAND 0x08#define QEMU_VM_SECTION_FOOTER 0x7estruct MigrationParams {
include/sysemu/sysemu.h +8 -0
@@ -84,6 +84,12 @@ void hmp_info_snapshots(Monitor *mon, const QDict *qdict);void qemu_announce_self (void ); +/* Subcommands for QEMU_VM_COMMAND */ +enum qemu_vm_cmd { +MIG_CMD_INVALID =0 ,/* Must be 0 */ +MIG_CMD_MAX +}; +bool qemu_savevm_state_blocked (Error **errp);void qemu_savevm_state_begin (QEMUFile *f,const MigrationParams *params);
@@ -92,6 +98,8 @@ int qemu_savevm_state_iterate(QEMUFile *f);void qemu_savevm_state_cleanup (void );void qemu_savevm_state_complete_precopy (QEMUFile *f);uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size); +void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command, +uint16_t len,uint8_t *data);int qemu_loadvm_state (QEMUFile *f); typedef enum DisplayType
migration/savevm.c +70 -0
94 lines not on this page · raw message
trace-events +2 -0
@@ -1204,6 +1204,8 @@ virtio_gpu_fence_resp(uint64_t fence) "fence 0x%" PRIx64 qemu_loadvm_state_section(unsigned int section_type) "%d" qemu_loadvm_state_section_partend(uint32_t section_id) "%u" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" +loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" +savevm_command_send(uint16_t command, uint16_t len) "com=0x%x len=%d" savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u" savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d" savevm_section_skip(const char *id, unsigned int section_id) "%s, section_id %u"
Signature · 1 line
2.5.0
PATCH v9 19/56 Return path: Control commands
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add two src->dest commands:
* OPEN_RETURN_PATH - To request that the destination open the return path
* PING - Request an acknowledge from the destination
--- include/migration/migration.h | 2 ++ include/sysemu/sysemu.h | 4 ++++ migration/savevm.c | 43 ++++++++++++++++++++++++++++++++++++++++++- trace-events | 3 +++ 4 files changed, 51 insertions(+), 1 deletion(-)
include/migration/migration.h +2 -0
@@ -48,6 +48,8 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;struct MigrationIncomingState {QEMUFile *from_src_file; +QEMUFile *to_src_file; +/* See savevm.c */ LoadStateEntry_Head loadvm_handlers; };
include/sysemu/sysemu.h +4 -0
@@ -87,6 +87,8 @@ void qemu_announce_self(void);/* Subcommands for QEMU_VM_COMMAND */ enum qemu_vm_cmd {MIG_CMD_INVALID =0 ,/* Must be 0 */ +MIG_CMD_OPEN_RETURN_PATH ,/* Tell the dest to open the Return path */ +MIG_CMD_PING ,/* Request a PONG on the RP */ MIG_CMD_MAX };
@@ -100,6 +102,8 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f);uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size);void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command,uint16_t len,uint8_t *data); +void qemu_savevm_send_ping (QEMUFile *f,uint32_t value); +void qemu_savevm_send_open_return_path (QEMUFile *f);int qemu_loadvm_state (QEMUFile *f); typedef enum DisplayType
migration/savevm.c +42 -1
69 lines not on this page · raw message
trace-events +3 -0
@@ -1205,10 +1205,13 @@ qemu_loadvm_state_section(unsigned int section_type) "%d" qemu_loadvm_state_section_partend(uint32_t section_id) "%u" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" +loadvm_process_command_ping(uint32_t val) "%x" savevm_command_send(uint16_t command, uint16_t len) "com=0x%x len=%d" savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u" savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d" savevm_section_skip(const char *id, unsigned int section_id) "%s, section_id %u" +savevm_send_open_return_path(void) "" +savevm_send_ping(uint32_t val) "%x" savevm_state_begin(void) "" savevm_state_header(void) "" savevm_state_iterate(void) ""
Signature · 1 line
2.5.0
PATCH v9 20/56 Return path: Send responses from destination to source
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add migrate_send_rp_message to send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Add migrate_send_rp_shut to send a 'shut' message to indicate
the destination is finished with the RP.
Add migrate_send_rp_ack to send a 'PONG' message in response to a PING
Use it in the MSG_RP_PING handler
--- include/migration/migration.h | 19 ++++++++++++++++++ migration/migration.c | 45 +++++++++++++++++++++++++++++++++++++++++++ migration/savevm.c | 2 +- trace-events | 1 + 4 files changed, 66 insertions(+), 1 deletion(-)
include/migration/migration.h +19 -0
@@ -43,12 +43,22 @@ struct MigrationParams {bool shared; }; +/* Messages sent on the return path from destination to source */ +enum mig_rp_message_type { +MIG_RP_MSG_INVALID =0 ,/* Must be 0 */ +MIG_RP_MSG_SHUT ,/* sibling will not send any more RP messages */ +MIG_RP_MSG_PONG ,/* Response to a PING; data (seq: be32 ) */ + +MIG_RP_MSG_MAX +}; + typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *from_src_file;QEMUFile *to_src_file; +QemuMutex rp_mutex;/* We send replies from multiple threads */ /* See savevm.c */ LoadStateEntry_Head loadvm_handlers;
@@ -181,6 +191,15 @@ int migrate_compress_threads(void);int migrate_decompress_threads (void );bool migrate_use_events (void ); +/* Sending on the return path - generic and then for each message type */ +void migrate_send_rp_message (MigrationIncomingState *mis, +enum mig_rp_message_type message_type, +uint16_t len,void *data); +void migrate_send_rp_shut (MigrationIncomingState *mis, +uint32_t value); +void migrate_send_rp_pong (MigrationIncomingState *mis, +uint32_t value); +void ram_control_before_iterate (QEMUFile *f,uint64_t flags);void ram_control_after_iterate (QEMUFile *f,uint64_t flags);void ram_control_load_hook (QEMUFile *f,uint64_t flags,void *data);
migration/migration.c +45 -0
57 lines not on this page · raw message
migration/savevm.c +1 -1
@@ -1102,7 +1102,7 @@ static int loadvm_process_command(QEMUFile *f) tmp32);return -1 ; } -/* migrate_send_rp_pong(mis, tmp32); TODO: gets added later */ +migrate_send_rp_pong (mis, tmp32);break ; }
trace-events +1 -0
@@ -1431,6 +1431,7 @@ migrate_fd_cleanup(void) "" migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64 +migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64 migrate_state_too_big(void) "" migrate_global_state_post_load(const char *state) "loaded state: %s"
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 20/56] Return path: Send responses from destination to source
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add
migrate_send_rp_messageto send a message from destination to source along the return path.
(It uses a mutex to let it be called from multiple threads)
Addmigrate_send_rp_shutto send a 'shut' message to indicate
the destination is finished with the RP.
Addmigrate_send_rp_ackto send a 'PONG' message in response to a PING
Use it in theMSG_RP_PINGhandlerSigned-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Amit Shah <[email protected]>
PATCH v9 21/56 migration_is_setup_or_active
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add 'migration_is_setup_or_active' utility function to check state. (It gets postcopy added to it's list later on in the series)
--- migration/migration.c | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)
migration/migration.c +20 -6
@@ -444,6 +444,23 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)return params; } +/* +* Return true if we're already in the middle of a migration +* (i.e. any of the active or setup states) +*/ +static bool migration_is_setup_or_active (int state) +{ +switch (state) { +case MIGRATION_STATUS_ACTIVE : +case MIGRATION_STATUS_SETUP : +return true; + +default : +return false; + + } +} +static void get_xbzrle_cache_stats (MigrationInfo *info) {if (migrate_use_xbzrle ()) {
@@ -551,8 +568,7 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,MigrationState *s =migrate_get_current ();MigrationCapabilityStatusList *cap; -if (s->state ==MIGRATION_STATUS_ACTIVE || - s->state ==MIGRATION_STATUS_SETUP ) { +if (migration_is_setup_or_active (s->state)) {error_setg (errp,QERR_MIGRATION_ACTIVE );return ; }
@@ -682,8 +698,7 @@ static void migrate_fd_cancel(MigrationState *s) do { old_state = s->state; - if (old_state != MIGRATION_STATUS_SETUP && - old_state != MIGRATION_STATUS_ACTIVE) { + if (!migration_is_setup_or_active(old_state)) { break; } migrate_set_state(s, old_state, MIGRATION_STATUS_CANCELLING);
@@ -815,8 +830,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, params.blk = has_blk && blk; params.shared = has_inc && inc; -if (s->state ==MIGRATION_STATUS_ACTIVE || - s->state ==MIGRATION_STATUS_SETUP || +if (migration_is_setup_or_active (s->state) || s->state ==MIGRATION_STATUS_CANCELLING ) {error_setg (errp,QERR_MIGRATION_ACTIVE );return ;
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 21/56] migration_is_setup_or_active
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add '
migration_is_setup_or_active' utility function to check state. (It gets postcopy added to it's list later on in the series)Signed-off-by Dr. David Alan Gilbert <[email protected]>
PATCH v9 22/56 Return path: Source handling of return path
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Open a return path, and handle messages that are received upon it.
--- include/migration/migration.h | 8 +++ migration/migration.c | 159 +++++++++++++++++++++++++++++++++++++++++- trace-events | 10 +++ 3 files changed, 175 insertions(+), 2 deletions(-)
include/migration/migration.h +8 -0
@@ -80,6 +80,14 @@ struct MigrationStateint state;MigrationParams params; + +/* State related to return path */ +struct { +QEMUFile *from_dst_file; +QemuThread rp_thread; +bool error; + } rp_state; +double mbps;int64_t total_time;int64_t downtime;
migration/migration.c +157 -2
177 lines not on this page · raw message
trace-events +10 -0
@@ -1426,12 +1426,22 @@ flic_no_device_api(int err) "flic: no Device Contral API support %d" flic_reset_failed(int err) "flic: reset failed %d" # migration.c +await_return_path_close_on_source_close(void) "" +await_return_path_close_on_source_joining(void) "" migrate_set_state(int new_state) "new state %d" migrate_fd_cleanup(void) "" migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64 migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" +open_return_path_on_source(void) "" +open_return_path_on_source_continue(void) "" +source_return_path_thread_bad_end(void) "" +source_return_path_thread_end(void) "" +source_return_path_thread_entry(void) "" +source_return_path_thread_loop_top(void) "" +source_return_path_thread_pong(uint32_t val) "%x" +source_return_path_thread_shut(uint32_t val) "%x" migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64 migrate_state_too_big(void) "" migrate_global_state_post_load(const char *state) "loaded state: %s"
Signature · 1 line
2.5.0
PATCH v9 23/56 Rework loadvm path for subloops
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy needs to have two migration streams loading concurrently; one from memory (with the device state) and the other from the fd with the memory transactions.
Split the core of qemu_loadvm_state out so we can use it for both.
Allow the inner loadvm loop to quit and cause the parent loops to exit as well.
--- include/migration/migration.h | 6 ++ migration/migration.c | 2 + migration/savevm.c | 141 ++++++++++++++++++++++-------------------- trace-events | 4 ++ 4 files changed, 86 insertions(+), 67 deletions(-)
include/migration/migration.h +6 -0
@@ -57,6 +57,12 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;struct MigrationIncomingState {QEMUFile *from_src_file; +/* +* Free at the start of the main state load, set as the main thread finishes +* loading state. +*/ +QemuEvent main_thread_load_event; +QEMUFile *to_src_file;QemuMutex rp_mutex;/* We send replies from multiple threads */
migration/migration.c +2 -0
@@ -98,12 +98,14 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f) mis_current->from_src_file = f;QLIST_INIT (&mis_current->loadvm_handlers);qemu_mutex_init (&mis_current->rp_mutex); +qemu_event_init (&mis_current->main_thread_load_event, false);return mis_current; }void migration_incoming_state_destroy (void ) { +qemu_event_destroy (&mis_current->main_thread_load_event);loadvm_free_handlers (mis_current);g_free (mis_current); mis_current =NULL ;
migration/savevm.c +74 -67
218 lines not on this page · raw message
trace-events +4 -0
@@ -1202,7 +1202,11 @@ virtio_gpu_fence_resp(uint64_t fence) "fence 0x%" PRIx64 # migration/savevm.c qemu_loadvm_state_section(unsigned int section_type) "%d" +qemu_loadvm_state_section_command(int ret) "%d" qemu_loadvm_state_section_partend(uint32_t section_id) "%u" +qemu_loadvm_state_main(void) "" +qemu_loadvm_state_main_quit_parent(void) "" +qemu_loadvm_state_post_main(int ret) "%d" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" loadvm_process_command_ping(uint32_t val) "%x"
Signature · 1 line
2.5.0
PATCH v9 24/56 Add migration-capability boolean for postcopy-ram.
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The 'postcopy ram' capability allows postcopy migration of RAM; note that the migration starts off in precopy mode until postcopy mode is triggered (see the migrate_start_postcopy patch later in the series).
--- include/migration/migration.h | 1 + migration/migration.c | 23 +++++++++++++++++++++++ qapi-schema.json | 6 +++++- 3 files changed, 29 insertions(+), 1 deletion(-)
include/migration/migration.h +1 -0
@@ -186,6 +186,7 @@ void migrate_add_blocker(Error *reason);*/ void migrate_del_blocker (Error *reason); +bool migrate_postcopy_ram (void );bool migrate_zero_blocks (void );bool migrate_auto_converge (void );
migration/migration.c +23 -0
@@ -578,6 +578,20 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,for (cap = params; cap; cap = cap->next) { s->enabled_capabilities[cap->value->capability] = cap->value->state; } + +if (migrate_postcopy_ram ()) { +if (migrate_use_compression ()) { +/* The decompression threads asynchronously write into RAM +* rather than use the atomic copies needed to avoid +* userfaulting. It should be possible to fix the decompression +* threads for compatibility in future. +*/ +error_report ("Postcopy is not currently compatible with " +"compression" ); + s->enabled_capabilities[MIGRATION_CAPABILITY_X_POSTCOPY_RAM ] = + false; + } + } } void qmp_migrate_set_parameters(bool has_compress_level,
@@ -956,6 +970,15 @@ void qmp_migrate_set_downtime(double value, Error **errp) max_downtime = (uint64_t )value; } +bool migrate_postcopy_ram (void ) +{ +MigrationState *s; + + s =migrate_get_current (); + +return s->enabled_capabilities[MIGRATION_CAPABILITY_X_POSTCOPY_RAM ]; +} +bool migrate_auto_converge (void ) {MigrationState *s;
qapi-schema.json +5 -1
@@ -540,11 +540,15 @@ # @auto-converge: If enabled, QEMU will automatically throttle down the guest # to speed up convergence of RAM migration. (since 1.6) # +# @x-postcopy-ram: Start executing on the migration target before all of RAM has +# been migrated, pulling the remaining pages along as needed. NOTE: If +# the migration fails during postcopy the VM will fail. (since 2.5) +# # Since: 1.2 ## { 'enum': 'MigrationCapability', 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', - 'compress', 'events'] } + 'compress', 'events', 'x-postcopy-ram'] } ## # @MigrationCapabilityStatus
Signature · 1 line
2.5.0
PATCH v9 25/56 Add wrappers and handlers for sending/receiving the postcopy-ram migration messages.
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The state of the postcopy process is managed via a series of messages;
* Add wrappers and handlers for sending/receiving these messages
* Add state variable that track the current state of postcopy
--- include/migration/migration.h | 27 +++++ include/sysemu/sysemu.h | 19 ++++ migration/migration.c | 20 ++++ migration/savevm.c | 254 ++++++++++++++++++++++++++++++++++++++++++ trace-events | 10 ++ 5 files changed, 330 insertions(+)
include/migration/migration.h +27 -0
@@ -53,6 +53,29 @@ enum mig_rp_message_type { }; typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head; + +/* The current postcopy state is read/set by postcopy_state_get/set +* which update it atomically. +* The state is updated as postcopy messages are received, and +* in general only one thread should be writing to the state at any one +* time, initially the main thread and then the listen thread; +* Corner cases are where either thread finishes early and/or errors. +* The state is checked as messages are received to ensure that +* the source is sending us messages in the correct order. +* The state is also used by the RAM reception code to know if it +* has to place pages atomically, and the cleanup code at the end of +* the main thread to know if it has to delay cleanup until the end +* of postcopy. +*/ +typedef enum { +POSTCOPY_INCOMING_NONE =0 ,/* Initial state - no postcopy */ +POSTCOPY_INCOMING_ADVISE , +POSTCOPY_INCOMING_DISCARD , +POSTCOPY_INCOMING_LISTENING , +POSTCOPY_INCOMING_RUNNING , +POSTCOPY_INCOMING_END +}PostcopyState ; +/* State for the incoming migration */ struct MigrationIncomingState {QEMUFile *from_src_file;
@@ -240,4 +263,8 @@ void global_state_set_optional(void);void savevm_skip_configuration (void );int global_state_store (void );void global_state_store_running (void ); + +PostcopyState postcopy_state_get (void ); +/* Set the state and return the old state */ +PostcopyState postcopy_state_set (PostcopyState new_state);#endif
include/sysemu/sysemu.h +19 -0
@@ -89,6 +89,16 @@ enum qemu_vm_cmd {MIG_CMD_INVALID =0 ,/* Must be 0 */ MIG_CMD_OPEN_RETURN_PATH ,/* Tell the dest to open the Return path */ MIG_CMD_PING ,/* Request a PONG on the RP */ + +MIG_CMD_POSTCOPY_ADVISE ,/* Prior to any page transfers, just +warn we might want to do PC */ +MIG_CMD_POSTCOPY_LISTEN ,/* Start listening for incoming +pages as it's running. */ +MIG_CMD_POSTCOPY_RUN ,/* Start execution */ + +MIG_CMD_POSTCOPY_RAM_DISCARD ,/* A list of pages to discard that +were previously sent during +precopy but are dirty. */ MIG_CMD_MAX };
@@ -104,6 +114,15 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command, uint16_t len, uint8_t *data);void qemu_savevm_send_ping (QEMUFile *f,uint32_t value);void qemu_savevm_send_open_return_path (QEMUFile *f); +void qemu_savevm_send_postcopy_advise (QEMUFile *f); +void qemu_savevm_send_postcopy_listen (QEMUFile *f); +void qemu_savevm_send_postcopy_run (QEMUFile *f); + +void qemu_savevm_send_postcopy_ram_discard (QEMUFile *f,const char *name, +uint16_t len, +uint64_t *start_list, +uint64_t *length_list); +int qemu_loadvm_state (QEMUFile *f); typedef enum DisplayType
migration/migration.c +20 -0
@@ -57,6 +57,13 @@ static NotifierList migration_state_notifiers =static bool deferred_incoming; +/* +* Current state of incoming postcopy; note this is not part of +* MigrationIncomingState since it's state is used during cleanup +* at the end as MIS is being freed. +*/ +static PostcopyState incoming_postcopy_state; +/* When we add fault tolerance, we could have several migrations at once. For now we don't need to add dynamic creation of migration */
@@ -284,6 +291,7 @@ static void process_incoming_migration_co(void *opaque)int ret;migration_incoming_state_new (f); +postcopy_state_set (POSTCOPY_INCOMING_NONE );migrate_generate_event (MIGRATION_STATUS_ACTIVE ); ret =qemu_loadvm_state (f);
@@ -1367,3 +1375,15 @@ void migrate_fd_connect(MigrationState *s)qemu_thread_create (&s->thread,"migration" , migration_thread, s,QEMU_THREAD_JOINABLE ); } + +PostcopyState postcopy_state_get (void ) +{ +return atomic_mb_read (&incoming_postcopy_state); +} + +/* Set the state and return the old state */ +PostcopyState postcopy_state_set (PostcopyState new_state) +{ +return atomic_xchg (&incoming_postcopy_state, new_state); +} +
migration/savevm.c +254 -0
290 lines not on this page · raw message
trace-events +10 -0
@@ -1208,14 +1208,24 @@ qemu_loadvm_state_main(void) "" qemu_loadvm_state_main_quit_parent(void) "" qemu_loadvm_state_post_main(int ret) "%d" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" +loadvm_postcopy_handle_advise(void) "" +loadvm_postcopy_handle_listen(void) "" +loadvm_postcopy_handle_run(void) "" +loadvm_postcopy_ram_handle_discard(void) "" +loadvm_postcopy_ram_handle_discard_end(void) "" +loadvm_postcopy_ram_handle_discard_header(const char *ramid, uint16_t len) "%s: %ud" loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" loadvm_process_command_ping(uint32_t val) "%x" +qemu_savevm_send_postcopy_advise(void) "" +qemu_savevm_send_postcopy_ram_discard(const char *id, uint16_t len) "%s: %ud" savevm_command_send(uint16_t command, uint16_t len) "com=0x%x len=%d" savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u" savevm_section_end(const char *id, unsigned int section_id, int ret) "%s, section_id %u -> %d" savevm_section_skip(const char *id, unsigned int section_id) "%s, section_id %u" savevm_send_open_return_path(void) "" savevm_send_ping(uint32_t val) "%x" +savevm_send_postcopy_listen(void) "" +savevm_send_postcopy_run(void) "" savevm_state_begin(void) "" savevm_state_header(void) "" savevm_state_iterate(void) ""
Signature · 1 line
2.5.0
PATCH v9 26/56 MIG_CMD_PACKAGED: Send a packaged chunk of migration stream
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
MIG_CMD_PACKAGED is a migration command that wraps a chunk of migration stream inside a package whose length can be determined purely by reading its header. The destination guarantees that the whole MIG_CMD_PACKAGED is read off the stream prior to parsing the contents.
This is used by postcopy to load device state (from the package) while leaving the main stream free to receive memory pages.
--- include/sysemu/sysemu.h | 4 ++ migration/savevm.c | 104 +++++++++++++++++++++++++++++++++++++++++++++--- trace-events | 4 ++ 3 files changed, 107 insertions(+), 5 deletions(-)
include/sysemu/sysemu.h +4 -0
@@ -99,9 +99,12 @@ enum qemu_vm_cmd {MIG_CMD_POSTCOPY_RAM_DISCARD ,/* A list of pages to discard that were previously sent during precopy but are dirty. */ +MIG_CMD_PACKAGED ,/* Send a wrapped stream within this stream */ MIG_CMD_MAX }; +#define MAX_VM_CMD_PACKAGED_SIZE (1ul << 24) +bool qemu_savevm_state_blocked (Error **errp);void qemu_savevm_state_begin (QEMUFile *f,const MigrationParams *params);
@@ -114,6 +117,7 @@ void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command, uint16_t len, uint8_t *data);void qemu_savevm_send_ping (QEMUFile *f,uint32_t value);void qemu_savevm_send_open_return_path (QEMUFile *f); +int qemu_savevm_send_packaged (QEMUFile *f,const QEMUSizedBuffer *qsb);void qemu_savevm_send_postcopy_advise (QEMUFile *f);void qemu_savevm_send_postcopy_listen (QEMUFile *f);void qemu_savevm_send_postcopy_run (QEMUFile *f);
migration/savevm.c +99 -5
129 lines not on this page · raw message
trace-events +4 -0
@@ -1208,6 +1208,10 @@ qemu_loadvm_state_main(void) "" qemu_loadvm_state_main_quit_parent(void) "" qemu_loadvm_state_post_main(int ret) "%d" qemu_loadvm_state_section_startfull(uint32_t section_id, const char *idstr, uint32_t instance_id, uint32_t version_id) "%u(%s) %u %u" +qemu_savevm_send_packaged(void) "" +loadvm_handle_cmd_packaged(unsigned int length) "%u" +loadvm_handle_cmd_packaged_main(int ret) "%d" +loadvm_handle_cmd_packaged_received(int ret) "%d" loadvm_postcopy_handle_advise(void) "" loadvm_postcopy_handle_listen(void) "" loadvm_postcopy_handle_run(void) ""
Signature · 1 line
2.5.0
PATCH v9 27/56 Modify save_live_pending for postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Modify save_live_pending to return separate postcopiable and non-postcopiable counts.
--- include/migration/vmstate.h | 5 +++-- include/sysemu/sysemu.h | 4 +++- migration/block.c | 7 +++++-- migration/migration.c | 9 +++++++-- migration/ram.c | 8 ++++++-- migration/savevm.c | 17 +++++++++++++---- trace-events | 2 +- 7 files changed, 38 insertions(+), 14 deletions(-)
include/migration/vmstate.h +3 -2
@@ -54,8 +54,9 @@ typedef struct SaveVMHandlers {/* This runs outside the iothread lock! */ int (*save_live_setup)(QEMUFile *f,void *opaque); -uint64_t (*save_live_pending)(QEMUFile *f,void *opaque,uint64_t max_size); - +void (*save_live_pending)(QEMUFile *f,void *opaque,uint64_t max_size, +uint64_t *non_postcopiable_pending, +uint64_t *postcopiable_pending);LoadStateHandler *load_state; } SaveVMHandlers;
include/sysemu/sysemu.h +3 -1
@@ -112,7 +112,9 @@ void qemu_savevm_state_header(QEMUFile *f);int qemu_savevm_state_iterate (QEMUFile *f);void qemu_savevm_state_cleanup (void );void qemu_savevm_state_complete_precopy (QEMUFile *f); -uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size); +void qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size, +uint64_t *res_non_postcopiable, +uint64_t *res_postcopiable);void qemu_savevm_command_send (QEMUFile *f,enum qemu_vm_cmd command,uint16_t len,uint8_t *data);void qemu_savevm_send_ping (QEMUFile *f,uint32_t value);
migration/block.c +5 -2
@@ -748,7 +748,9 @@ static int block_save_complete(QEMUFile *f, void *opaque)return 0 ; } -static uint64_t block_save_pending (QEMUFile *f,void *opaque,uint64_t max_size) +static void block_save_pending (QEMUFile *f,void *opaque,uint64_t max_size, +uint64_t *non_postcopiable_pending, +uint64_t *postcopiable_pending) {/* Estimate pending number of bytes to send */ uint64_t pending;
@@ -767,7 +769,8 @@ static uint64_t block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)qemu_mutex_unlock_iothread ();DPRINTF ("Enter save live pending %" PRIu64"\n" , pending); -return pending; +/* We don't do postcopy */ + *non_postcopiable_pending += pending; }static int block_load (QEMUFile *f,void *opaque,int version_id)
migration/migration.c +7 -2
@@ -1288,8 +1288,13 @@ static void *migration_thread(void *opaque)uint64_t pending_size;if (!qemu_file_rate_limit (s->file)) { - pending_size =qemu_savevm_state_pending (s->file, max_size); -trace_migrate_pending (pending_size, max_size); +uint64_t pend_post, pend_nonpost; + +qemu_savevm_state_pending (s->file, max_size, &pend_nonpost, + &pend_post); + pending_size = pend_nonpost + pend_post; +trace_migrate_pending (pending_size, max_size, + pend_post, pend_nonpost);if (pending_size && pending_size >= max_size) {qemu_savevm_state_iterate (s->file); }else {
migration/ram.c +6 -2
@@ -1383,7 +1383,9 @@ static int ram_save_complete(QEMUFile *f, void *opaque)return 0 ; } -static uint64_t ram_save_pending (QEMUFile *f,void *opaque,uint64_t max_size) +static void ram_save_pending (QEMUFile *f,void *opaque,uint64_t max_size, +uint64_t *non_postcopiable_pending, +uint64_t *postcopiable_pending) {uint64_t remaining_size;
@@ -1397,7 +1399,9 @@ static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)qemu_mutex_unlock_iothread (); remaining_size =ram_save_remaining () *TARGET_PAGE_SIZE ; } -return remaining_size; + +/* We can do postcopy, and all the data is postcopiable */ + *postcopiable_pending += remaining_size; }static int load_xbzrle (QEMUFile *f,ram_addr_t addr,void *host)
migration/savevm.c +13 -4
@@ -1053,10 +1053,19 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f)qemu_fflush (f); } -uint64_t qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size) +/* Give an estimate of the amount left to be transferred, +* the result is split into the amount for units that can and +* for units that can't do postcopy. +*/ +void qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size, +uint64_t *res_non_postcopiable, +uint64_t *res_postcopiable) {SaveStateEntry *se; -uint64_t ret =0 ; + + *res_non_postcopiable =0 ; + *res_postcopiable =0 ; + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {if (!se->ops || !se->ops->save_live_pending) {
@@ -1067,9 +1076,9 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)continue ; } } - ret += se->ops->save_live_pending (f, se->opaque, max_size); + se->ops->save_live_pending (f, se->opaque, max_size, + res_non_postcopiable, res_postcopiable); } -return ret; }void qemu_savevm_state_cleanup (void )
trace-events +1 -1
@@ -1450,7 +1450,7 @@ migrate_set_state(int new_state) "new state %d" migrate_fd_cleanup(void) "" migrate_fd_error(void) "" migrate_fd_cancel(void) "" -migrate_pending(uint64_t size, uint64_t max) "pending size %" PRIu64 " max %" PRIu64 +migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")" migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" open_return_path_on_source(void) "" open_return_path_on_source_continue(void) ""
Signature · 1 line
2.5.0
PATCH v9 28/56 postcopy: OS support test
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Provide a check to see if the OS we're running on has all the bits needed for postcopy.
Creates postcopy-ram.c which will get most of the other helpers we need.
--- include/migration/postcopy-ram.h | 19 +++++ migration/Makefile.objs | 2 +- migration/postcopy-ram.c | 157 +++++++++++++++++++++++++++++++++++++++ migration/savevm.c | 5 ++ 4 files changed, 182 insertions(+), 1 deletion(-) create mode 100644 include/migration/postcopy-ram.h create mode 100644 migration/postcopy-ram.c
include/migration/postcopy-ram.h new file +19 -0
@@ -0,0 +1,19 @@ +/* +* Postcopy migration for RAM +* +* Copyright 2013 Red Hat, Inc. and/or its affiliates +* +* Authors: +* Dave Gilbert <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ +#ifndef QEMU_POSTCOPY_RAM_H +#define QEMU_POSTCOPY_RAM_H + +/* Return true if the host supports everything we need to do postcopy-ram */ +bool postcopy_ram_supported_by_host (void ); + +#endif
migration/Makefile.objs +1 -1
@@ -1,7 +1,7 @@ common-obj-y += migration.o tcp.o common-obj-y += vmstate.o common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o -common-obj-y += xbzrle.o +common-obj-y += xbzrle.o postcopy-ram.o common-obj-$(CONFIG_RDMA) += rdma.o common-obj-$(CONFIG_POSIX) += exec.o unix.o fd.o
migration/postcopy-ram.c new file +157 -0
157 lines not on this page · raw message
migration/savevm.c +5 -0
@@ -37,6 +37,7 @@#include "qemu/timer.h" #include "audio/audio.h" #include "migration/migration.h" +#include "migration/postcopy-ram.h" #include "qapi/qmp/qerror.h" #include "qemu/error-report.h" #include "qemu/sockets.h"
@@ -1205,6 +1206,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis)return -1 ; } +if (!postcopy_ram_supported_by_host ()) { +return -1 ; + } + remote_hps =qemu_get_be64 (mis->from_src_file);if (remote_hps !=getpagesize ()) {/*
Signature · 1 line
2.5.0
PATCH v9 29/56 migrate_start_postcopy: Command to trigger transition to postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Once postcopy is enabled (with migrate_set_capability), the migration will still start on precopy mode. To cause a transition into postcopy the:
migrate_start_postcopy
command must be issued. Postcopy will start sometime after this (when it's next checked in the migration loop).
Issuing the command before migration has started will error, and issuing after it has finished is ignored.
--- hmp-commands.hx | 15 +++++++++++++++ hmp.c | 7 +++++++ hmp.h | 1 + include/migration/migration.h | 3 +++ migration/migration.c | 22 ++++++++++++++++++++++ qapi-schema.json | 8 ++++++++ qmp-commands.hx | 19 +++++++++++++++++++ 7 files changed, 75 insertions(+)
hmp-commands.hx +15 -0
@@ -1008,6 +1008,21 @@ Set the parameter @var{parameter} for migration. ETEXI { + .name = "migrate_start_postcopy", + .args_type = "", + .params = "", + .help = "Switch migration to postcopy mode", + .mhandler.cmd = hmp_migrate_start_postcopy, + }, + +STEXI +@item migrate_start_postcopy +@findex migrate_start_postcopy +Switch in-progress migration to postcopy mode. Ignored after the end of +migration (or once already in postcopy). +ETEXI + + { .name = "client_migrate_info", .args_type = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?", .params = "protocol hostname port tls-port cert-subject",
hmp.c +7 -0
@@ -1293,6 +1293,13 @@ void hmp_client_migrate_info(Monitor *mon, const QDict *qdict)hmp_handle_error (mon, &err); } +void hmp_migrate_start_postcopy (Monitor *mon,const QDict *qdict) +{ +Error *err =NULL ; +qmp_migrate_start_postcopy (&err); +hmp_handle_error (mon, &err); +} +void hmp_set_password (Monitor *mon,const QDict *qdict) {const char *protocol =qdict_get_str (qdict,"protocol" );
hmp.h +1 -0
@@ -69,6 +69,7 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);void hmp_migrate_set_parameter (Monitor *mon,const QDict *qdict);void hmp_migrate_set_cache_size (Monitor *mon,const QDict *qdict);void hmp_client_migrate_info (Monitor *mon,const QDict *qdict); +void hmp_migrate_start_postcopy (Monitor *mon,const QDict *qdict);void hmp_set_password (Monitor *mon,const QDict *qdict);void hmp_expire_password (Monitor *mon,const QDict *qdict);void hmp_eject (Monitor *mon,const QDict *qdict);
include/migration/migration.h +3 -0
@@ -127,6 +127,9 @@ struct MigrationStateint64_t xbzrle_cache_size;int64_t setup_time;int64_t dirty_sync_count; + +/* Flag set once the migration has been asked to enter postcopy */ +bool start_postcopy; };void process_incoming_migration (QEMUFile *f);
migration/migration.c +22 -0
@@ -668,6 +668,28 @@ void qmp_migrate_set_parameters(bool has_compress_level, } } +void qmp_migrate_start_postcopy (Error **errp) +{ +MigrationState *s =migrate_get_current (); + +if (!migrate_postcopy_ram ()) { +error_setg (errp,"Enable postcopy with migration_set_capability before" +" the start of migration" ); +return ; + } + +if (s->state ==MIGRATION_STATUS_NONE ) { +error_setg (errp,"Postcopy must be started after migration has been" +" started" ); +return ; + } +/* +* we don't error if migration has finished since that would be racy +* with issuing this command. +*/ +atomic_set (&s->start_postcopy, true); +} +/* shared migration helpers */ static void migrate_set_state (MigrationState *s,int old_state,int new_state)
qapi-schema.json +8 -0
@@ -702,6 +702,14 @@ '*tls-port': 'int', '*cert-subject': 'str' } } ## +# @migrate-start-postcopy +# +# Switch migration to postcopy mode +# +# Since: 2.5 +{ 'command': 'migrate-start-postcopy' } + +## # @MouseInfo: # # Information about a mouse device.
qmp-commands.hx +19 -0
@@ -718,6 +718,25 @@ Example: EQMP { + .name = "migrate-start-postcopy", + .args_type = "", + .mhandler.cmd_new = qmp_marshal_migrate_start_postcopy, + }, + +SQMP +migrate-start-postcopy +---------------------- + +Switch an in-progress migration to postcopy mode. Ignored after the end of +migration (or once already in postcopy). + +Example: +-> { "execute": "migrate-start-postcopy" } +<- { "return": {} } + +EQMP + + { .name = "query-migrate-cache-size", .args_type = "", .mhandler.cmd_new = qmp_marshal_query_migrate_cache_size,
Signature · 1 line
2.5.0
PATCH v9 30/56 migration_completion: Take current state
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Soon we'll be in either ACTIVE or POSTCOPY_ACTIVE when we complete migration, and we need to know which we expect to be in to change state safely.
--- migration/migration.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
migration/migration.c +7 -4
@@ -1243,10 +1243,12 @@ static int await_return_path_close_on_source(MigrationState *ms)* The caller 'breaks' the loop when this returns. * * @s: Current migration state +* @current_active_state: The migration state we expect to be in * @*old_vm_running: Pointer to old_vm_running flag * @*start_time: Pointer to time to update */ -static void migration_completion (MigrationState *s,bool *old_vm_running, +static void migration_completion (MigrationState *s,int current_active_state, +bool *old_vm_running,int64_t *start_time) {int ret;
@@ -1275,11 +1277,11 @@ static void migration_completion(MigrationState *s, bool *old_vm_running, goto fail; } -migrate_set_state (s,MIGRATION_STATUS_ACTIVE ,MIGRATION_STATUS_COMPLETED ); +migrate_set_state (s, current_active_state,MIGRATION_STATUS_COMPLETED );return ; fail: -migrate_set_state (s,MIGRATION_STATUS_ACTIVE ,MIGRATION_STATUS_FAILED ); +migrate_set_state (s, current_active_state,MIGRATION_STATUS_FAILED ); }/*
@@ -1321,7 +1323,8 @@ static void *migration_thread(void *opaque)qemu_savevm_state_iterate (s->file); }else {trace_migration_thread_low_pending (pending_size); -migration_completion (s, &old_vm_running, &start_time); +migration_completion (s,MIGRATION_STATUS_ACTIVE , + &old_vm_running, &start_time);break ; } }
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 30/56] migration_completion: Take current state
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Soon we'll be in either ACTIVE or
POSTCOPY_ACTIVEwhen we complete migration, and we need to know which we expect to be in to change state safely.Signed-off-by Dr. David Alan Gilbert <[email protected]>
PATCH v9 31/56 MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
'MIGRATION_STATUS_POSTCOPY_ACTIVE' is entered after migrate_start_postcopy
'migration_in_postcopy' is provided for other sections to know if they're in postcopy.
--- include/migration/migration.h | 2 ++ migration/migration.c | 47 +++++++++++++++++++++++++++++++++++++++++-- qapi-schema.json | 4 +++- trace-events | 1 + 4 files changed, 51 insertions(+), 3 deletions(-)
include/migration/migration.h +2 -0
@@ -170,6 +170,8 @@ MigrationState *migrate_init(const MigrationParams *params);bool migration_in_setup (MigrationState *);bool migration_has_finished (MigrationState *);bool migration_has_failed (MigrationState *); +/* True if outgoing migration has entered postcopy phase */ +bool migration_in_postcopy (MigrationState *);MigrationState *migrate_get_current (void );void migrate_compress_threads_create (void );
migration/migration.c +45 -2
77 lines not on this page · raw message
qapi-schema.json +3 -1
@@ -430,6 +430,8 @@ # # @active: in the process of doing migration. # +# @postcopy-active: like active, but now in postcopy mode. (since 2.5) +# # @completed: migration is finished. # # @failed: some error occurred during migration process.
@@ -439,7 +441,7 @@ ## { 'enum': 'MigrationStatus', 'data': [ 'none', 'setup', 'cancelling', 'cancelled', - 'active', 'completed', 'failed' ] } + 'active', 'postcopy-active', 'completed', 'failed' ] } ## # @MigrationInfo
trace-events +1 -0
@@ -1452,6 +1452,7 @@ migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")" migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" +migration_thread_setup_complete(void) "" open_return_path_on_source(void) "" open_return_path_on_source_continue(void) "" source_return_path_thread_bad_end(void) ""
Signature · 1 line
2.5.0
PATCH v9 32/56 Avoid sending vmdescription during postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
VMDescription is normally sent at the end, after all of the devices; however that's not the end for postcopy, so just don't send it when in postcopy.
--- migration/savevm.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
migration/savevm.c +2 -1
@@ -974,7 +974,8 @@ int qemu_savevm_state_iterate(QEMUFile *f)static bool should_send_vmdesc (void ) {MachineState *machine =MACHINE (qdev_get_machine ()); -return !machine->suppress_vmdesc; +bool in_postcopy =migration_in_postcopy (migrate_get_current ()); +return !machine->suppress_vmdesc && !in_postcopy; }void qemu_savevm_state_complete_precopy (QEMUFile *f)
Signature · 1 line
2.5.0
PATCH v9 33/56 Add qemu_savevm_state_complete_postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add qemu_savevm_state_complete_postcopy to complement qemu_savevm_state_complete_precopy together with a new save_live_complete_postcopy method on devices.
The save_live_complete_precopy method is called on all devices during a precopy migration, and all non-postcopy devices during a postcopy migration at the transition.
The save_live_complete_postcopy method is called at the end of postcopy for all postcopiable devices.
--- include/migration/vmstate.h | 1 + include/sysemu/sysemu.h | 1 + migration/ram.c | 1 + migration/savevm.c | 49 +++++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 50 insertions(+), 2 deletions(-)
include/migration/vmstate.h +1 -0
@@ -40,6 +40,7 @@ typedef struct SaveVMHandlers {SaveStateHandler *save_state;void (*cleanup)(void *opaque); +int (*save_live_complete_postcopy)(QEMUFile *f,void *opaque);int (*save_live_complete_precopy)(QEMUFile *f,void *opaque);/* This runs both outside and inside the iothread lock. */
include/sysemu/sysemu.h +1 -0
@@ -111,6 +111,7 @@ void qemu_savevm_state_begin(QEMUFile *f,void qemu_savevm_state_header (QEMUFile *f);int qemu_savevm_state_iterate (QEMUFile *f);void qemu_savevm_state_cleanup (void ); +void qemu_savevm_state_complete_postcopy (QEMUFile *f);void qemu_savevm_state_complete_precopy (QEMUFile *f); void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size, uint64_t *res_non_postcopiable,
migration/ram.c +1 -0
@@ -1704,6 +1704,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)static SaveVMHandlers savevm_ram_handlers = { .save_live_setup = ram_save_setup, .save_live_iterate = ram_save_iterate, + .save_live_complete_postcopy = ram_save_complete, .save_live_complete_precopy = ram_save_complete, .save_live_pending = ram_save_pending, .load_state = ram_load,
migration/savevm.c +47 -2
73 lines not on this page · raw message
Signature · 1 line
2.5.0
PATCH v9 34/56 Postcopy: Maintain unsentmap
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Maintain an 'unsentmap' of pages that have yet to be sent. This is used in the following patches to discard some set of the pages already sent as we enter postcopy mode.
--- migration/ram.c | 51 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 45 insertions(+), 6 deletions(-)
migration/ram.c +45 -6
@@ -237,7 +237,14 @@ typedef struct PageSearchStatus PageSearchStatus;static struct BitmapRcu {struct rcu_head rcu; +/* Main migration bitmap */ unsigned long *bmap; +/* bitmap of pages that haven't been sent even once +* only maintained and used in postcopy at the moment +* where it's used to send the dirtymap at the start +* of the postcopy phase +*/ +unsigned long *unsentmap; } *migration_bitmap_rcu;struct CompressParam {
@@ -531,10 +538,18 @@ static int save_xbzrle_page(QEMUFile *f, uint8_t **current_data,return 1 ; } -/* Called with rcu_read_lock() to protect migration_bitmap */ +/* Called with rcu_read_lock() to protect migration_bitmap +* rb: The RAMBlock to search for dirty pages in +* start: Start address (typically so we can continue from previous page) +* ram_addr_abs: Pointer into which to store the address of the dirty page +* within the global ram_addr space +* +* Returns: byte offset within memory region of the start of a dirty page +*/ static inline ram_addr_t migration_bitmap_find_and_reset_dirty (RAMBlock *rb, -ram_addr_t start) +ram_addr_t start, +ram_addr_t *ram_addr_abs) {unsigned long base = rb->offset >>TARGET_PAGE_BITS ;unsigned long nr = base + (start >>TARGET_PAGE_BITS );
@@ -555,6 +570,7 @@ ram_addr_t migration_bitmap_find_and_reset_dirty(RAMBlock *rb,clear_bit (next, bitmap); migration_dirty_pages--; } + *ram_addr_abs = next <<TARGET_PAGE_BITS ;return (next - base) <<TARGET_PAGE_BITS ; }
@@ -953,10 +969,11 @@ static int ram_save_compressed_page(QEMUFile *f, RAMBlock *block,* @*again: Set to false if the search has scanned the whole of RAM */ static bool find_dirty_block (QEMUFile *f,PageSearchStatus *pss, -bool *again) +bool *again,ram_addr_t *ram_addr_abs) { pss->offset =migration_bitmap_find_and_reset_dirty (pss->block, - pss->offset); + pss->offset, + ram_addr_abs);if (pss->complete_round && pss->block == last_seen_block && pss->offset >= last_offset) {/*
@@ -1014,6 +1031,8 @@ static int ram_find_and_save_block(QEMUFile *f, bool last_stage,PageSearchStatus pss;int pages =0 ;bool again, found; +ram_addr_t dirty_ram_abs;/* Address of the start of the dirty page in +ram_addr_t space */ pss.block = last_seen_block; pss.offset = last_offset;
@@ -1024,7 +1043,7 @@ static int ram_find_and_save_block(QEMUFile *f, bool last_stage, } do { - found = find_dirty_block(f, &pss, &again); + found = find_dirty_block(f, &pss, &again, &dirty_ram_abs); if (found) { if (compression_switch && migrate_use_compression()) {
@@ -1038,7 +1057,14 @@ static int ram_find_and_save_block(QEMUFile *f, bool last_stage,/* if page is unmodified, continue to the next */ if (pages >0 ) { +unsigned long *unsentmap; + + unsentmap =atomic_rcu_read (&migration_bitmap_rcu)->unsentmap; last_sent_block = pss.block; +if (unsentmap) { +clear_bit (dirty_ram_abs >>TARGET_PAGE_BITS , unsentmap); + } + } } }while (!pages && again);
@@ -1097,6 +1123,7 @@ void free_xbzrle_decoded_buf(void)static void migration_bitmap_free (struct BitmapRcu *bmap) {g_free (bmap->bmap); +g_free (bmap->unsentmap);g_free (bmap); }
@@ -1153,6 +1180,13 @@ void migration_bitmap_extend(ram_addr_t old, ram_addr_t new)qemu_mutex_lock (&migration_bitmap_mutex);bitmap_copy (bitmap->bmap, old_bitmap->bmap, old);bitmap_set (bitmap->bmap, old, new - old); + +/* We don't have a way to safely extend the sentmap +* with RCU; so mark it as missing, entry to postcopy +* will fail. +*/ + bitmap->unsentmap =NULL ; +atomic_rcu_set (&migration_bitmap_rcu, bitmap);qemu_mutex_unlock (&migration_bitmap_mutex); migration_dirty_pages += new - old;
@@ -1253,10 +1287,15 @@ static int ram_save_setup(QEMUFile *f, void *opaque)reset_ram_globals (); ram_bitmap_pages =last_ram_offset () >>TARGET_PAGE_BITS ; - migration_bitmap_rcu = g_new(struct BitmapRcu, 1); + migration_bitmap_rcu = g_new0(struct BitmapRcu, 1); migration_bitmap_rcu->bmap =bitmap_new (ram_bitmap_pages);bitmap_set (migration_bitmap_rcu->bmap,0 , ram_bitmap_pages); +if (migrate_postcopy_ram ()) { + migration_bitmap_rcu->unsentmap =bitmap_new (ram_bitmap_pages); +bitmap_set (migration_bitmap_rcu->unsentmap,0 , ram_bitmap_pages); + } +/* * Count the total number of pages used by ram blocks not including any * gaps due to alignment or unplugs.
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 34/56] Postcopy: Maintain unsentmap
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Maintain an 'unsentmap' of pages that have yet to be sent. This is used in the following patches to discard some set of the pages already sent as we enter postcopy mode.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
PATCH v9 35/56 Postcopy: Calculate discard
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.
Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.
--- include/migration/migration.h | 5 ++ include/migration/postcopy-ram.h | 35 ++++++++ include/qemu/typedefs.h | 1 + migration/migration.c | 1 + migration/postcopy-ram.c | 126 +++++++++++++++++++++++++++ migration/ram.c | 181 ++++++++++++++++++++++++++++++++++++++- migration/savevm.c | 2 - trace-events | 6 ++ 8 files changed, 354 insertions(+), 3 deletions(-)
include/migration/migration.h +5 -0
@@ -199,6 +199,11 @@ double xbzrle_mig_cache_miss_rate(void);void ram_handle_compressed (void *host,uint8_t ch,uint64_t size);void ram_debug_dump_bitmap (unsigned long *todump,bool expected); +/* For outgoing discard bitmap */ +int ram_postcopy_send_discard_bitmap (MigrationState *ms); +/* For incoming postcopy discard */ +int ram_discard_range (MigrationIncomingState *mis,const char *block_name, +uint64_t start,size_t length);/** * @migrate_add_blocker - prevent migration from proceeding
include/migration/postcopy-ram.h +35 -0
@@ -16,4 +16,39 @@/* Return true if the host supports everything we need to do postcopy-ram */ bool postcopy_ram_supported_by_host (void ); +/* +* Discard the contents of 'length' bytes from 'start' +* We can assume that if we've been called postcopy_ram_hosttest returned true +*/ +int postcopy_ram_discard_range (MigrationIncomingState *mis,uint8_t *start, +size_t length); + + +/* +* Called at the start of each RAMBlock by the bitmap code. +* 'offset' is the bitmap offset of the named RAMBlock in the migration +* bitmap. +* Returns a new PDS +*/ +PostcopyDiscardState *postcopy_discard_send_init (MigrationState *ms, +unsigned long offset, +const char *name); + +/* +* Called by the bitmap code for each chunk to discard. +* May send a discard message, may just leave it queued to +* be sent later. +* @start,@length: a range of pages in the migration bitmap in the +* RAM block passed to postcopy_discard_send_init() (length=1 is one page) +*/ +void postcopy_discard_send_range (MigrationState *ms,PostcopyDiscardState *pds, +unsigned long start,unsigned long length); + +/* +* Called at the end of each RAMBlock by the bitmap code. +* Sends any outstanding discard messages, frees the PDS. +*/ +void postcopy_discard_send_finish (MigrationState *ms, +PostcopyDiscardState *pds); +#endif
include/qemu/typedefs.h +1 -0
@@ -66,6 +66,7 @@ typedef struct PCMachineState PCMachineState;typedef struct PCMachineClass PCMachineClass ;typedef struct PCMCIACardState PCMCIACardState ;typedef struct PixelFormat PixelFormat ; +typedef struct PostcopyDiscardState PostcopyDiscardState ;typedef struct PropertyInfo PropertyInfo ;typedef struct Property Property ;typedef struct QEMUBH QEMUBH ;
migration/migration.c +1 -0
@@ -24,6 +24,7 @@#include "qemu/sockets.h" #include "qemu/rcu.h" #include "migration/block.h" +#include "migration/postcopy-ram.h" #include "qemu/thread.h" #include "qmp-commands.h" #include "trace.h"
migration/postcopy-ram.c +126 -0
143 lines not on this page · raw message
migration/ram.c +180 -1
205 lines not on this page · raw message
migration/savevm.c +0 -2
@@ -1349,7 +1349,6 @@ static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis, }trace_loadvm_postcopy_ram_handle_discard_header (ramid, len);while (len) { -/* TODO - ram_discard_range gets added in a later patch uint64_t start_addr, block_length; start_addr =qemu_get_be64 (mis->from_src_file); block_length =qemu_get_be64 (mis->from_src_file);
@@ -1360,7 +1359,6 @@ static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis,if (ret) {return ret; } -*/ }trace_loadvm_postcopy_ram_handle_discard_end ();
trace-events +6 -0
@@ -1255,6 +1255,7 @@ qemu_file_fclose(void) "" migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64"" migration_throttle(void) "" +ram_postcopy_send_discard_bitmap(void) "" # hw/display/qxl.c disable qxl_interface_set_mm_time(int qid, uint32_t mm_time) "%d %d"
@@ -1532,6 +1533,11 @@ rdma_start_incoming_migration_after_rdma_listen(void) "" rdma_start_outgoing_migration_after_rdma_connect(void) "" rdma_start_outgoing_migration_after_rdma_source_init(void) "" +# migration/postcopy-ram.c +postcopy_discard_send_finish(const char *ramblock, int nwords, int ncmds) "%s mask words sent=%d in %d commands" +postcopy_discard_send_range(const char *ramblock, unsigned long start, unsigned long length) "%s:%lx/%lx" +postcopy_ram_discard_range(void *start, size_t length) "%p,+%zx" + # kvm-all.c kvm_ioctl(int type, void *arg) "type 0x%x, arg %p" kvm_vm_ioctl(int type, void *arg) "type 0x%x, arg %p"
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 35/56] Postcopy: Calculate discard
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.
Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.Signed-off-by Dr. David Alan Gilbert <[email protected]>
But please, send a fix for:
Quoted 7 lines
return ret;}
migration/postcopy-ram.c +4
-0+/** +* postcopy_ram_discard_range: Discard a range of memory. +* We can assume that if we've been called postcopy_ram_hosttest returned true. +*
migration/postcopy-ram.c out: · line 179
Quoted 10 lines
migration/postcopy-ram.c +10
-0+* @mis: Current incoming migration state. +* @start, @length: range of memory to discard. +* +* returns: 0 on success. +*/ +int postcopy_ram_discard_range (MigrationIncomingState *mis,uint8_t *start, +size_t length) +{ +trace_postcopy_ram_discard_range (start, length); +if (madvise (start, length,MADV_DONTNEED )) {
qemu_madvise(start, length, QEMU_MADV_DONTNEED)
Re: [Qemu-devel] [PATCH v9 35/56] Postcopy: Calculate discard
Quoted 20 lines · * Juan Quintela ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs.
Calculate list of sent & dirty pages
Provide helpers on the destination side to discard these.Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by Juan Quintela <[email protected]>But please, send a fix for:
Quoted 3 lines
return ret;}
Quoted 16 lines
Quoted 4 lines
migration/postcopy-ram.c +4
-0+/** +* postcopy_ram_discard_range: Discard a range of memory. +* We can assume that if we've been called postcopy_ram_hosttest returned true. +* migration/postcopy-ram.c out: · line 179
Quoted 10 lines
migration/postcopy-ram.c +10
-0+* @mis: Current incoming migration state. +* @start, @length: range of memory to discard. +* +* returns: 0 on success. +*/ +int postcopy_ram_discard_range (MigrationIncomingState *mis,uint8_t *start, +size_t length) +{ +trace_postcopy_ram_discard_range (start, length); +if (madvise (start, length,MADV_DONTNEED )) {
qemu_madvise(start, length, QEMU_MADV_DONTNEED)
I thought I'd better check before making a patch for that, and it turns out you have to be really careful. qemu_madvise may call posix_madvise instead of madvise (depending what config decides); you might think that a madvise(..,..,MADV_DONTNEED) is the same as a posix_madvise(..,..,POSIX_MADV_DONTNEED)
but actually on Linux a posix_madvise does precisely nothing; from the glibc source:
{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>} posix_madvise (void *addr, size_t len, int advice)
{
/* We have one problem: the kernel's MADV_DONTNEED does not
correspond to POSIX's POSIX_MADV_DONTNEED. The former simply discards changes made to the memory without writing it back to disk, if this would be necessary. The POSIX behavior does not allow this. There is no functionality mapping the POSIX behavior so far so we ignore that advice for now. */
if (advice == POSIX_MADV_DONTNEED)
return 0;
{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}{<>}
oh and notice how it returns with a 'return 0' so we can't tell if we fall down that trap.
So, it's best to stick to an actual, solid, good old fashioned madvise which does what it was asked.
Dave
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v9 36/56 postcopy: Incoming initialisation
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
--- include/migration/migration.h | 3 ++ include/migration/postcopy-ram.h | 12 +++++ migration/postcopy-ram.c | 102 +++++++++++++++++++++++++++++++++++++++ migration/ram.c | 11 +++++ migration/savevm.c | 6 +++ trace-events | 2 + 6 files changed, 136 insertions(+)
include/migration/migration.h +3 -0
@@ -86,6 +86,8 @@ struct MigrationIncomingState {*/ QemuEvent main_thread_load_event; +/* For the kernel to send us notifications */ +int userfault_fd;QEMUFile *to_src_file;QemuMutex rp_mutex;/* We send replies from multiple threads */
@@ -204,6 +206,7 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms);/* For incoming postcopy discard */ int ram_discard_range (MigrationIncomingState *mis,const char *block_name,uint64_t start,size_t length); +int ram_postcopy_incoming_init (MigrationIncomingState *mis);/** * @migrate_add_blocker - prevent migration from proceeding
include/migration/postcopy-ram.h +12 -0
@@ -17,6 +17,18 @@bool postcopy_ram_supported_by_host (void );/* +* Initialise postcopy-ram, setting the RAM to a state where we can go into +* postcopy later; must be called prior to any precopy. +* called from ram.c's similarly named ram_postcopy_incoming_init +*/ +int postcopy_ram_incoming_init (MigrationIncomingState *mis,size_t ram_pages); + +/* +* At the end of a migration where postcopy_ram_incoming_init was called. +*/ +int postcopy_ram_incoming_cleanup (MigrationIncomingState *mis); + +/* * Discard the contents of 'length' bytes from 'start' * We can assume that if we've been called postcopy_ram_hosttest returned true */
migration/postcopy-ram.c +102 -0
114 lines not on this page · raw message
migration/ram.c +11 -0
@@ -1794,6 +1794,17 @@ static void decompress_data_with_multi_threads(uint8_t *compbuf, } } +/* +* Allocate data structures etc needed by incoming migration with postcopy-ram +* postcopy-ram's similarly names postcopy_ram_incoming_init does the work +*/ +int ram_postcopy_incoming_init (MigrationIncomingState *mis) +{ +size_t ram_pages =last_ram_offset () >>TARGET_PAGE_BITS ; + +return postcopy_ram_incoming_init (mis, ram_pages); +} +static int ram_load (QEMUFile *f,void *opaque,int version_id) {int flags =0 , ret =0 ;
migration/savevm.c +6 -0
@@ -1281,6 +1281,12 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis)return -1 ; } +if (ram_postcopy_incoming_init (mis)) { +return -1 ; + } + +postcopy_state_set (POSTCOPY_INCOMING_ADVISE ); +return 0 ; }
trace-events +2 -0
@@ -1537,6 +1537,8 @@ rdma_start_outgoing_migration_after_rdma_source_init(void) "" postcopy_discard_send_finish(const char *ramblock, int nwords, int ncmds) "%s mask words sent=%d in %d commands" postcopy_discard_send_range(const char *ramblock, unsigned long start, unsigned long length) "%s:%lx/%lx" postcopy_ram_discard_range(void *start, size_t length) "%p,+%zx" +postcopy_cleanup_range(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx" +postcopy_init_range(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx" # kvm-all.c kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 36/56] postcopy: Incoming initialisation
Quoted 5 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Signed-off-by Dr. David Alan Gilbert <[email protected]>Reviewed-by David Gibson <[email protected]>Reviewed-by Amit Shah <[email protected]>
PATCH v9 37/56 postcopy: ram_enable_notify to switch on userfault
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Mark the area of RAM as 'userfault'
Start up a fault-thread to handle any userfaults we might receive from it (to be filled in later)
--- include/migration/migration.h | 3 ++ include/migration/postcopy-ram.h | 6 ++++ migration/postcopy-ram.c | 68 ++++++++++++++++++++++++++++++++++++++++ migration/savevm.c | 9 ++++++ 4 files changed, 86 insertions(+)
include/migration/migration.h +3 -0
@@ -86,6 +86,9 @@ struct MigrationIncomingState {*/ QemuEvent main_thread_load_event; +QemuThread fault_thread; +QemuSemaphore fault_thread_sem; +/* For the kernel to send us notifications */ int userfault_fd;QEMUFile *to_src_file;
include/migration/postcopy-ram.h +6 -0
@@ -17,6 +17,12 @@bool postcopy_ram_supported_by_host (void );/* +* Make all of RAM sensitive to accesses to areas that haven't yet been written +* and wire up anything necessary to deal with it. +*/ +int postcopy_ram_enable_notify (MigrationIncomingState *mis); + +/* * Initialise postcopy-ram, setting the RAM to a state where we can go into * postcopy later; must be called prior to any precopy. * called from ram.c's similarly named ram_postcopy_incoming_init
migration/postcopy-ram.c +68 -0
80 lines not on this page · raw message
migration/savevm.c +9 -0
@@ -1381,6 +1381,15 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)return -1 ; } +/* +* Sensitise RAM - can now generate requests for blocks that don't exist +* However, at this point the CPU shouldn't be running, and the IO +* shouldn't be doing anything yet so don't actually expect requests +*/ +if (postcopy_ram_enable_notify (mis)) { +return -1 ; + } +/* TODO start up the postcopy listening thread */ return 0 ; }
Signature · 1 line
2.5.0
PATCH v9 38/56 Postcopy: Postcopy startup in migration thread
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Rework the migration thread to setup and start postcopy.
--- include/migration/migration.h | 3 + migration/migration.c | 174 ++++++++++++++++++++++++++++++++++++++++-- trace-events | 4 + 3 files changed, 174 insertions(+), 7 deletions(-)
include/migration/migration.h +3 -0
@@ -135,6 +135,9 @@ struct MigrationState/* Flag set once the migration has been asked to enter postcopy */ bool start_postcopy; + +/* Flag set once the migration thread is running (and needs joining) */ +bool migration_thread_running; };void process_incoming_migration (QEMUFile *f);
migration/migration.c +167 -7
253 lines not on this page · raw message
trace-events +4 -0
@@ -1453,9 +1453,13 @@ migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")" migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" +migration_thread_after_loop(void) "" +migration_thread_file_err(void) "" migration_thread_setup_complete(void) "" open_return_path_on_source(void) "" open_return_path_on_source_continue(void) "" +postcopy_start(void) "" +postcopy_start_set_run(void) "" source_return_path_thread_bad_end(void) "" source_return_path_thread_end(void) "" source_return_path_thread_entry(void) ""
Signature · 1 line
2.5.0
PATCH v9 39/56 Postcopy: End of iteration
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The end of migration in postcopy is a bit different since some of the things normally done at the end of migration have already been done on the transition to postcopy.
--- migration/migration.c | 51 +++++++++++++++++++++++++++++++++++++-------------- trace-events | 6 +++++- 2 files changed, 42 insertions(+), 15 deletions(-)
migration/migration.c +37 -14
@@ -1258,7 +1258,6 @@ static int open_return_path_on_source(MigrationState *ms)return 0 ; } -__attribute__ (( unused ))/* Until later in patch series */ /* Returns 0 if the RP was ok, otherwise there was an error on the RP */ static int await_return_path_close_on_source (MigrationState *ms) {
@@ -1399,23 +1398,47 @@ static void migration_completion(MigrationState *s, int current_active_state, {int ret; -qemu_mutex_lock_iothread (); - *start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); -qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); - *old_vm_running =runstate_is_running (); +if (s->state ==MIGRATION_STATUS_ACTIVE ) { +qemu_mutex_lock_iothread (); + *start_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME ); +qemu_system_wakeup_request (QEMU_WAKEUP_REASON_OTHER ); + *old_vm_running =runstate_is_running (); + ret =global_state_store (); + +if (!ret) { + ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); +if (ret >=0 ) { +qemu_file_set_rate_limit (s->file,INT64_MAX ); +qemu_savevm_state_complete_precopy (s->file); + } + } +qemu_mutex_unlock_iothread (); - ret =global_state_store (); -if (!ret) { - ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE ); -if (ret >=0 ) { -qemu_file_set_rate_limit (s->file,INT64_MAX ); -qemu_savevm_state_complete_precopy (s->file); +if (ret <0 ) { + goto fail; } + }else if (s->state ==MIGRATION_STATUS_POSTCOPY_ACTIVE ) { +trace_migration_completion_postcopy_end (); + +qemu_savevm_state_complete_postcopy (s->file); +trace_migration_completion_postcopy_end_after_complete (); } -qemu_mutex_unlock_iothread (); -if (ret <0 ) { - goto fail; +/* +* If rp was opened we must clean up the thread before +* cleaning everything else up (since if there are no failures +* it will wait for the destination to send it's status in +* a SHUT command). +* Postcopy opens rp if enabled (even if it's not avtivated) +*/ +if (migrate_postcopy_ram ()) { +int rp_error; +trace_migration_completion_postcopy_end_before_rp (); + rp_error =await_return_path_close_on_source (s); +trace_migration_completion_postcopy_end_after_rp (rp_error); +if (rp_error) { + goto fail; + } }if (qemu_file_get_error (s->file)) {
trace-events +5 -1
@@ -1453,6 +1453,11 @@ migrate_fd_error(void) "" migrate_fd_cancel(void) "" migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")" migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" +migration_completion_file_err(void) "" +migration_completion_postcopy_end(void) "" +migration_completion_postcopy_end_after_complete(void) "" +migration_completion_postcopy_end_before_rp(void) "" +migration_completion_postcopy_end_after_rp(int rp_error) "%d" migration_thread_after_loop(void) "" migration_thread_file_err(void) "" migration_thread_setup_complete(void) ""
@@ -1470,7 +1475,6 @@ migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, migrate_state_too_big(void) "" migrate_global_state_post_load(const char *state) "loaded state: %s" migrate_global_state_pre_save(const char *state) "saved state: %s" -migration_completion_file_err(void) "" migration_thread_low_pending(uint64_t pending) "%" PRIu64 # migration/rdma.c
Signature · 1 line
2.5.0
PATCH v9 40/56 Page request: Add MIG_RP_MSG_REQ_PAGES reverse command
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add MIG_RP_MSG_REQ_PAGES command on Return path for the postcopy destination to request a page from the source.
Two versions exist:
MIG_RP_MSG_REQ_PAGES_ID that includes a RAMBlock name and start/len MIG_RP_MSG_REQ_PAGES that just has start/len for use with the same
RAMBlock as a previous MIG_RP_MSG_REQ_PAGES_ID
--- include/migration/migration.h | 5 ++++ migration/migration.c | 70 +++++++++++++++++++++++++++++++++++++++++++ trace-events | 1 + 3 files changed, 76 insertions(+)
include/migration/migration.h +5 -0
@@ -49,6 +49,9 @@ enum mig_rp_message_type {MIG_RP_MSG_SHUT ,/* sibling will not send any more RP messages */ MIG_RP_MSG_PONG ,/* Response to a PING; data (seq: be32 ) */ +MIG_RP_MSG_REQ_PAGES_ID ,/* data (start: be64, len: be32, id: string) */ +MIG_RP_MSG_REQ_PAGES ,/* data (start: be64, len: be32) */ + MIG_RP_MSG_MAX };
@@ -256,6 +259,8 @@ void migrate_send_rp_shut(MigrationIncomingState *mis, uint32_t value);void migrate_send_rp_pong (MigrationIncomingState *mis,uint32_t value); +void migrate_send_rp_req_pages (MigrationIncomingState *mis,const char * rbname, +ram_addr_t start,size_t len);void ram_control_before_iterate (QEMUFile *f,uint64_t flags);void ram_control_after_iterate (QEMUFile *f,uint64_t flags);
migration/migration.c +70 -0
98 lines not on this page · raw message
trace-events +1 -0
@@ -1451,6 +1451,7 @@ migrate_set_state(int new_state) "new state %d" migrate_fd_cleanup(void) "" migrate_fd_error(void) "" migrate_fd_cancel(void) "" +migrate_handle_rp_req_pages(const char *rbname, size_t start, size_t len) "in %s at %zx len %zx" migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")" migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" migration_completion_file_err(void) ""
Signature · 1 line
2.5.0
PATCH v9 41/56 Page request: Process incoming page request
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
On receiving MIG_RPCOMM_REQ_PAGES look up the address and queue the page.
--- include/migration/migration.h | 22 +++++++++++ migration/migration.c | 31 +++++++++++++++- migration/ram.c | 85 +++++++++++++++++++++++++++++++++++++++++++ trace-events | 1 + 4 files changed, 138 insertions(+), 1 deletion(-)
include/migration/migration.h +22 -0
@@ -105,6 +105,18 @@ MigrationIncomingState *migration_incoming_get_current(void);MigrationIncomingState *migration_incoming_state_new (QEMUFile *f);void migration_incoming_state_destroy (void ); +/* +* An outstanding page request, on the source, having been received +* and queued +*/ +struct MigrationSrcPageRequest { +RAMBlock *rb; +hwaddr offset; +hwaddr len; + +QSIMPLEQ_ENTRY (MigrationSrcPageRequest) next_req; +}; +struct MigrationState {int64_t bandwidth_limit;
@@ -141,6 +153,12 @@ struct MigrationState/* Flag set once the migration thread is running (and needs joining) */ bool migration_thread_running; + +/* Queue of outstanding page requests from the destination */ +QemuMutex src_page_req_mutex; + QSIMPLEQ_HEAD(src_page_requests, MigrationSrcPageRequest) src_page_requests; +/* The RAMBlock used in the last src_page_request */ +RAMBlock *last_req_rb; };void process_incoming_migration (QEMUFile *f);
@@ -288,6 +306,10 @@ void savevm_skip_configuration(void);int global_state_store (void );void global_state_store_running (void ); +void flush_page_queue (MigrationState *ms); +int ram_save_queue_pages (MigrationState *ms,const char *rbname, +ram_addr_t start,ram_addr_t len); +PostcopyState postcopy_state_get (void );/* Set the state and return the old state */ PostcopyState postcopy_state_set (PostcopyState new_state);
migration/migration.c +30 -1
@@ -21,6 +21,7 @@#include "sysemu/sysemu.h" #include "block/block.h" #include "qapi/qmp/qerror.h" +#include "qapi/util.h" #include "qemu/sockets.h" #include "qemu/rcu.h" #include "migration/block.h"
@@ -28,9 +29,10 @@#include "qemu/thread.h" #include "qmp-commands.h" #include "trace.h" -#include "qapi/util.h" #include "qapi-event.h" #include "qom/cpu.h" +#include "exec/memory.h" +#include "exec/address-spaces.h" #define MAX_THROTTLE (32 << 20)/* Migration transfer speed throttling */
@@ -72,6 +74,7 @@ static PostcopyState incoming_postcopy_state;/* For outgoing */ MigrationState *migrate_get_current (void ) { +static bool once;static MigrationState current_migration = { .state =MIGRATION_STATUS_NONE , .bandwidth_limit =MAX_THROTTLE ,
@@ -89,6 +92,10 @@ MigrationState *migrate_get_current(void) DEFAULT_MIGRATE_X_CPU_THROTTLE_INCREMENT, }; +if (!once) { +qemu_mutex_init (¤t_migration.src_page_req_mutex); + once = true; + }return ¤t_migration; }
@@ -771,6 +778,8 @@ static void migrate_fd_cleanup(void *opaque)qemu_bh_delete (s->cleanup_bh); s->cleanup_bh =NULL ; +flush_page_queue (s); +if (s->file) {trace_migrate_fd_cleanup ();qemu_mutex_unlock_iothread ();
@@ -903,6 +912,8 @@ MigrationState *migrate_init(const MigrationParams *params) s->bandwidth_limit = bandwidth_limit;migrate_set_state (s,MIGRATION_STATUS_NONE ,MIGRATION_STATUS_SETUP ); +QSIMPLEQ_INIT (&s->src_page_requests); + s->total_time =qemu_clock_get_ms (QEMU_CLOCK_REALTIME );return s; }
@@ -1193,7 +1204,25 @@ static struct rp_cmd_args {static void migrate_handle_rp_req_pages (MigrationState *ms,const char * rbname,ram_addr_t start,size_t len) { +long our_host_ps =getpagesize (); +trace_migrate_handle_rp_req_pages (rbname, start, len); + +/* +* Since we currently insist on matching page sizes, just sanity check +* we're being asked for whole host pages. +*/ +if (start & (our_host_ps-1 ) || + (len & (our_host_ps-1 ))) { +error_report ("%s: Misaligned page request, start: " RAM_ADDR_FMT +" len: " RAM_ADDR_FMT , __func__, start, len); +mark_source_rp_bad (ms); +return ; + } + +if (ram_save_queue_pages (ms, rbname, start, len)) { +mark_source_rp_bad (ms); + } }/*
migration/ram.c +85 -0
91 lines not on this page · raw message
trace-events +1 -0
@@ -1256,6 +1256,7 @@ migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64"" migration_throttle(void) "" ram_postcopy_send_discard_bitmap(void) "" +ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: %zx len: %zx" # hw/display/qxl.c disable qxl_interface_set_mm_time(int qid, uint32_t mm_time) "%d %d"
Signature · 1 line
2.5.0
PATCH v9 42/56 Page request: Consume pages off the post-copy queue
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
When transmitting RAM pages, consume pages that have been queued by MIG_RPCOMM_REQPAGE commands and send them ahead of normal page scanning.
Note:
a) After a queued page the linear walk carries on from after the
unqueued page; there is a reasonable chance that the destination was about to ask for other closeby pages anyway.
b) We have to be careful of any assumptions that the page walking
code makes, in particular it does some short cuts on its first linear walk that break as soon as we do a queued page.
c) We have to be careful to not break up host-page size chunks, since
this makes it harder to place the pages on the destination.
--- migration/ram.c | 249 +++++++++++++++++++++++++++++++++++++++++++++++++------- trace-events | 2 + 2 files changed, 220 insertions(+), 31 deletions(-)
migration/ram.c +218 -31
@@ -548,9 +548,9 @@ static int save_xbzrle_page(QEMUFile *f, uint8_t **current_data,* Returns: byte offset within memory region of the start of a dirty page */ static inline -ram_addr_t migration_bitmap_find_and_reset_dirty (RAMBlock *rb, -ram_addr_t start, -ram_addr_t *ram_addr_abs) +ram_addr_t migration_bitmap_find_dirty (RAMBlock *rb, +ram_addr_t start, +ram_addr_t *ram_addr_abs) {unsigned long base = rb->offset >>TARGET_PAGE_BITS ;unsigned long nr = base + (start >>TARGET_PAGE_BITS );
@@ -567,15 +567,24 @@ ram_addr_t migration_bitmap_find_and_reset_dirty(RAMBlock *rb, next =find_next_bit (bitmap, size, nr); } -if (next < size) { -clear_bit (next, bitmap); - migration_dirty_pages--; - } *ram_addr_abs = next <<TARGET_PAGE_BITS ;return (next - base) <<TARGET_PAGE_BITS ; } -/* Called with rcu_read_lock() to protect migration_bitmap */ +static inline bool migration_bitmap_clear_dirty (ram_addr_t addr) +{ +bool ret; +int nr = addr >>TARGET_PAGE_BITS ; +unsigned long *bitmap =atomic_rcu_read (&migration_bitmap_rcu)->bmap; + + ret =test_and_clear_bit (nr, bitmap); + +if (ret) { + migration_dirty_pages--; + } +return ret; +} +static void migration_bitmap_sync_range (ram_addr_t start,ram_addr_t length) {unsigned long *bitmap;
@@ -974,9 +983,8 @@ static int ram_save_compressed_page(QEMUFile *f, RAMBlock *block,static bool find_dirty_block (QEMUFile *f,PageSearchStatus *pss,bool *again,ram_addr_t *ram_addr_abs) { - pss->offset =migration_bitmap_find_and_reset_dirty (pss->block, - pss->offset, - ram_addr_abs); + pss->offset =migration_bitmap_find_dirty (pss->block, pss->offset, + ram_addr_abs);if (pss->complete_round && pss->block == last_seen_block && pss->offset >= last_offset) {/*
@@ -1015,6 +1023,107 @@ static bool find_dirty_block(QEMUFile *f, PageSearchStatus *pss, } } +/* +* Helper for 'get_queued_page' - gets a page off the queue +* ms: MigrationState in +* *offset: Used to return the offset within the RAMBlock +* ram_addr_abs: global offset in the dirty/sent bitmaps +* +* Returns: block (or NULL if none available) +*/ +static RAMBlock *unqueue_page (MigrationState *ms,ram_addr_t *offset, +ram_addr_t *ram_addr_abs) +{ +RAMBlock *block =NULL ; + +qemu_mutex_lock (&ms->src_page_req_mutex); +if (!QSIMPLEQ_EMPTY (&ms->src_page_requests)) { +struct MigrationSrcPageRequest *entry = +QSIMPLEQ_FIRST (&ms->src_page_requests); + block = entry->rb; + *offset = entry->offset; + *ram_addr_abs = (entry->offset + entry->rb->offset) & +TARGET_PAGE_MASK ; + +if (entry->len >TARGET_PAGE_SIZE ) { + entry->len -=TARGET_PAGE_SIZE ; + entry->offset +=TARGET_PAGE_SIZE ; + }else { +memory_region_unref (block->mr); +QSIMPLEQ_REMOVE_HEAD (&ms->src_page_requests, next_req); +g_free (entry); + } + } +qemu_mutex_unlock (&ms->src_page_req_mutex); + +return block; +} + +/* +* Unqueue a page from the queue fed by postcopy page requests; skips pages +* that are already sent (!dirty) +* +* ms: MigrationState in +* pss: PageSearchStatus structure updated with found block/offset +* ram_addr_abs: global offset in the dirty/sent bitmaps +* +* Returns: true if a queued page is found +*/ +static bool get_queued_page (MigrationState *ms,PageSearchStatus *pss, +ram_addr_t *ram_addr_abs) +{ +RAMBlock *block; +ram_addr_t offset; +bool dirty; + +do { + block =unqueue_page (ms, &offset, ram_addr_abs); +/* +* We're sending this page, and since it's postcopy nothing else +* will dirty it, and we must make sure it doesn't get sent again +* even if this queue request was received after the background +* search already sent it. +*/ +if (block) { +unsigned long *bitmap; + bitmap =atomic_rcu_read (&migration_bitmap_rcu)->bmap; + dirty =test_bit (*ram_addr_abs >>TARGET_PAGE_BITS , bitmap); +if (!dirty) { +trace_get_queued_page_not_dirty ( + block->idstr, (uint64_t )offset, + (uint64_t )*ram_addr_abs, +test_bit (*ram_addr_abs >>TARGET_PAGE_BITS , +atomic_rcu_read (&migration_bitmap_rcu)->unsentmap)); + }else { +trace_get_queued_page (block->idstr, + (uint64_t )offset, + (uint64_t )*ram_addr_abs); + } + } + + }while (block && !dirty); + +if (block) { +/* +* As soon as we start servicing pages out of order, then we have +* to kill the bulk stage, since the bulk stage assumes +* in (migration_bitmap_find_and_reset_dirty) that every page is +* dirty, that's no longer true. +*/ + ram_bulk_stage = false; + +/* +* We want the background search to continue from the queued page +* since the guest is likely to want other pages near to the page +* it just requested. +*/ + pss->block = block; + pss->offset = offset; + } + +return !!block; +} +/** * flush_page_queue: Flush any remaining pages in the ram request queue * it should be empty at the end anyway, but in error cases there may be
@@ -1099,6 +1208,90 @@ err:return -1 ; } +/** +* ram_save_target_page: Save one target page +* +* +* @f: QEMUFile where to send the data +* @block: pointer to block that contains the page we want to send +* @offset: offset inside the block for the page; +* @last_stage: if we are at the completion stage +* @bytes_transferred: increase it with the number of transferred bytes +* @dirty_ram_abs: Address of the start of the dirty page in ram_addr_t space +* +* Returns: Number of pages written. +*/ +static int ram_save_target_page (MigrationState *ms,QEMUFile *f, +RAMBlock *block,ram_addr_t offset, +bool last_stage, +uint64_t *bytes_transferred, +ram_addr_t dirty_ram_abs) +{ +int res =0 ; + +/* Check the pages is dirty and if it is send it */ +if (migration_bitmap_clear_dirty (dirty_ram_abs)) { +unsigned long *unsentmap; +if (compression_switch &&migrate_use_compression ()) { + res =ram_save_compressed_page (f, block, offset, + last_stage, + bytes_transferred); + }else { + res =ram_save_page (f, block, offset, last_stage, + bytes_transferred); + } + +if (res <0 ) { +return res; + } + unsentmap =atomic_rcu_read (&migration_bitmap_rcu)->unsentmap; +if (unsentmap) { +clear_bit (dirty_ram_abs >>TARGET_PAGE_BITS , unsentmap); + } + } + +return res; +} + +/** +* ram_save_host_page: Starting at *offset send pages upto the end +* of the current host page. It's valid for the initial +* offset to point into the middle of a host page +* in which case the remainder of the hostpage is sent. +* Only dirty target pages are sent. +* +* Returns: Number of pages written. +* +* @f: QEMUFile where to send the data +* @block: pointer to block that contains the page we want to send +* @offset: offset inside the block for the page; updated to last target page +* sent +* @last_stage: if we are at the completion stage +* @bytes_transferred: increase it with the number of transferred bytes +* @dirty_ram_abs: Address of the start of the dirty page in ram_addr_t space +*/ +static int ram_save_host_page (MigrationState *ms,QEMUFile *f,RAMBlock *block, +ram_addr_t *offset,bool last_stage, +uint64_t *bytes_transferred, +ram_addr_t dirty_ram_abs) +{ +int tmppages, pages =0 ; +do { + tmppages =ram_save_target_page (ms, f, block, *offset, last_stage, + bytes_transferred, dirty_ram_abs); +if (tmppages <0 ) { +return tmppages; + } + + pages += tmppages; + *offset +=TARGET_PAGE_SIZE ; + dirty_ram_abs +=TARGET_PAGE_SIZE ; + }while (*offset & (qemu_host_page_size -1 )); + +/* The offset we leave with is the last one we looked at */ + *offset -=TARGET_PAGE_SIZE ; +return pages; +}/** * ram_find_and_save_block: Finds a dirty page and sends it to f
@@ -1111,12 +1304,16 @@ err:* @f: QEMUFile where to send the data * @last_stage: if we are at the completion stage * @bytes_transferred: increase it with the number of transferred bytes +* +* On systems where host-page-size > target-page-size it will send all the +* pages in a host page that are dirty. */ static int ram_find_and_save_block (QEMUFile *f,bool last_stage,uint64_t *bytes_transferred) {PageSearchStatus pss; +MigrationState *ms =migrate_get_current ();int pages =0 ;bool again, found;ram_addr_t dirty_ram_abs;/* Address of the start of the dirty page in
@@ -1131,28 +1328,18 @@ static int ram_find_and_save_block(QEMUFile *f, bool last_stage, }do { - found =find_dirty_block (f, &pss, &again, &dirty_ram_abs); + again = true; + found =get_queued_page (ms, &pss, &dirty_ram_abs); -if (found) { -if (compression_switch &&migrate_use_compression ()) { - pages =ram_save_compressed_page (f, pss.block, pss.offset, - last_stage, - bytes_transferred); - }else { - pages =ram_save_page (f, pss.block, pss.offset, last_stage, - bytes_transferred); - } - -/* if page is unmodified, continue to the next */ -if (pages >0 ) { -unsigned long *unsentmap; +if (!found) { +/* priority queue empty, so just search for something dirty */ + found =find_dirty_block (f, &pss, &again, &dirty_ram_abs); + } - unsentmap =atomic_rcu_read (&migration_bitmap_rcu)->unsentmap; - last_sent_block = pss.block; -if (unsentmap) { -clear_bit (dirty_ram_abs >>TARGET_PAGE_BITS , unsentmap); - } - } +if (found) { + pages =ram_save_host_page (ms, f, pss.block, &pss.offset, + last_stage, bytes_transferred, + dirty_ram_abs); } }while (!pages && again);
trace-events +2 -0
@@ -1252,6 +1252,8 @@ vmstate_subsection_load_good(const char *parent) "%s" qemu_file_fclose(void) "" # migration/ram.c +get_queued_page(const char *block_name, uint64_t tmp_offset, uint64_t ram_addr) "%s/%" PRIx64 " ram_addr=%" PRIx64 +get_queued_page_not_dirty(const char *block_name, uint64_t tmp_offset, uint64_t ram_addr, int sent) "%s/%" PRIx64 " ram_addr=%" PRIx64 " (sent=%d)" migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64"" migration_throttle(void) ""
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 42/56] Page request: Consume pages off the post-copy queue
Quoted 10 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
When transmitting RAM pages, consume pages that have been queued by
MIG_RPCOMM_REQPAGEcommands and send them ahead of normal page scanning.Note:
a) After a queued page the linear walk carries on from after the
unqueued page; there is a reasonable chance that the destination was about to ask for other closeby pages anyway.
Quoted 8 lines
b) We have to be careful of any assumptions that the page walking
code makes, in particular it does some short cuts on its first linear walk that break as soon as we do a queued page.c) We have to be careful to not break up host-page size chunks, since
this makes it harder to place the pages on the destination.Signed-off-by Dr. David Alan Gilbert <[email protected]>
PATCH v9 43/56 postcopy_ram.c: place_page and helpers
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
postcopy_place_page (etc) provide a way for postcopy to place a page into guests memory atomically (using the copy ioctl on the ufd).
--- include/migration/migration.h | 1 + include/migration/postcopy-ram.h | 21 +++++++++ migration/postcopy-ram.c | 97 ++++++++++++++++++++++++++++++++++++++++ trace-events | 2 + 4 files changed, 121 insertions(+)
include/migration/migration.h +1 -0
@@ -96,6 +96,7 @@ struct MigrationIncomingState {int userfault_fd;QEMUFile *to_src_file;QemuMutex rp_mutex;/* We send replies from multiple threads */ +void *postcopy_tmp_page;/* See savevm.c */ LoadStateEntry_Head loadvm_handlers;
include/migration/postcopy-ram.h +21 -0
@@ -69,4 +69,25 @@ void postcopy_discard_send_range(MigrationState *ms, PostcopyDiscardState *pds,void postcopy_discard_send_finish (MigrationState *ms,PostcopyDiscardState *pds); +/* +* Place a page (from) at (host) efficiently +* There are restrictions on how 'from' must be mapped, in general best +* to use other postcopy_ routines to allocate. +* returns 0 on success +*/ +int postcopy_place_page (MigrationIncomingState *mis,void *host,void *from); + +/* +* Place a zero page at (host) atomically +* returns 0 on success +*/ +int postcopy_place_page_zero (MigrationIncomingState *mis,void *host); + +/* +* Allocate a page of memory that can be mapped at a later point in time +* using postcopy_place_page +* Returns: Pointer to allocated page +*/ +void *postcopy_get_tmp_page (MigrationIncomingState *mis); +#endif
migration/postcopy-ram.c +97 -0
115 lines not on this page · raw message
trace-events +2 -0
@@ -1551,6 +1551,8 @@ postcopy_discard_send_range(const char *ramblock, unsigned long start, unsigned postcopy_ram_discard_range(void *start, size_t length) "%p,+%zx" postcopy_cleanup_range(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx" postcopy_init_range(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx" +postcopy_place_page(void *host_addr) "host=%p" +postcopy_place_page_zero(void *host_addr) "host=%p" # kvm-all.c kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
Signature · 1 line
2.5.0
PATCH v9 44/56 Postcopy: Use helpers to map pages during migration
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
In postcopy, the destination guest is running at the same time as it's receiving pages; as we receive new pages we must put them into the guests address space atomically to avoid a running CPU accessing a partially written page.
Use the helpers in postcopy-ram.c to map these pages.
qemu_get_buffer_in_place is used to avoid a copy out of qemu_file in the case that postcopy is going to do a copy anyway.
--- migration/ram.c | 130 +++++++++++++++++++++++++++++++++++++++++++++++++++++++- trace-events | 1 + 2 files changed, 130 insertions(+), 1 deletion(-)
migration/ram.c +129 -1
@@ -1932,6 +1932,14 @@ static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)/* Must be called from within a rcu critical section. * Returns a pointer from within the RCU-protected ram_list. */ +/* +* Read a RAMBlock ID from the stream f, find the host address of the +* start of that block and add on 'offset' +* +* f: Stream to read from +* offset: Offset within the block +* flags: Page flags (mostly to see if it's a continuation of previous block) +*/ static inline void *host_from_stream_offset (QEMUFile *f,ram_addr_t offset,int flags)
@@ -2077,11 +2085,126 @@ int ram_postcopy_incoming_init(MigrationIncomingState *mis)return postcopy_ram_incoming_init (mis, ram_pages); } +/* +* Called in postcopy mode by ram_load(). +* rcu_read_lock is taken prior to this being called. +*/ +static int ram_load_postcopy (QEMUFile *f) +{ +int flags =0 , ret =0 ; +bool place_needed = false; +bool matching_page_sizes = qemu_host_page_size ==TARGET_PAGE_SIZE ; +MigrationIncomingState *mis =migration_incoming_get_current (); +/* Temporary page that is later 'placed' */ +void *postcopy_host_page =postcopy_get_tmp_page (mis); + +while (!ret && !(flags &RAM_SAVE_FLAG_EOS )) { +ram_addr_t addr; +void *host =NULL ; +void *page_buffer =NULL ; +void *place_source =NULL ; +uint8_t ch; +bool all_zero = false; + + addr =qemu_get_be64 (f); + flags = addr & ~TARGET_PAGE_MASK ; + addr &=TARGET_PAGE_MASK ; + +trace_ram_load_postcopy_loop ((uint64_t )addr, flags); + place_needed = false; +if (flags & (RAM_SAVE_FLAG_COMPRESS |RAM_SAVE_FLAG_PAGE )) { + host =host_from_stream_offset (f, addr, flags); +if (!host) { +error_report ("Illegal RAM offset " RAM_ADDR_FMT , addr); + ret = -EINVAL ; +break ; + } + page_buffer = host; +/* +* Postcopy requires that we place whole host pages atomically. +* To make it atomic, the data is read into a temporary page +* that's moved into place later. +* The migration protocol uses, possibly smaller, target-pages +* however the source ensures it always sends all the components +* of a host page in order. +*/ + page_buffer = postcopy_host_page + + ((uintptr_t )host & ~qemu_host_page_mask); +/* If all TP are zero then we can optimise the place */ +if (!((uintptr_t )host & ~qemu_host_page_mask)) { + all_zero = true; + } + +/* +* If it's the last part of a host page then we place the host +* page +*/ + place_needed = (((uintptr_t )host +TARGET_PAGE_SIZE ) & + ~qemu_host_page_mask) ==0 ; + place_source = postcopy_host_page; + } + +switch (flags & ~RAM_SAVE_FLAG_CONTINUE ) { +case RAM_SAVE_FLAG_COMPRESS : + ch =qemu_get_byte (f); +memset (page_buffer, ch,TARGET_PAGE_SIZE ); +if (ch) { + all_zero = false; + } +break ; + +case RAM_SAVE_FLAG_PAGE : + all_zero = false; +if (!place_needed || !matching_page_sizes) { +qemu_get_buffer (f, page_buffer,TARGET_PAGE_SIZE ); + }else { +/* Avoids the qemu_file copy during postcopy, which is +* going to do a copy later; can only do it when we +* do this read in one go (matching page sizes) +*/ +qemu_get_buffer_in_place (f, (uint8_t **)&place_source, +TARGET_PAGE_SIZE ); + } +break ; +case RAM_SAVE_FLAG_EOS : +/* normal exit */ +break ; +default : +error_report ("Unknown combination of migration flags: %#x" +" (postcopy mode)" , flags); + ret = -EINVAL ; + } + +if (place_needed) { +/* This gets called at the last target page in the host page */ +if (all_zero) { + ret =postcopy_place_page_zero (mis, + host +TARGET_PAGE_SIZE - + qemu_host_page_size); + }else { + ret =postcopy_place_page (mis, host +TARGET_PAGE_SIZE - + qemu_host_page_size, + place_source); + } + } +if (!ret) { + ret =qemu_file_get_error (f); + } + } + +return ret; +} +static int ram_load (QEMUFile *f,void *opaque,int version_id) {int flags =0 , ret =0 ;static uint64_t seq_iter;int len =0 ; +/* +* If system is running in postcopy mode, page inserts to host memory must +* be atomic +*/ +bool postcopy_running =postcopy_state_get () >=POSTCOPY_INCOMING_LISTENING ; seq_iter++;
@@ -2095,7 +2218,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)* critical section. */ rcu_read_lock (); -while (!ret && !(flags &RAM_SAVE_FLAG_EOS )) { + +if (postcopy_running) { + ret =ram_load_postcopy (f); + } + +while (!postcopy_running && !ret && !(flags &RAM_SAVE_FLAG_EOS )) {ram_addr_t addr, total_ram_bytes;void *host =NULL ;uint8_t ch;
trace-events +1 -0
@@ -1257,6 +1257,7 @@ get_queued_page_not_dirty(const char *block_name, uint64_t tmp_offset, uint64_t migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64"" migration_throttle(void) "" +ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: %zx len: %zx"
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 44/56] Postcopy: Use helpers to map pages during migration
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
In postcopy, the destination guest is running at the same time as it's receiving pages; as we receive new pages we must put them into the guests address space atomically to avoid a running CPU accessing a partially written page.
Quoted 6 lines
Use the helpers in
postcopy-ram.cto map these pages.
qemu_get_buffer_in_placeis used to avoid a copy out ofqemu_filein the case that postcopy is going to do a copy anyway.Signed-off-by Dr. David Alan Gilbert <[email protected]>
PATCH v9 45/56 postcopy: Check order of received target pages
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Ensure that target pages received within a host page are in order. This shouldn't trigger, but in the cases where the sender goes wrong and sends stuff out of order it produces a corruption that's really nasty to debug.
--- migration/ram.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
migration/ram.c +11 -0
@@ -2097,6 +2097,7 @@ static int ram_load_postcopy(QEMUFile *f)MigrationIncomingState *mis =migration_incoming_get_current ();/* Temporary page that is later 'placed' */ void *postcopy_host_page =postcopy_get_tmp_page (mis); +void *last_host =NULL ;while (!ret && !(flags &RAM_SAVE_FLAG_EOS )) {ram_addr_t addr;
@@ -2133,8 +2134,17 @@ static int ram_load_postcopy(QEMUFile *f)/* If all TP are zero then we can optimise the place */ if (!((uintptr_t )host & ~qemu_host_page_mask)) { all_zero = true; + }else { +/* not the 1st TP within the HP */ +if (host != (last_host +TARGET_PAGE_SIZE )) { +error_report ("Non-sequential target page %p/%p\n" , + host, last_host); + ret = -EINVAL ; +break ; + } } +/* * If it's the last part of a host page then we place the host * page
@@ -2143,6 +2153,7 @@ static int ram_load_postcopy(QEMUFile *f) ~qemu_host_page_mask) == 0; place_source = postcopy_host_page; } + last_host = host;switch (flags & ~RAM_SAVE_FLAG_CONTINUE ) {case RAM_SAVE_FLAG_COMPRESS :
Signature · 1 line
2.5.0
PATCH v9 46/56 Don't sync dirty bitmaps in postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Once we're in postcopy the source processors are stopped and memory shouldn't change any more, so there's no need to look at the dirty map.
There are two notes to this:
1) If we do resync and a page had changed then the page would get sent again, which the destination wouldn't allow (since it might have also modified the page)
2) Before disabling this I'd seen very rare cases where a page had been marked dirtied although the memory contents are apparently identical
--- migration/ram.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
migration/ram.c +5 -2
@@ -1846,7 +1846,9 @@ static int ram_save_complete(QEMUFile *f, void *opaque) {rcu_read_lock (); -migration_bitmap_sync (); +if (!migration_in_postcopy (migrate_get_current ())) { +migration_bitmap_sync (); + }ram_control_before_iterate (f,RAM_CONTROL_FINISH );
@@ -1881,7 +1883,8 @@ static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size, remaining_size =ram_save_remaining () *TARGET_PAGE_SIZE ; -if (remaining_size < max_size) { +if (!migration_in_postcopy (migrate_get_current ()) && + remaining_size < max_size) {qemu_mutex_lock_iothread ();rcu_read_lock ();migration_bitmap_sync ();
Signature · 1 line
2.5.0
PATCH v9 47/56 Don't iterate on precopy-only devices during postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
During the postcopy phase we must not call the iterate method on precopy-only devices, since they may have done some cleanup during the _complete call at the end of the precopy phase.
--- include/sysemu/sysemu.h | 2 +- migration/migration.c | 2 +- migration/savevm.c | 13 +++++++++++-- 3 files changed, 13 insertions(+), 4 deletions(-)
include/sysemu/sysemu.h +1 -1
@@ -109,7 +109,7 @@ bool qemu_savevm_state_blocked(Error **errp);void qemu_savevm_state_begin (QEMUFile *f,const MigrationParams *params);void qemu_savevm_state_header (QEMUFile *f); -int qemu_savevm_state_iterate (QEMUFile *f); +int qemu_savevm_state_iterate (QEMUFile *f,bool postcopy);void qemu_savevm_state_cleanup (void );void qemu_savevm_state_complete_postcopy (QEMUFile *f);void qemu_savevm_state_complete_precopy (QEMUFile *f);
migration/migration.c +1 -1
@@ -1628,7 +1628,7 @@ static void *migration_thread(void *opaque)continue ; }/* Just another iteration step */ -qemu_savevm_state_iterate (s->file); +qemu_savevm_state_iterate (s->file, entered_postcopy); }else {trace_migration_thread_low_pending (pending_size); migration_completion(s, current_active_state,
migration/savevm.c +11 -2
@@ -931,7 +931,7 @@ void qemu_savevm_state_begin(QEMUFile *f,* 0 : We haven't finished, caller have to go again * 1 : We have finished, we can go to complete phase */ -int qemu_savevm_state_iterate (QEMUFile *f) +int qemu_savevm_state_iterate (QEMUFile *f,bool postcopy) {SaveStateEntry *se;int ret =1 ;
@@ -946,6 +946,15 @@ int qemu_savevm_state_iterate(QEMUFile *f)continue ; } } +/* +* In the postcopy phase, any device that doesn't know how to +* do postcopy should have saved it's state in the _complete +* call that's already run, it might get confused if we call +* iterate afterwards. +*/ +if (postcopy && !se->ops->save_live_complete_postcopy) { +continue ; + }if (qemu_file_rate_limit (f)) {return 0 ; }
@@ -1160,7 +1169,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)qemu_mutex_lock_iothread ();while (qemu_file_get_error (f) ==0 ) { -if (qemu_savevm_state_iterate (f) >0 ) { +if (qemu_savevm_state_iterate (f, false) >0 ) {break ; } }
Signature · 1 line
2.5.0
PATCH v9 48/56 Host page!=target page: Cleanup bitmaps
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Prior to the start of postcopy, ensure that everything that will be transferred later is a whole host-page in size.
This is accomplished by discarding partially transferred host pages and marking any that are partially dirty as fully dirty.
--- migration/ram.c | 172 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 172 insertions(+)
migration/ram.c +172 -0
@@ -1584,6 +1584,171 @@ static int postcopy_each_ram_send_discard(MigrationState *ms) }/* +* Helper for postcopy_chunk_hostpages; it's called twice to cleanup +* the two bitmaps, that are similar, but one is inverted. +* +* We search for runs of target-pages that don't start or end on a +* host page boundary; +* unsent_pass=true: Cleans up partially unsent host pages by searching +* the unsentmap +* unsent_pass=false: Cleans up partially dirty host pages by searching +* the main migration bitmap +* +*/ +static void postcopy_chunk_hostpages_pass (MigrationState *ms,bool unsent_pass, +RAMBlock *block, +PostcopyDiscardState *pds) +{ +unsigned long *bitmap; +unsigned long *unsentmap; +unsigned int host_ratio = qemu_host_page_size /TARGET_PAGE_SIZE ; +unsigned long first = block->offset >>TARGET_PAGE_BITS ; +unsigned long len = block->used_length >>TARGET_PAGE_BITS ; +unsigned long last = first + (len -1 ); +unsigned long run_start; + + bitmap =atomic_rcu_read (&migration_bitmap_rcu)->bmap; + unsentmap =atomic_rcu_read (&migration_bitmap_rcu)->unsentmap; + +if (unsent_pass) { +/* Find a sent page */ + run_start =find_next_zero_bit (unsentmap, last +1 , first); + }else { +/* Find a dirty page */ + run_start =find_next_bit (bitmap, last +1 , first); + } + +while (run_start <= last) { +bool do_fixup = false; +unsigned long fixup_start_addr; +unsigned long host_offset; + +/* +* If the start of this run of pages is in the middle of a host +* page, then we need to fixup this host page. +*/ + host_offset = run_start % host_ratio; +if (host_offset) { + do_fixup = true; + run_start -= host_offset; + fixup_start_addr = run_start; +/* For the next pass */ + run_start = run_start + host_ratio; + }else { +/* Find the end of this run */ +unsigned long run_end; +if (unsent_pass) { + run_end =find_next_bit (unsentmap, last +1 , run_start +1 ); + }else { + run_end =find_next_zero_bit (bitmap, last +1 , run_start +1 ); + } +/* +* If the end isn't at the start of a host page, then the +* run doesn't finish at the end of a host page +* and we need to discard. +*/ + host_offset = run_end % host_ratio; +if (host_offset) { + do_fixup = true; + fixup_start_addr = run_end - host_offset; +/* +* This host page has gone, the next loop iteration starts +* from after the fixup +*/ + run_start = fixup_start_addr + host_ratio; + }else { +/* +* No discards on this iteration, next loop starts from +* next sent/dirty page +*/ + run_start = run_end +1 ; + } + } + +if (do_fixup) { +unsigned long page; + +/* Tell the destination to discard this page */ +if (unsent_pass || !test_bit (fixup_start_addr, unsentmap)) { +/* For the unsent_pass we: +* discard partially sent pages +* For the !unsent_pass (dirty) we: +* discard partially dirty pages that were sent +* (any partially sent pages were already discarded +* by the previous unsent_pass) +*/ +postcopy_discard_send_range (ms, pds, fixup_start_addr, + host_ratio); + } + +/* Clean up the bitmap */ +for (page = fixup_start_addr; + page < fixup_start_addr + host_ratio; page++) { +/* All pages in this host page are now not sent */ +set_bit (page, unsentmap); + +/* +* Remark them as dirty, updating the count for any pages +* that weren't previously dirty. +*/ + migration_dirty_pages += !test_and_set_bit (page, bitmap); + } + } + +if (unsent_pass) { +/* Find the next sent page for the next iteration */ + run_start =find_next_zero_bit (unsentmap, last +1 , + run_start); + }else { +/* Find the next dirty page for the next iteration */ + run_start =find_next_bit (bitmap, last +1 , run_start); + } + } +} + +/* +* Utility for the outgoing postcopy code. +* +* Discard any partially sent host-page size chunks, mark any partially +* dirty host-page size chunks as all dirty. +* +* Returns: 0 on success +*/ +static int postcopy_chunk_hostpages (MigrationState *ms) +{ +struct RAMBlock *block; + +if (qemu_host_page_size ==TARGET_PAGE_SIZE ) { +/* Easy case - TPS==HPS - nothing to be done */ +return 0 ; + } + +/* Easiest way to make sure we don't resume in the middle of a host-page */ + last_seen_block =NULL ; + last_sent_block =NULL ; + last_offset =0 ; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { +unsigned long first = block->offset >>TARGET_PAGE_BITS ; + +PostcopyDiscardState *pds = +postcopy_discard_send_init (ms, first, block->idstr); + +/* First pass: Discard all partially sent host pages */ +postcopy_chunk_hostpages_pass (ms, true, block, pds); +/* +* Second pass: Ensure that all partially dirty host pages are made +* fully dirty. +*/ +postcopy_chunk_hostpages_pass (ms, false, block, pds); + +postcopy_discard_send_finish (ms, pds); + }/* ram_list loop */ + +return 0 ; +} + +/* * Transmit the set of pages to be discarded after precopy to the target * these are pages that: * a) Have been previously transmitted but are now dirty again
@@ -1613,6 +1778,13 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms)return -EINVAL ; } +/* Deal with TPS != HPS */ + ret =postcopy_chunk_hostpages (ms); +if (ret) { +rcu_read_unlock (); +return ret; + } +/* * Update the unsentmap to be unsentmap = unsentmap | dirty */
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v9 48/56] Host page!=target page: Cleanup bitmaps
Quoted 9 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Prior to the start of postcopy, ensure that everything that will be transferred later is a whole host-page in size.
This is accomplished by discarding partially transferred host pages and marking any that are partially dirty as fully dirty.
Signed-off-by Dr. David Alan Gilbert <[email protected]>
PATCH v9 49/56 Round up RAMBlock sizes to host page sizes
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
RAMBlocks that are not a multiple of host pages in length cause problems for postcopy (I've seen an ACPI table on aarch64 be 5k in length - i.e. 5x target-page), so round RAMBlock sizes up to a host-page.
This potentially breaks migration compatibility due to changes in RAMBlock sizes; however:
1) x86 and s390 I think always have host=target page size
2) When I've tried on Power the block sizes already seem aligned.
3) I don't think there's anything else that maintains per-version machine-types for compatibility.
--- exec.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
exec.c +4 -4
@@ -1450,7 +1450,7 @@ int qemu_ram_resize(ram_addr_t base, ram_addr_t newsize, Error **errp)assert (block); - newsize =TARGET_PAGE_ALIGN (newsize); + newsize =HOST_PAGE_ALIGN (newsize);if (block->used_length == newsize) {return 0 ;
@@ -1594,7 +1594,7 @@ ram_addr_t qemu_ram_alloc_from_file(ram_addr_t size, MemoryRegion *mr,return -1 ; } - size =TARGET_PAGE_ALIGN (size); + size =HOST_PAGE_ALIGN (size); new_block =g_malloc0 (sizeof (*new_block)); new_block->mr = mr; new_block->used_length = size;
@@ -1630,8 +1630,8 @@ ram_addr_t qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,ram_addr_t addr;Error *local_err =NULL ; - size =TARGET_PAGE_ALIGN (size); - max_size =TARGET_PAGE_ALIGN (max_size); + size =HOST_PAGE_ALIGN (size); + max_size =HOST_PAGE_ALIGN (max_size); new_block =g_malloc0 (sizeof (*new_block)); new_block->mr = mr; new_block->resized = resized;
Signature · 1 line
2.5.0
PATCH v9 50/56 Postcopy; Handle userfault requests
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
userfaultfd is a Linux syscall that gives an fd that receives a stream of notifications of accesses to pages registered with it and allows the program to acknowledge those stalls and tell the accessing thread to carry on.
We convert the requests from the kernel into messages back to the source asking for the pages.
--- include/migration/migration.h | 3 + migration/postcopy-ram.c | 155 +++++++++++++++++++++++++++++++++++++++--- trace-events | 9 +++ 3 files changed, 158 insertions(+), 9 deletions(-)
include/migration/migration.h +3 -0
@@ -89,11 +89,14 @@ struct MigrationIncomingState {*/ QemuEvent main_thread_load_event; +bool have_fault_thread;QemuThread fault_thread;QemuSemaphore fault_thread_sem;/* For the kernel to send us notifications */ int userfault_fd; +/* To tell the fault_thread to quit */ +int userfault_quit_fd;QEMUFile *to_src_file;QemuMutex rp_mutex;/* We send replies from multiple threads */ void *postcopy_tmp_page;
migration/postcopy-ram.c +146 -9
198 lines not on this page · raw message
trace-events +9 -0
@@ -1554,6 +1554,15 @@ postcopy_cleanup_range(const char *ramblock, void *host_addr, size_t offset, siz postcopy_init_range(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx" postcopy_place_page(void *host_addr) "host=%p" postcopy_place_page_zero(void *host_addr) "host=%p" +postcopy_ram_enable_notify(void) "" +postcopy_ram_fault_thread_entry(void) "" +postcopy_ram_fault_thread_exit(void) "" +postcopy_ram_fault_thread_quit(void) "" +postcopy_ram_fault_thread_request(uint64_t hostaddr, const char *ramblock, size_t offset) "Request for HVA=%" PRIx64 " rb=%s offset=%zx" +postcopy_ram_incoming_cleanup_closeuf(void) "" +postcopy_ram_incoming_cleanup_entry(void) "" +postcopy_ram_incoming_cleanup_exit(void) "" +postcopy_ram_incoming_cleanup_join(void) "" # kvm-all.c kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
Signature · 1 line
2.5.0
PATCH v9 51/56 Start up a postcopy/listener thread ready for incoming page data
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
The loading of a device state (during postcopy) may access guest memory that's still on the source machine and thus might need a page fill; split off a separate thread that handles the incoming page data so that the original incoming migration code can finish off the device data.
--- include/migration/migration.h | 4 +++ migration/migration.c | 6 ++++ migration/savevm.c | 79 ++++++++++++++++++++++++++++++++++++++++++- trace-events | 2 ++ 4 files changed, 90 insertions(+), 1 deletion(-)
include/migration/migration.h +4 -0
@@ -93,6 +93,10 @@ struct MigrationIncomingState {QemuThread fault_thread;QemuSemaphore fault_thread_sem; +bool have_listen_thread; +QemuThread listen_thread; +QemuSemaphore listen_thread_sem; +/* For the kernel to send us notifications */ int userfault_fd;/* To tell the fault_thread to quit */
migration/migration.c +6 -0
@@ -1441,6 +1441,12 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running) goto fail; } +/* +* Make sure the receiver can get incoming pages before we send the rest +* of the state +*/ +qemu_savevm_send_postcopy_listen (fb); +qemu_savevm_state_complete_precopy (fb);qemu_savevm_send_ping (fb,3 );
migration/savevm.c +78 -1
97 lines not on this page · raw message
trace-events +2 -0
@@ -1220,6 +1220,8 @@ loadvm_postcopy_ram_handle_discard_end(void) "" loadvm_postcopy_ram_handle_discard_header(const char *ramid, uint16_t len) "%s: %ud" loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" loadvm_process_command_ping(uint32_t val) "%x" +postcopy_ram_listen_thread_exit(void) "" +postcopy_ram_listen_thread_start(void) "" qemu_savevm_send_postcopy_advise(void) "" qemu_savevm_send_postcopy_ram_discard(const char *id, uint16_t len) "%s: %ud" savevm_command_send(uint16_t command, uint16_t len) "com=0x%x len=%d"
Signature · 1 line
2.5.0
PATCH v9 52/56 postcopy: Wire up loadvm_postcopy_handle_ commands
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Wire up more of the handlers for the commands on the destination side, in particular loadvm_postcopy_handle_run now has enough to start the guest running.
--- migration/savevm.c | 28 +++++++++++++++++++++++++++- trace-events | 2 ++ 2 files changed, 29 insertions(+), 1 deletion(-)
migration/savevm.c +27 -1
@@ -1479,12 +1479,33 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)static int loadvm_postcopy_handle_run (MigrationIncomingState *mis) {PostcopyState ps =postcopy_state_set (POSTCOPY_INCOMING_RUNNING ); +Error *local_err =NULL ; +trace_loadvm_postcopy_handle_run ();if (ps !=POSTCOPY_INCOMING_LISTENING ) {error_report ("CMD_POSTCOPY_RUN in wrong postcopy state (%d)" , ps);return -1 ; } +/* TODO we should move all of this lot into postcopy_ram.c or a shared code +* in migration.c +*/ +cpu_synchronize_all_post_init (); + +qemu_announce_self (); + +/* Make sure all file formats flush their mutable metadata */ +bdrv_invalidate_cache_all (&local_err); +if (local_err) { +error_report_err (local_err); +return -1 ; + } + +trace_loadvm_postcopy_handle_run_cpu_sync (); +cpu_synchronize_all_post_init (); + +trace_loadvm_postcopy_handle_run_vmstart (); +if (autostart) {/* Hold onto your hats, starting the CPU */ vm_start ();
@@ -1493,7 +1514,12 @@ static int loadvm_postcopy_handle_run(MigrationIncomingState *mis)runstate_set (RUN_STATE_PAUSED ); } -return 0 ; +/* We need to finish reading the stream from the package +* and also stop reading anything more from the stream that loaded the +* package (since it's now being read by the listener thread). +* LOADVM_QUIT will quit all the layers of nested loadvm loops. +*/ +return LOADVM_QUIT ; }/**
trace-events +2 -0
@@ -1215,6 +1215,8 @@ loadvm_handle_cmd_packaged_received(int ret) "%d" loadvm_postcopy_handle_advise(void) "" loadvm_postcopy_handle_listen(void) "" loadvm_postcopy_handle_run(void) "" +loadvm_postcopy_handle_run_cpu_sync(void) "" +loadvm_postcopy_handle_run_vmstart(void) "" loadvm_postcopy_ram_handle_discard(void) "" loadvm_postcopy_ram_handle_discard_end(void) "" loadvm_postcopy_ram_handle_discard_header(const char *ramid, uint16_t len) "%s: %ud"
Signature · 1 line
2.5.0
PATCH v9 53/56 Postcopy: Mark nohugepage before discard
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Prior to servicing userfault requests we must ensure we've not got huge pages in the area that might include non-transferred memory, since a hugepage could incorrectly mark the whole huge page as present.
We mark the area as non-huge page (nhp) just before we perform discards; the discard code now tells us to discard any areas that haven't been sent (as well as any that are redirtied); any already formed transparent-huge-pages get fragmented by this discard process if they cotnain any discards.
Transparent huge pages that have been entirely transferred and don't contain any discards are not broken by this mechanism; they stay as huge pages.
By starting postcopy after a full precopy pass, many of the pages then stay as huge pages; this is important for maintaining performance after the end of the migration.
--- include/migration/postcopy-ram.h | 6 ++++++ migration/postcopy-ram.c | 46 +++++++++++++++++++++++++++++++++++++--- migration/savevm.c | 9 +++++++- trace-events | 1 + 4 files changed, 58 insertions(+), 4 deletions(-)
include/migration/postcopy-ram.h +6 -0
@@ -41,6 +41,12 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis);int postcopy_ram_discard_range (MigrationIncomingState *mis,uint8_t *start,size_t length); +/* +* Userfault requires us to mark RAM as NOHUGEPAGE prior to discard +* however leaving it until after precopy means that most of the precopy +* data is still THPd +*/ +int postcopy_ram_prepare_discard (MigrationIncomingState *mis);/* * Called at the start of each RAMBlock by the bitmap code.
migration/postcopy-ram.c +43 -3
67 lines not on this page · raw message
migration/savevm.c +8 -1
@@ -1316,7 +1316,7 @@ static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis,switch (ps) {case POSTCOPY_INCOMING_ADVISE :/* 1st discard */ - tmp =0 ;/* TODO: later patch postcopy_ram_prepare_discard(mis); */ + tmp =postcopy_ram_prepare_discard (mis);if (tmp) {return tmp; }
@@ -1448,6 +1448,13 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)error_report ("CMD_POSTCOPY_LISTEN in wrong postcopy state (%d)" , ps);return -1 ; } +if (ps ==POSTCOPY_INCOMING_ADVISE ) { +/* +* A rare case, we entered listen without having to do any discards, +* so do the setup that's normally done at the time of the 1st discard. +*/ +postcopy_ram_prepare_discard (mis); + }/* * Sensitise RAM - can now generate requests for blocks that don't exist
trace-events +1 -0
@@ -1556,6 +1556,7 @@ postcopy_discard_send_range(const char *ramblock, unsigned long start, unsigned postcopy_ram_discard_range(void *start, size_t length) "%p,+%zx" postcopy_cleanup_range(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx" postcopy_init_range(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx" +postcopy_nhp_range(const char *ramblock, void *host_addr, size_t offset, size_t length) "%s: %p offset=%zx length=%zx" postcopy_place_page(void *host_addr) "host=%p" postcopy_place_page_zero(void *host_addr) "host=%p" postcopy_ram_enable_notify(void) ""
Signature · 1 line
2.5.0
PATCH v9 54/56 End of migration for postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Tweak the end of migration cleanup; we don't want to close stuff down at the end of the main stream, since the postcopy is still sending pages on the other thread.
--- migration/migration.c | 26 +++++++++++++++++++++++++- trace-events | 6 ++++-- 2 files changed, 29 insertions(+), 3 deletions(-)
migration/migration.c +25 -1
@@ -325,13 +325,37 @@ static void process_incoming_migration_co(void *opaque) {QEMUFile *f = opaque;Error *local_err =NULL ; +MigrationIncomingState *mis; +PostcopyState ps;int ret; -migration_incoming_state_new (f); + mis =migration_incoming_state_new (f);postcopy_state_set (POSTCOPY_INCOMING_NONE );migrate_generate_event (MIGRATION_STATUS_ACTIVE ); + ret =qemu_loadvm_state (f); + ps =postcopy_state_get (); +trace_process_incoming_migration_co_end (ret, ps); +if (ps !=POSTCOPY_INCOMING_NONE ) { +if (ps ==POSTCOPY_INCOMING_ADVISE ) { +/* +* Where a migration had postcopy enabled (and thus went to advise) +* but managed to complete within the precopy period, we can use +* the normal exit. +*/ +postcopy_ram_incoming_cleanup (mis); + }else if (ret >=0 ) { +/* +* Postcopy was started, cleanup should happen at the end of the +* postcopy thread. +*/ +trace_process_incoming_migration_co_postcopy_end_main (); +return ; + } +/* Else if something went wrong then just fall out of the normal exit */ + } +qemu_fclose (f);free_xbzrle_decoded_buf ();migration_incoming_state_destroy ();
trace-events +4 -2
@@ -1480,11 +1480,13 @@ source_return_path_thread_entry(void) "" source_return_path_thread_loop_top(void) "" source_return_path_thread_pong(uint32_t val) "%x" source_return_path_thread_shut(uint32_t val) "%x" -migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64 -migrate_state_too_big(void) "" migrate_global_state_post_load(const char *state) "loaded state: %s" migrate_global_state_pre_save(const char *state) "saved state: %s" migration_thread_low_pending(uint64_t pending) "%" PRIu64 +migrate_state_too_big(void) "" +migrate_transferred(uint64_t tranferred, uint64_t time_spent, double bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %g max_size %" PRId64 +process_incoming_migration_co_end(int ret, int ps) "ret=%d postcopy-state=%d" +process_incoming_migration_co_postcopy_end_main(void) "" # migration/rdma.c qemu_rdma_accept_incoming_migration(void) ""
Signature · 1 line
2.5.0
PATCH v9 55/56 Disable mlock around incoming postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Userfault doesn't work with mlock; mlock is designed to nail down pages so they don't move, userfault is designed to tell you when they're not there.
munlock the pages we userfault protect before postcopy. mlock everything again at the end if mlock is enabled.
--- include/sysemu/sysemu.h | 1 + migration/postcopy-ram.c | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+)
include/sysemu/sysemu.h +1 -0
@@ -172,6 +172,7 @@ extern int boot_menu;extern bool boot_strict;extern uint8_t *boot_splash_filedata;extern size_t boot_splash_filedata_size; +extern bool enable_mlock;extern uint8_t qemu_extra_params_fw[2 ];extern QEMUClockType rtc_clock;extern const char *mem_path;
migration/postcopy-ram.c +24 -0
@@ -87,6 +87,11 @@ static bool ufd_version_check(int ufd)return true; } +/* +* Note: This has the side effect of munlock'ing all of RAM, that's +* normally fine since if the postcopy succeeds it gets turned back on at the +* end. +*/ bool postcopy_ram_supported_by_host (void ) {long pagesize =getpagesize ();
@@ -115,6 +120,15 @@ bool postcopy_ram_supported_by_host(void) }/* +* userfault and mlock don't go together; we'll put it back later if +* it was enabled. +*/ +if (munlockall ()) { +error_report ("%s: munlockall: %s" , __func__,strerror (errno)); +return -1 ; + } + +/* * We need to check that the ops we need are supported on anon memory * To do that we need to register a chunk and see the flags that * are returned.
@@ -294,6 +308,16 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) mis->have_fault_thread = false; } +if (enable_mlock) { +if (os_mlock () <0 ) { +error_report ("mlock: %s" ,strerror (errno)); +/* +* It doesn't feel right to fail at this point, we have a valid +* VM state. +*/ + } + } +postcopy_state_set (POSTCOPY_INCOMING_END );migrate_send_rp_shut (mis,qemu_file_get_error (mis->from_src_file) !=0 );
Signature · 1 line
2.5.0
PATCH v9 56/56 Inhibit ballooning during postcopy
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Postcopy detects accesses to pages that haven't been transferred yet using userfaultfd, and it causes exceptions on pages that are 'not present'.
Ballooning also causes pages to be marked as 'not present' when the guest inflates the balloon.
Potentially a balloon could be inflated to discard pages that are currently inflight during postcopy and that may be arriving at about the same time.
To avoid this confusion, disable ballooning during postcopy.
When disabled we drop balloon requests from the guest. Since ballooning is generally initiated by the host, the management system should avoid initiating any balloon instructions to the guest during migration, although it's not possible to know how long it would take a guest to process a request made prior to the start of migration. Guest initiated ballooning will not know if it's really freed a page of host memory or not.
Queueing the requests until after migration would be nice, but is non-trivial, since the set of inflate/deflate requests have to be compared with the state of the page to know what the final outcome is allowed to be.
--- balloon.c | 11 +++++++++++ hw/virtio/virtio-balloon.c | 4 +++- include/sysemu/balloon.h | 2 ++ migration/postcopy-ram.c | 9 +++++++++ 4 files changed, 25 insertions(+), 1 deletion(-)
balloon.c +11 -0
@@ -36,6 +36,17 @@static QEMUBalloonEvent *balloon_event_fn;static QEMUBalloonStatus *balloon_stat_fn;static void *balloon_opaque; +static bool balloon_inhibited; + +bool qemu_balloon_is_inhibited (void ) +{ +return balloon_inhibited; +} + +void qemu_balloon_inhibit (bool state) +{ + balloon_inhibited = state; +}static bool have_balloon (Error **errp) {
hw/virtio/virtio-balloon.c +3 -1
@@ -37,9 +37,11 @@static void balloon_page (void *addr,int deflate) {#if defined(__linux__) -if (!kvm_enabled () ||kvm_has_sync_mmu ()) +if (!qemu_balloon_is_inhibited () && (!kvm_enabled () || +kvm_has_sync_mmu ())) {qemu_madvise (addr,TARGET_PAGE_SIZE , deflate ?QEMU_MADV_WILLNEED :QEMU_MADV_DONTNEED ); + }#endif }
include/sysemu/balloon.h +2 -0
@@ -22,5 +22,7 @@ typedef void (QEMUBalloonStatus)(void *opaque, BalloonInfo *info);int qemu_add_balloon_handler (QEMUBalloonEvent *event_func,QEMUBalloonStatus *stat_func,void *opaque);void qemu_remove_balloon_handler (void *opaque); +bool qemu_balloon_is_inhibited (void ); +void qemu_balloon_inhibit (bool state);#endif
migration/postcopy-ram.c +9 -0
@@ -24,6 +24,7 @@#include "migration/migration.h" #include "migration/postcopy-ram.h" #include "sysemu/sysemu.h" +#include "sysemu/balloon.h" #include "qemu/error-report.h" #include "trace.h"
@@ -308,6 +309,8 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) mis->have_fault_thread = false; } +qemu_balloon_inhibit (false); +if (enable_mlock) {if (os_mlock () <0 ) {error_report ("mlock: %s" ,strerror (errno));
@@ -533,6 +536,12 @@ int postcopy_ram_enable_notify(MigrationIncomingState *mis)return -1 ; } +/* +* Ballooning can mark pages as absent while we're postcopying +* that would cause false userfaults. +*/ +qemu_balloon_inhibit (true); +trace_postcopy_ram_enable_notify ();return 0 ;
Signature · 1 line
2.5.0
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
Quoted 7 lines
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tagTesting status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)
Tested minimally on ppc64 with back and forth postcopy migration of unloaded pseries guest within the localhost - works as expected.
However I am seeing a failure in one case. I am not sure if this is a user error or a real issue in postcopy migration. If I switch to postcopy migration immediately after starting the migration, I see the migration failing with error:
qemu-system-ppc64: qemu_savevm_send_packaged: Unreasonably large packaged state: 25905005
1. (qemu) migrate_set_capability x-postcopy-ram on
2. (qemu) migrate -d tcp:localhost:4444
3. (qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on Migration status: active
total time: 4177 milliseconds
expected downtime: 300 milliseconds
setup: 75 milliseconds
transferred ram: 115523 kbytes
throughput: 155.69 mbps
remaining ram: 30117196 kbytes
total ram: 33554688 kbytes
duplicate: 835311 pages
skipped: 0 pages
normal: 24061 pages
normal bytes: 96244 kbytes
dirty sync count: 1
4. (qemu) migrate_start_postcopy
5. (qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on Migration status: failed
total time: 0 milliseconds
If I run 'info migrate' in step 3 a few more times before step 4, then the migration succeeds. So is there a way to figure out when to switch over to postcopy migration ?
QEMU cmdline: ./ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -vga none -machine pseries -m 32G,slots=32,maxmem=64G -smp 16 -device virtio-blk-pci,drive=rootdisk -drive file=/home/bharata/F20-snap1,if=none,cache=none,id=rootdisk,format=qcow2 -monitor telnet:127.0.0.1:1234,server,nowait
Host: 4.3.0-rc7+
Regards,
Bharata.
Quoted 19 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
Quoted 7 lines
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tagTesting status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)Tested minimally on ppc64 with back and forth postcopy migration of unloaded pseries guest within the localhost - works as expected.
Quoted 6 lines
However I am seeing a failure in one case. I am not sure if this is a user error or a real issue in postcopy migration. If I switch to postcopy migration immediately after starting the migration, I see the migration failing with error:
qemu-system-ppc64:
qemu_savevm_send_packaged: Unreasonably large packaged state: 25905005
I put an arbitrary limit of 16MB (see MAX_VM_CMD_PACKAGED_SIZE in include/sysemu/sysemu.h) on the size of the data accepted into the packaged blob. How big is the htab data likely to be?
Dave
Quoted 31 lines
1. (qemu)
migrate_set_capabilityx-postcopy-ram on2. (qemu) migrate -d tcp:localhost:4444
3. (qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on Migration status: active
total time: 4177 milliseconds
expected downtime: 300 milliseconds
setup: 75 milliseconds
transferred ram: 115523 kbytes
throughput: 155.69 mbps
remaining ram: 30117196 kbytes
total ram: 33554688 kbytes
duplicate: 835311 pages
skipped: 0 pages
normal: 24061 pages
normal bytes: 96244 kbytes
dirty sync count: 14. (qemu)
migrate_start_postcopy5. (qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on Migration status: failed
total time: 0 millisecondsIf I run 'info migrate' in step 3 a few more times before step 4, then the migration succeeds. So is there a way to figure out when to switch over to postcopy migration ?
Quoted 7 lines
QEMU cmdline: ./ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -vga none -machine pseries -m 32G,slots=32,maxmem=64G -smp 16 -device virtio-blk-pci,drive=rootdisk -drive file=/home/bharata/F20-snap1,if=none,cache=none,id=rootdisk,format=qcow2 -monitor telnet:127.0.0.1:1234,server,nowait
Host: 4.3.0-rc7+
Regards,
Bharata.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
Quoted 7 lines
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tagTesting status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)Tested minimally on ppc64 with back and forth postcopy migration of unloaded pseries guest within the localhost - works as expected.
However I am seeing a failure in one case. I am not sure if this is a user error or a real issue in postcopy migration. If I switch to postcopy migration immediately after starting the migration, I see the migration failing with error:
Quoted 4 lines
Quoted 1 line
qemu-system-ppc64:
qemu_savevm_send_packaged: Unreasonably large packaged state: 25905005I put an arbitrary limit of 16MB (see
MAX_VM_CMD_PACKAGED_SIZEininclude/sysemu/sysemu.h) on the size of the data accepted into the packaged blob. How big is the htab data likely to be?
HTAB size is a variable and depends on maxmem size. It will be 1/128 th of maxmem. So for a 32G guest, HTAB will be 256M in size.
Regards,
Bharata.
Quoted 27 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
Quoted 7 lines
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tagTesting status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)Tested minimally on ppc64 with back and forth postcopy migration of unloaded pseries guest within the localhost - works as expected.
However I am seeing a failure in one case. I am not sure if this is a user error or a real issue in postcopy migration. If I switch to postcopy migration immediately after starting the migration, I see the migration failing with error:
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
qemu-system-ppc64:
qemu_savevm_send_packaged: Unreasonably large packaged state: 25905005I put an arbitrary limit of 16MB (see
MAX_VM_CMD_PACKAGED_SIZEininclude/sysemu/sysemu.h) on the size of the data accepted into the packaged blob. How big is the htab data likely to be?HTAB size is a variable and depends on maxmem size. It will be 1/128 th of maxmem. So for a 32G guest, HTAB will be 256M in size.
OK, that does get a bit big.
Two possible fixes;
1 - postcopy htab (I don't know htab to know how hard that is)
2 - do one pass of iterable/non-postcopiable devices before we start the package;
I'm just writing a patch to try that; I'll send it to you to let you try once I get it to not-break normal migration.
Dave
Quoted 3 lines
Regards,
Bharata.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 36 lines · * Dr. David Alan Gilbert ([email protected]) wrote:
Quoted 27 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
Quoted 7 lines
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tagTesting status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)Tested minimally on ppc64 with back and forth postcopy migration of unloaded pseries guest within the localhost - works as expected.
However I am seeing a failure in one case. I am not sure if this is a user error or a real issue in postcopy migration. If I switch to postcopy migration immediately after starting the migration, I see the migration failing with error:
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
qemu-system-ppc64:
qemu_savevm_send_packaged: Unreasonably large packaged state: 25905005I put an arbitrary limit of 16MB (see
MAX_VM_CMD_PACKAGED_SIZEininclude/sysemu/sysemu.h) on the size of the data accepted into the packaged blob. How big is the htab data likely to be?HTAB size is a variable and depends on maxmem size. It will be 1/128 th of maxmem. So for a 32G guest, HTAB will be 256M in size.
Quoted 7 lines
OK, that does get a bit big.
Two possible fixes;
1 - postcopy htab (I don't know htab to know how hard that is)
2 - do one pass of iterable/non-postcopiable devices before we start the package;
I'm just writing a patch to try that; I'll send it to you to let you try once I get it to not-break normal migration.
Hi Bharata,
Can you try the patch below and let me know if it solves the problem;
if it doesn't, I'd be interested to know when the HTAB routines get called in the precopy/postcopy phases.
Dave
From 0f965d4dec7b188aec5324c3350704f993517cc8 Mon Sep 17 00:00:00 2001
From: "Dr. David Alan Gilbert" <[email protected]>
Date: Fri, 6 Nov 2015 12:06:16 +0000
Subject: [PATCH] Finish non-postcopiable iterative devices before package
Where we have iterable, but non-postcopiable devices (e.g. htab or block migration), complete them before forming the 'package' but with the CPUs stopped. This stops them filling up the package.
--- include/sysemu/sysemu.h | 2 +- migration/migration.c | 10 ++++++++-- migration/savevm.c | 10 ++++++++-- 3 files changed, 17 insertions(+), 5 deletions(-)
include/sysemu/sysemu.h +1 -1
@@ -112,7 +112,7 @@ void qemu_savevm_state_header(QEMUFile *f);int qemu_savevm_state_iterate (QEMUFile *f,bool postcopy);void qemu_savevm_state_cleanup (void );void qemu_savevm_state_complete_postcopy (QEMUFile *f); -void qemu_savevm_state_complete_precopy (QEMUFile *f); +void qemu_savevm_state_complete_precopy (QEMUFile *f,bool iterable_only);void qemu_savevm_state_pending (QEMUFile *f,uint64_t max_size,uint64_t *res_non_postcopiable,uint64_t *res_postcopiable);
migration/migration.c +8 -2
@@ -1429,6 +1429,12 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running) }/* +* Cause any non-postcopiable, but iterative devices to +* send out their final data. +*/ +qemu_savevm_state_complete_precopy (ms->file, true); + +/* * in Finish migrate and with the io-lock held everything should * be quiet, but we've potentially still got dirty pages and we * need to tell the destination to throw any pages it's already received
@@ -1471,7 +1477,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)*/ qemu_savevm_send_postcopy_listen (fb); -qemu_savevm_state_complete_precopy (fb); +qemu_savevm_state_complete_precopy (fb, false);qemu_savevm_send_ping (fb,3 );qemu_savevm_send_postcopy_run (fb);
@@ -1538,7 +1544,7 @@ static void migration_completion(MigrationState *s, int current_active_state, ret =vm_stop_force_state (RUN_STATE_FINISH_MIGRATE );if (ret >=0 ) {qemu_file_set_rate_limit (s->file,INT64_MAX ); -qemu_savevm_state_complete_precopy (s->file); +qemu_savevm_state_complete_precopy (s->file, false); } }qemu_mutex_unlock_iothread ();
migration/savevm.c +8 -2
@@ -1026,7 +1026,7 @@ void qemu_savevm_state_complete_postcopy(QEMUFile *f)qemu_fflush (f); } -void qemu_savevm_state_complete_precopy (QEMUFile *f) +void qemu_savevm_state_complete_precopy (QEMUFile *f,bool iterable_only) {QJSON *vmdesc;int vmdesc_len;
@@ -1041,9 +1041,11 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f) QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {if (!se->ops || (in_postcopy && se->ops->save_live_complete_postcopy) || + (in_postcopy && !iterable_only) || !se->ops->save_live_complete_precopy) {continue ; } +if (se->ops && se->ops->is_active) {if (!se->ops->is_active (se->opaque)) {continue ;
@@ -1062,6 +1064,10 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f) } } +if (iterable_only) { +return ; + } + vmdesc =qjson_new ();json_prop_int (vmdesc,"page_size" ,TARGET_PAGE_SIZE );json_start_array (vmdesc,"devices" );
@@ -1176,7 +1182,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) ret =qemu_file_get_error (f);if (ret ==0 ) { -qemu_savevm_state_complete_precopy (f); +qemu_savevm_state_complete_precopy (f, false); ret =qemu_file_get_error (f); }if (ret !=0 ) {
Signature · 4 lines
2.5.0 -- Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 52 lines · Dr. David Alan Gilbert
Quoted 36 lines · * Dr. David Alan Gilbert ([email protected]) wrote:
Quoted 27 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 25 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
Quoted 7 lines
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tagTesting status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)Tested minimally on ppc64 with back and forth postcopy migration of unloaded pseries guest within the localhost - works as expected.
However I am seeing a failure in one case. I am not sure if this is a user error or a real issue in postcopy migration. If I switch to postcopy migration immediately after starting the migration, I see the migration failing with error:
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
qemu-system-ppc64:
qemu_savevm_send_packaged: Unreasonably large packaged state: 25905005I put an arbitrary limit of 16MB (see
MAX_VM_CMD_PACKAGED_SIZEininclude/sysemu/sysemu.h) on the size of the data accepted into the packaged blob. How big is the htab data likely to be?HTAB size is a variable and depends on maxmem size. It will be 1/128 th of maxmem. So for a 32G guest, HTAB will be 256M in size.
Quoted 7 lines
OK, that does get a bit big.
Two possible fixes;
1 - postcopy htab (I don't know htab to know how hard that is)
2 - do one pass of iterable/non-postcopiable devices before we start the package;
I'm just writing a patch to try that; I'll send it to you to let you try once I get it to not-break normal migration.Hi Bharata,
Can you try the patch below and let me know if it solves the problem;
if it doesn't, I'd be interested to know when the HTAB routines get called in the precopy/postcopy phases.Dave
Quoted 8 lines
Quoted 1 line
From
0f965d4dec7b188aec5324c3350704f993517cc8Mon Sep 17 00:00:00 2001From: "Dr. David Alan Gilbert" <[email protected]>
Date: Fri, 6 Nov 2015 12:06:16 +0000
Subject: [PATCH] Finish non-postcopiable iterative devices before packageWhere we have iterable, but non-postcopiable devices (e.g. htab or block migration), complete them before forming the 'package' but with the CPUs stopped. This stops them filling up the package.
That helps and the migration suceeds now when I switch to postcopy immediately after starting the migration.
However after postcopy migration, when I attempt to start an incoming instance again to migrate the guest back, I see this failure:
qemu-system-ppc64: cannot set up guest memory 'ppc_spapr.ram': Cannot allocate memory
The same doesn't happen with normal migration.
I will see if I can debug this more tomorrow.
Regards,
Bharata.
Quoted 6 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 3 lines
Where we have iterable, but non-postcopiable devices (e.g. htab or block migration), complete them before forming the 'package' but with the CPUs stopped. This stops them filling up the package.
That helps and the migration suceeds now when I switch to postcopy immediately after starting the migration.
Excellent.
Quoted 6 lines
However after postcopy migration, when I attempt to start an incoming instance again to migrate the guest back, I see this failure:
qemu-system-ppc64: cannot set up guest memory 'ppc_spapr.ram': Cannot allocate memory
The same doesn't happen with normal migration.
Huh that's fun; that's the original source guest that's running out of RAM? It's original QEMU should be gone by that point.
Quoted 2 lines
I will see if I can debug this more tomorrow.
Thanks.
Dave
Quoted 4 lines
Regards,
Bharata.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 14 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 3 lines
Where we have iterable, but non-postcopiable devices (e.g. htab or block migration), complete them before forming the 'package' but with the CPUs stopped. This stops them filling up the package.
That helps and the migration suceeds now when I switch to postcopy immediately after starting the migration.
Excellent.
Quoted 3 lines
However after postcopy migration, when I attempt to start an incoming instance again to migrate the guest back, I see this failure:
Quoted 6 lines
Quoted 3 lines
qemu-system-ppc64: cannot set up guest memory 'ppc_spapr.ram': Cannot allocate memory
The same doesn't happen with normal migration.
Huh that's fun; that's the original source guest that's running out of RAM? It's original QEMU should be gone by that point.
Yes, the original source QEMU is gone, but there is not enough memory left in the host to start another incoming QEMU instance because...
At the beginning
-----------------
$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 128781632 kB MemAvailable: 131668224 kB
After starting the guest (-m 64G,slots=32,maxmem=128G)
------------------------
$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 124866880 kB MemAvailable: 127753728 kB
After starting the destination instance (incoming)
-------------------------------------------------
$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 122514880 kB MemAvailable: 125401920 kB
After postcopy migration completes
----------------------------------
$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 55150592 kB MemAvailable: 58037888 kB
After terminating the source instance
-------------------------------------
$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 59432448 kB MemAvailable: 62319872 kB
So as you can see, postcopy migration will result in guest claiming its entire RAM memory from host. This doesn't happen during normal migration.
The above results are with the additional patch you sent in this thread and I was switching to postcopy migration immediately after starting the migration. Without this patch, when I delay the switching to postcopy migration a bit, I see the free memory as below when postcopy migration completes.
$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 85018176 kB MemAvailable: 87937024 kB
Regards,
Bharata.
Quoted 17 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 16 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 3 lines
Where we have iterable, but non-postcopiable devices (e.g. htab or block migration), complete them before forming the 'package' but with the CPUs stopped. This stops them filling up the package.
That helps and the migration suceeds now when I switch to postcopy immediately after starting the migration.
Excellent.
Quoted 5 lines
However after postcopy migration, when I attempt to start an incoming instance again to migrate the guest back, I see this failure:
qemu-system-ppc64: cannot set up guest memory 'ppc_spapr.ram': Cannot allocate memory
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
The same doesn't happen with normal migration.
Huh that's fun; that's the original source guest that's running out of RAM? It's original QEMU should be gone by that point.
Yes, the original source QEMU is gone, but there is not enough memory left in the host to start another incoming QEMU instance because...
Ah, so this is ping-pong on one host?
Quoted 30 lines
At the beginning
-----------------$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 128781632 kB MemAvailable: 131668224 kBAfter starting the guest (-m 64G,slots=32,maxmem=128G)
------------------------$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 124866880 kB MemAvailable: 127753728 kBAfter starting the destination instance (incoming)
-------------------------------------------------$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 122514880 kB MemAvailable: 125401920 kBAfter postcopy migration completes
----------------------------------$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 55150592 kB MemAvailable: 58037888 kBAfter terminating the source instance
-------------------------------------
Quoted 7 lines
$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 59432448 kB MemAvailable: 62319872 kBSo as you can see, postcopy migration will result in guest claiming its entire RAM memory from host. This doesn't happen during normal migration.
I'll try and see if I can replicate this.
Can you :
1) show me the command line you're using
2) Show me /proc/pid/smaps for the destination qemu
3) Turn on the trace postcopy_place_page_zero
the theory is that most of your pages should end up as zero pages and not be allocated.
Dave
Quoted 6 lines
The above results are with the additional patch you sent in this thread and I was switching to postcopy migration immediately after starting the migration. Without this patch, when I delay the switching to postcopy migration a bit, I see the free memory as below when postcopy migration completes.
Quoted 8 lines
$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 85018176 kB MemAvailable: 87937024 kBRegards,
Bharata.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 20 lines · Dr. David Alan Gilbert
Quoted 19 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 18 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 3 lines
Where we have iterable, but non-postcopiable devices (e.g. htab or block migration), complete them before forming the 'package' but with the CPUs stopped. This stops them filling up the package.
That helps and the migration suceeds now when I switch to postcopy immediately after starting the migration.
Excellent.
Quoted 6 lines
However after postcopy migration, when I attempt to start an incoming instance again to migrate the guest back, I see this failure:
qemu-system-ppc64: cannot set up guest memory 'ppc_spapr.ram': Cannot allocate memory
The same doesn't happen with normal migration.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Huh that's fun; that's the original source guest that's running out of RAM? It's original QEMU should be gone by that point.
Yes, the original source QEMU is gone, but there is not enough memory left in the host to start another incoming QEMU instance because...
Ah, so this is ping-pong on one host?
Yes, migrating within the localhost.
Quoted 36 lines
Quoted 36 lines
At the beginning
-----------------$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 128781632 kB MemAvailable: 131668224 kBAfter starting the guest (-m 64G,slots=32,maxmem=128G)
------------------------$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 124866880 kB MemAvailable: 127753728 kBAfter starting the destination instance (incoming)
-------------------------------------------------$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 122514880 kB MemAvailable: 125401920 kBAfter postcopy migration completes
----------------------------------$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 55150592 kB MemAvailable: 58037888 kBAfter terminating the source instance
-------------------------------------$ grep -i mem /proc/meminfo MemTotal: 132816832 kB MemFree: 59432448 kB MemAvailable: 62319872 kB
Quoted 7 lines
Quoted 2 lines
So as you can see, postcopy migration will result in guest claiming its entire RAM memory from host. This doesn't happen during normal migration.
I'll try and see if I can replicate this.
Can you :
1) show me the command line you're using
./ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -vga none -machine pseries -m 64G,slots=32,maxmem=128G -smp 16 -device virtio-blk-pci,drive=rootdisk -drive file=/home/bharata/F20-snap1,if=none,cache=none,id=rootdisk,format=qcow2 -monitor telnet:127.0.0.1:1234,server,nowait -trace events=my-trace-events
Quoted 1 line
2) Show me
/proc/pid/smapsfor the destination qemu
Attached.
Quoted 3 lines
3) Turn on the trace
postcopy_place_page_zero
the theory is that most of your pages should end up as zero pages and not be allocated.
No hits for postcopy_place_page_zero either in source or destination QEMU.
Regards,
Bharata.
10000000-10850000 r-xp 00000000 08:10 35919361 /home/bharata/qemu-gilbert/qemu/ppc64-softmmu/qemu-system-ppc64
Size: 8512 kB Rss: 6912 kB Pss: 3456 kB Shared_Clean: 6912 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 6912 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me dw 10850000-10860000 r--p 00840000 08:10 35919361 /home/bharata/qemu-gilbert/qemu/ppc64-softmmu/qemu-system-ppc64 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me dw ac 10860000-108e0000 rw-p 00850000 08:10 35919361 /home/bharata/qemu-gilbert/qemu/ppc64-softmmu/qemu-system-ppc64 Size: 512 kB Rss: 448 kB Pss: 352 kB Shared_Clean: 192 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 256 kB Referenced: 448 kB Anonymous: 256 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me dw ac 108e0000-10db0000 rw-p 00000000 00:00 0 Size: 4928 kB Rss: 320 kB Pss: 320 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 320 kB Referenced: 320 kB Anonymous: 320 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 10009980000-10015e40000 rw-p 00000000 00:00 0 [heap] Size: 201472 kB Rss: 32576 kB Pss: 32576 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 32576 kB Referenced: 32576 kB Anonymous: 32576 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fee1c000000-3fee1c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee1c030000-3fee20000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee24000000-3fee24030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee24030000-3fee28000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee28000000-3fee28030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee28030000-3fee2c000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee2c000000-3fee2c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee2c030000-3fee30000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee30000000-3fee30030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee30030000-3fee34000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee34000000-3fee34030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee34030000-3fee38000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee38000000-3fee38030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee38030000-3fee3c000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee3c000000-3fee3c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee3c030000-3fee40000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee40000000-3fee40030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee40030000-3fee44000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee45800000-3fee45810000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fee45810000-3fee46000000 rw-p 00000000 00:00 0 [stack:55877] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fee46000000-3fee46010000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fee46010000-3fee46800000 rw-p 00000000 00:00 0 [stack:55878] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fee46800000-3fee46810000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fee46810000-3fee47000000 rw-p 00000000 00:00 0 Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fee47000000-3fee47010000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fee47010000-3fee47800000 rw-p 00000000 00:00 0 Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fee47800000-3fee47810000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fee47810000-3fee48000000 rw-p 00000000 00:00 0 Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fee48000000-3fee48030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee48030000-3fee4c000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee4c000000-3fee4c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee4c030000-3fee50000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee50000000-3fee50030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee50030000-3fee54000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee54000000-3fee54030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee54030000-3fee58000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee58000000-3fee58030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee58030000-3fee5c000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee5c000000-3fee5c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee5c030000-3fee60000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee60000000-3fee60030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee60030000-3fee64000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee64000000-3fee64030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee64030000-3fee68000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee6c000000-3fee6c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee6c030000-3fee70000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee70000000-3fee70030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee70030000-3fee74000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee74000000-3fee74030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee74030000-3fee78000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee78000000-3fee78030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee78030000-3fee7c000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee7c000000-3fee7c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee7c030000-3fee80000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee84000000-3fee84030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee84030000-3fee88000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee88000000-3fee88030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee88030000-3fee8c000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee8c000000-3fee8c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee8c030000-3fee90000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee90000000-3fee90030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee90030000-3fee94000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee94000000-3fee94030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee94030000-3fee98000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee98000000-3fee98030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee98030000-3fee9c000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fee9c000000-3fee9c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fee9c030000-3feea0000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feea4000000-3feea4030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feea4030000-3feea8000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feea8000000-3feea8030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feea8030000-3feeac000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feeac000000-3feeac030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feeac030000-3feeb0000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feeb0000000-3feeb0030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feeb0030000-3feeb4000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feeb4000000-3feeb4030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feeb4030000-3feeb8000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feeb8000000-3feeb8030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feeb8030000-3feebc000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feebc000000-3feebc030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feebc030000-3feec0000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feec0000000-3feec0030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feec0030000-3feec4000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feec8000000-3feec8030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feec8030000-3feecc000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feecc000000-3feecc030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feecc030000-3feed0000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feed0000000-3feed0030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feed0030000-3feed4000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feed4000000-3feed4030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feed4030000-3feed8000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feed8000000-3feed8030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feed8030000-3feedc000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feedc000000-3feedc030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feedc030000-3feee0000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feee0000000-3feee0030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feee0030000-3feee4000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feee4000000-3feee4030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feee4030000-3feee8000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feeec000000-3feeec030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feeec030000-3feef0000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feef0000000-3feef0030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feef0030000-3feef4000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feef4000000-3feef4030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feef4030000-3feef8000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feef8000000-3feef8030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feef8030000-3feefc000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3feefc000000-3feefc030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3feefc030000-3fef00000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fef00000000-3fef00030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fef00030000-3fef04000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fef04000000-3fef04030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fef04030000-3fef08000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fef0c000000-3fef0c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fef0c030000-3fef10000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fef10000000-3fef10030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fef10030000-3fef14000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fef14000000-3fef14040000 rw-p 00000000 00:00 0 Size: 256 kB Rss: 192 kB Pss: 192 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 192 kB Referenced: 192 kB Anonymous: 192 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fef14040000-3fef18000000 ---p 00000000 00:00 0 Size: 65280 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fef18000000-3fef18030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fef18030000-3fef1c000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fef1c000000-3fef1c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fef1c030000-3fef20000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fef23ff0000-3fff23ff0000 rw-p 00000000 00:00 0 Size: 67108864 kB Rss: 67108864 kB Pss: 67108864 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 67108864 kB Referenced: 67108864 kB Anonymous: 67108864 kB AnonHugePages: 32768 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me dc ac hg mg 3fff23ff0000-3fff24000000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me 3fff24000000-3fff24030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff24030000-3fff28000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff28000000-3fff28030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff28030000-3fff2c000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff2c000000-3fff2c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff2c030000-3fff30000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff30000000-3fff30030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff30030000-3fff34000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff344d0000-3fff344e0000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff344e0000-3fff34cd0000 rw-p 00000000 00:00 0 Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff354d0000-3fff354e0000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff354e0000-3fff35fd0000 rw-p 00000000 00:00 0 [stack:55378] Size: 11200 kB Rss: 2176 kB Pss: 2176 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 2176 kB Referenced: 2176 kB Anonymous: 2176 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff35fd0000-3fff361d0000 rw-s 00000000 00:0b 10275 anon_inode:kvm-spapr-tce Size: 2048 kB Rss: 2048 kB Pss: 2048 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 2048 kB Referenced: 2048 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff361d0000-3fff36800000 rw-p 00000000 00:00 0 Size: 6336 kB Rss: 6336 kB Pss: 6336 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 6336 kB Referenced: 6336 kB Anonymous: 6336 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff36800000-3fff36810000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff36810000-3fff37000000 rw-p 00000000 00:00 0 [stack:55361] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff37000000-3fff37010000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff37010000-3fff37800000 rw-p 00000000 00:00 0 [stack:55360] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff37800000-3fff37810000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff37810000-3fff38000000 rw-p 00000000 00:00 0 [stack:55359] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff38000000-3fff38030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff38030000-3fff3c000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff3c000000-3fff3c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff3c030000-3fff40000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff40000000-3fff40030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff40030000-3fff44000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff44000000-3fff44030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff44030000-3fff48000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff48000000-3fff48030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 128 kB Pss: 128 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 128 kB Referenced: 128 kB Anonymous: 128 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff48030000-3fff4c000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff4c000000-3fff4c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff4c030000-3fff50000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff50000000-3fff50030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff50030000-3fff54000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff54000000-3fff54030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff54030000-3fff58000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff58000000-3fff58010000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff58010000-3fff58800000 rw-p 00000000 00:00 0 [stack:55358] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff58800000-3fff58810000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff58810000-3fff59000000 rw-p 00000000 00:00 0 [stack:55357] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff59000000-3fff59010000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff59010000-3fff59800000 rw-p 00000000 00:00 0 [stack:55356] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff59800000-3fff59810000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff59810000-3fff5a000000 rw-p 00000000 00:00 0 [stack:55355] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff5a000000-3fff5a010000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff5a010000-3fff5a800000 rw-p 00000000 00:00 0 [stack:55354] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff5a800000-3fff5a810000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff5a810000-3fff5b000000 rw-p 00000000 00:00 0 [stack:55353] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff5b000000-3fff5b010000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff5b010000-3fff5b800000 rw-p 00000000 00:00 0 [stack:55352] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff5b800000-3fff5b810000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff5b810000-3fff5c000000 rw-p 00000000 00:00 0 [stack:55351] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff5c000000-3fff5c030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff5c030000-3fff60000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff60000000-3fff60030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff60030000-3fff64000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff64000000-3fff64030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff64030000-3fff68000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff68000000-3fff68030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff68030000-3fff6c000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff6c000000-3fff6c040000 rw-p 00000000 00:00 0 Size: 256 kB Rss: 192 kB Pss: 192 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 192 kB Referenced: 192 kB Anonymous: 192 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff6c040000-3fff70000000 ---p 00000000 00:00 0 Size: 65280 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff70090000-3fff70110000 rw-s 00000000 00:0b 10275 anon_inode:kvm-spapr-tce Size: 512 kB Rss: 512 kB Pss: 512 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 512 kB Referenced: 512 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff70110000-3fff70190000 rw-s 00000000 00:0b 10275 anon_inode:kvm-spapr-tce Size: 512 kB Rss: 512 kB Pss: 512 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 512 kB Referenced: 512 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff70190000-3fff701b0000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff701b0000-3fff701d0000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff701d0000-3fff701f0000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff701f0000-3fff70210000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff70210000-3fff70230000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff70230000-3fff70250000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff70250000-3fff70270000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff70270000-3fff70290000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff70290000-3fff702b0000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff702b0000-3fff702d0000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff702d0000-3fff702f0000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff702f0000-3fff70310000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff70310000-3fff70320000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff70320000-3fff70b10000 rw-p 00000000 00:00 0 [stack:55350] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff70b10000-3fff70b30000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff70b30000-3fff70b40000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff70b40000-3fff71330000 rw-p 00000000 00:00 0 [stack:55349] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff71330000-3fff71340000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff71340000-3fff71b30000 rw-p 00000000 00:00 0 [stack:55348] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff71b30000-3fff71b40000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff71b40000-3fff72330000 rw-p 00000000 00:00 0 [stack:55347] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff72330000-3fff72340000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff72340000-3fff73800000 rw-p 00000000 00:00 0 [stack:55346] Size: 21248 kB Rss: 1664 kB Pss: 1664 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 1664 kB Referenced: 1664 kB Anonymous: 1664 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff73800000-3fff73810000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff73810000-3fff74000000 rw-p 00000000 00:00 0 [stack:55377] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff74000000-3fff74030000 rw-p 00000000 00:00 0 Size: 192 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me nr 3fff74030000-3fff78000000 ---p 00000000 00:00 0 Size: 65344 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me nr 3fff78000000-3fff78020000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff78020000-3fff78040000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff78040000-3fff78060000 rw-s 00000000 00:0b 10275 anon_inode:kvm-vcpu Size: 128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms 3fff78060000-3fff780c0000 rw-p 00000000 00:00 0 Size: 384 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff78460000-3fff78680000 rw-p 00000000 00:00 0 Size: 2176 kB Rss: 256 kB Pss: 256 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 256 kB Referenced: 256 kB Anonymous: 256 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff78680000-3fff78690000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff78690000-3fff78e80000 rw-p 00000000 00:00 0 [stack:55344] Size: 8128 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff78e80000-3fff78e90000 ---p 00000000 00:00 0 Size: 64 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: mr mw me ac 3fff78e90000-3fff79690000 rw-p 00000000 00:00 0 [stack:55343] Size: 8192 kB Rss: 128 kB Pss: 128 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 128 kB Referenced: 128 kB Anonymous: 128 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79690000-3fff796d0000 r-xp 00000000 fd:02 1181273 /usr/lib64/liblzma.so.5.0.99 Size: 256 kB Rss: 64 kB Pss: 2 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff796d0000-3fff796e0000 r--p 00030000 fd:02 1181273 /usr/lib64/liblzma.so.5.0.99 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff796e0000-3fff796f0000 rw-p 00040000 fd:02 1181273 /usr/lib64/liblzma.so.5.0.99 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff796f0000-3fff79740000 r-xp 00000000 fd:02 1181276 /usr/lib64/libpcre.so.1.2.0 Size: 320 kB Rss: 64 kB Pss: 2 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79740000-3fff79750000 r--p 00040000 fd:02 1181276 /usr/lib64/libpcre.so.1.2.0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79750000-3fff79760000 rw-p 00050000 fd:02 1181276 /usr/lib64/libpcre.so.1.2.0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79760000-3fff79790000 r-xp 00000000 fd:02 1181286 /usr/lib64/libselinux.so.1 Size: 192 kB Rss: 128 kB Pss: 4 kB Shared_Clean: 128 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 128 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79790000-3fff797a0000 r--p 00020000 fd:02 1181286 /usr/lib64/libselinux.so.1 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff797a0000-3fff797b0000 rw-p 00030000 fd:02 1181286 /usr/lib64/libselinux.so.1 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff797b0000-3fff797c0000 r-xp 00000000 fd:02 1181469 /usr/lib64/libffi.so.6.0.1 Size: 64 kB Rss: 64 kB Pss: 9 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff797c0000-3fff797d0000 r--p 00000000 fd:02 1181469 /usr/lib64/libffi.so.6.0.1 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff797d0000-3fff797e0000 rw-p 00010000 fd:02 1181469 /usr/lib64/libffi.so.6.0.1 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff797e0000-3fff797f0000 r-xp 00000000 fd:02 1181074 /usr/lib64/libfreebl3.so Size: 64 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff797f0000-3fff79800000 r--p 00000000 fd:02 1181074 /usr/lib64/libfreebl3.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79800000-3fff79810000 rw-p 00010000 fd:02 1181074 /usr/lib64/libfreebl3.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79810000-3fff79820000 r-xp 00000000 fd:02 1181743 /usr/lib64/libkeyutils.so.1.5 Size: 64 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79820000-3fff79830000 r--p 00000000 fd:02 1181743 /usr/lib64/libkeyutils.so.1.5 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79830000-3fff79840000 rw-p 00010000 fd:02 1181743 /usr/lib64/libkeyutils.so.1.5 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79840000-3fff79850000 r-xp 00000000 fd:02 1187467 /usr/lib64/libkrb5support.so.0.1 Size: 64 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79850000-3fff79860000 r--p 00010000 fd:02 1187467 /usr/lib64/libkrb5support.so.0.1 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79860000-3fff79870000 rw-p 00020000 fd:02 1187467 /usr/lib64/libkrb5support.so.0.1 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79870000-3fff79900000 r-xp 00000000 fd:02 1181808 /usr/lib64/libgmp.so.10.2.0 Size: 576 kB Rss: 512 kB Pss: 73 kB Shared_Clean: 512 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 512 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79900000-3fff79910000 r--p 00080000 fd:02 1181808 /usr/lib64/libgmp.so.10.2.0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79910000-3fff79920000 rw-p 00090000 fd:02 1181808 /usr/lib64/libgmp.so.10.2.0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79920000-3fff79950000 r-xp 00000000 fd:02 1181970 /usr/lib64/libhogweed.so.2.5 Size: 192 kB Rss: 128 kB Pss: 18 kB Shared_Clean: 128 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 128 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79950000-3fff79960000 r--p 00020000 fd:02 1181970 /usr/lib64/libhogweed.so.2.5 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79960000-3fff79970000 rw-p 00030000 fd:02 1181970 /usr/lib64/libhogweed.so.2.5 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79970000-3fff79990000 r-xp 00000000 fd:02 1181946 /usr/lib64/libtasn1.so.6.2.3 Size: 128 kB Rss: 128 kB Pss: 18 kB Shared_Clean: 128 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 128 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79990000-3fff799a0000 r--p 00010000 fd:02 1181946 /usr/lib64/libtasn1.so.6.2.3 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff799a0000-3fff799b0000 rw-p 00020000 fd:02 1181946 /usr/lib64/libtasn1.so.6.2.3 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff799b0000-3fff79a40000 r-xp 00000000 fd:02 1191079 /usr/lib64/libtspi.so.1.2.0 Size: 576 kB Rss: 256 kB Pss: 38 kB Shared_Clean: 256 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 256 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79a40000-3fff79a50000 r--p 00080000 fd:02 1191079 /usr/lib64/libtspi.so.1.2.0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79a50000-3fff79a60000 rw-p 00090000 fd:02 1191079 /usr/lib64/libtspi.so.1.2.0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79a60000-3fff79ab0000 r-xp 00000000 fd:02 1181750 /usr/lib64/libp11-kit.so.0.0.0 Size: 320 kB Rss: 192 kB Pss: 26 kB Shared_Clean: 192 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 192 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79ab0000-3fff79ac0000 r--p 00050000 fd:02 1181750 /usr/lib64/libp11-kit.so.0.0.0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79ac0000-3fff79ad0000 rw-p 00060000 fd:02 1181750 /usr/lib64/libp11-kit.so.0.0.0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79ad0000-3fff79ae0000 r-xp 00000000 fd:02 1181117 /usr/lib64/libcrypt-2.17.so Size: 64 kB Rss: 64 kB Pss: 3 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79ae0000-3fff79af0000 r--p 00000000 fd:02 1181117 /usr/lib64/libcrypt-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79af0000-3fff79b00000 rw-p 00010000 fd:02 1181117 /usr/lib64/libcrypt-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79b00000-3fff79b20000 rw-p 00000000 00:00 0 Size: 128 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79b20000-3fff79b40000 r-xp 00000000 fd:02 1181141 /usr/lib64/libresolv-2.17.so Size: 128 kB Rss: 64 kB Pss: 3 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79b40000-3fff79b50000 r--p 00010000 fd:02 1181141 /usr/lib64/libresolv-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79b50000-3fff79b60000 rw-p 00020000 fd:02 1181141 /usr/lib64/libresolv-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79b60000-3fff79be0000 r-xp 00000000 fd:02 1194938 /usr/lib64/libssl.so.1.0.1e Size: 512 kB Rss: 128 kB Pss: 17 kB Shared_Clean: 128 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 128 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79be0000-3fff79bf0000 r--p 00070000 fd:02 1194938 /usr/lib64/libssl.so.1.0.1e Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79bf0000-3fff79c00000 rw-p 00080000 fd:02 1194938 /usr/lib64/libssl.so.1.0.1e Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79c00000-3fff79c10000 rw-p 00000000 00:00 0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79c10000-3fff79e60000 r-xp 00000000 fd:02 1187475 /usr/lib64/libcrypto.so.1.0.1e Size: 2368 kB Rss: 1728 kB Pss: 107 kB Shared_Clean: 1728 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 1728 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79e60000-3fff79e80000 r--p 00240000 fd:02 1187475 /usr/lib64/libcrypto.so.1.0.1e Size: 128 kB Rss: 128 kB Pss: 128 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 128 kB Referenced: 128 kB Anonymous: 128 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79e80000-3fff79e90000 rw-p 00260000 fd:02 1187475 /usr/lib64/libcrypto.so.1.0.1e Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79e90000-3fff79ea0000 r-xp 00000000 fd:02 1181806 /usr/lib64/libboost_system-mt.so.1.53.0 Size: 64 kB Rss: 64 kB Pss: 12 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79ea0000-3fff79eb0000 r--p 00000000 fd:02 1181806 /usr/lib64/libboost_system-mt.so.1.53.0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79eb0000-3fff79ec0000 rw-p 00010000 fd:02 1181806 /usr/lib64/libboost_system-mt.so.1.53.0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79ec0000-3fff79ee0000 r-xp 00000000 fd:02 1181975 /usr/lib64/libboost_thread-mt.so.1.53.0 Size: 128 kB Rss: 128 kB Pss: 25 kB Shared_Clean: 128 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 128 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79ee0000-3fff79ef0000 r--p 00010000 fd:02 1181975 /usr/lib64/libboost_thread-mt.so.1.53.0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79ef0000-3fff79f00000 rw-p 00020000 fd:02 1181975 /usr/lib64/libboost_thread-mt.so.1.53.0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79f00000-3fff79f60000 r-xp 00000000 fd:02 1196042 /usr/lib64/libldap-2.4.so.2.10.2 Size: 384 kB Rss: 128 kB Pss: 8 kB Shared_Clean: 128 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 128 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79f60000-3fff79f70000 r--p 00050000 fd:02 1196042 /usr/lib64/libldap-2.4.so.2.10.2 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79f70000-3fff79f80000 rw-p 00060000 fd:02 1196042 /usr/lib64/libldap-2.4.so.2.10.2 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79f80000-3fff79fa0000 r-xp 00000000 fd:02 1196041 /usr/lib64/liblber-2.4.so.2.10.2 Size: 128 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79fa0000-3fff79fb0000 r--p 00010000 fd:02 1196041 /usr/lib64/liblber-2.4.so.2.10.2 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79fb0000-3fff79fc0000 rw-p 00020000 fd:02 1196041 /usr/lib64/liblber-2.4.so.2.10.2 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79fc0000-3fff79fd0000 r-xp 00000000 fd:02 1181312 /usr/lib64/libcom_err.so.2.1 Size: 64 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff79fd0000-3fff79fe0000 r--p 00000000 fd:02 1181312 /usr/lib64/libcom_err.so.2.1 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff79fe0000-3fff79ff0000 rw-p 00010000 fd:02 1181312 /usr/lib64/libcom_err.so.2.1 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff79ff0000-3fff7a030000 r-xp 00000000 fd:02 1187455 /usr/lib64/libk5crypto.so.3.1 Size: 256 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a030000-3fff7a040000 r--p 00030000 fd:02 1187455 /usr/lib64/libk5crypto.so.3.1 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a040000-3fff7a050000 rw-p 00040000 fd:02 1187455 /usr/lib64/libk5crypto.so.3.1 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a050000-3fff7a150000 r-xp 00000000 fd:02 1187465 /usr/lib64/libkrb5.so.3.3 Size: 1024 kB Rss: 192 kB Pss: 12 kB Shared_Clean: 192 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 192 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a150000-3fff7a160000 r--p 000f0000 fd:02 1187465 /usr/lib64/libkrb5.so.3.3 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a160000-3fff7a170000 rw-p 00100000 fd:02 1187465 /usr/lib64/libkrb5.so.3.3 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a170000-3fff7a1d0000 r-xp 00000000 fd:02 1187451 /usr/lib64/libgssapi_krb5.so.2.2 Size: 384 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a1d0000-3fff7a1e0000 r--p 00050000 fd:02 1187451 /usr/lib64/libgssapi_krb5.so.2.2 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a1e0000-3fff7a1f0000 rw-p 00060000 fd:02 1187451 /usr/lib64/libgssapi_krb5.so.2.2 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a1f0000-3fff7a200000 r-xp 00000000 fd:02 1182381 /usr/lib64/libdl-2.17.so Size: 64 kB Rss: 64 kB Pss: 1 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a200000-3fff7a210000 r--p 00000000 fd:02 1182381 /usr/lib64/libdl-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a210000-3fff7a220000 rw-p 00010000 fd:02 1182381 /usr/lib64/libdl-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a220000-3fff7a270000 r-xp 00000000 fd:02 1181360 /usr/lib64/libnspr4.so Size: 320 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a270000-3fff7a280000 r--p 00040000 fd:02 1181360 /usr/lib64/libnspr4.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a280000-3fff7a290000 rw-p 00050000 fd:02 1181360 /usr/lib64/libnspr4.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a290000-3fff7a2a0000 r-xp 00000000 fd:02 1195068 /usr/lib64/libplc4.so Size: 64 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a2a0000-3fff7a2b0000 r--p 00000000 fd:02 1195068 /usr/lib64/libplc4.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a2b0000-3fff7a2c0000 rw-p 00010000 fd:02 1195068 /usr/lib64/libplc4.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a2c0000-3fff7a2d0000 r-xp 00000000 fd:02 1195069 /usr/lib64/libplds4.so Size: 64 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a2d0000-3fff7a2e0000 r--p 00000000 fd:02 1195069 /usr/lib64/libplds4.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a2e0000-3fff7a2f0000 rw-p 00010000 fd:02 1195069 /usr/lib64/libplds4.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a2f0000-3fff7a320000 r-xp 00000000 fd:02 1181445 /usr/lib64/libnssutil3.so Size: 192 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a320000-3fff7a330000 r--p 00030000 fd:02 1181445 /usr/lib64/libnssutil3.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a330000-3fff7a340000 rw-p 00040000 fd:02 1181445 /usr/lib64/libnssutil3.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a340000-3fff7a490000 r-xp 00000000 fd:02 1187564 /usr/lib64/libnss3.so Size: 1344 kB Rss: 128 kB Pss: 8 kB Shared_Clean: 128 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 128 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a490000-3fff7a4a0000 r--p 00150000 fd:02 1187564 /usr/lib64/libnss3.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a4a0000-3fff7a4b0000 rw-p 00160000 fd:02 1187564 /usr/lib64/libnss3.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a4b0000-3fff7a4e0000 r-xp 00000000 fd:02 1196022 /usr/lib64/libsmime3.so Size: 192 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a4e0000-3fff7a4f0000 r--p 00020000 fd:02 1196022 /usr/lib64/libsmime3.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a4f0000-3fff7a500000 rw-p 00030000 fd:02 1196022 /usr/lib64/libsmime3.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a500000-3fff7a550000 r-xp 00000000 fd:02 1196023 /usr/lib64/libssl3.so Size: 320 kB Rss: 64 kB Pss: 4 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a550000-3fff7a560000 r--p 00040000 fd:02 1196023 /usr/lib64/libssl3.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a560000-3fff7a570000 rw-p 00050000 fd:02 1196023 /usr/lib64/libssl3.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a570000-3fff7a5b0000 r-xp 00000000 fd:02 1181563 /usr/lib64/libidn.so.11.6.11 Size: 256 kB Rss: 64 kB Pss: 7 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a5b0000-3fff7a5c0000 r--p 00030000 fd:02 1181563 /usr/lib64/libidn.so.11.6.11 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a5c0000-3fff7a5d0000 rw-p 00040000 fd:02 1181563 /usr/lib64/libidn.so.11.6.11 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a5d0000-3fff7a790000 r-xp 00000000 fd:02 133399 /usr/lib64/power8/libc-2.17.so Size: 1792 kB Rss: 1472 kB Pss: 31 kB Shared_Clean: 1472 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 1472 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a790000-3fff7a7a0000 r--p 001b0000 fd:02 133399 /usr/lib64/power8/libc-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a7a0000-3fff7a7b0000 rw-p 001c0000 fd:02 133399 /usr/lib64/power8/libc-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a7b0000-3fff7a7d0000 r-xp 00000000 fd:02 137997 /usr/lib64/power8/libpthread-2.17.so Size: 128 kB Rss: 128 kB Pss: 3 kB Shared_Clean: 128 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 128 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a7d0000-3fff7a7e0000 r--p 00010000 fd:02 137997 /usr/lib64/power8/libpthread-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a7e0000-3fff7a7f0000 rw-p 00020000 fd:02 137997 /usr/lib64/power8/libpthread-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a7f0000-3fff7a810000 r-xp 00000000 fd:02 1195028 /usr/lib64/libgcc_s-4.8.3-20140911.so.1 Size: 128 kB Rss: 64 kB Pss: 2 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a810000-3fff7a820000 r--p 00010000 fd:02 1195028 /usr/lib64/libgcc_s-4.8.3-20140911.so.1 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a820000-3fff7a830000 rw-p 00020000 fd:02 1195028 /usr/lib64/libgcc_s-4.8.3-20140911.so.1 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a830000-3fff7a900000 r-xp 00000000 fd:02 137950 /usr/lib64/power8/libm-2.17.so Size: 832 kB Rss: 256 kB Pss: 22 kB Shared_Clean: 256 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 256 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7a900000-3fff7a910000 r--p 000c0000 fd:02 137950 /usr/lib64/power8/libm-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7a910000-3fff7a920000 rw-p 000d0000 fd:02 137950 /usr/lib64/power8/libm-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7a920000-3fff7aa70000 r-xp 00000000 fd:02 1181158 /usr/lib64/libstdc++.so.6.0.19 Size: 1344 kB Rss: 960 kB Pss: 153 kB Shared_Clean: 960 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 960 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7aa70000-3fff7aa80000 r--p 00150000 fd:02 1181158 /usr/lib64/libstdc++.so.6.0.19 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd mr mw me ac 3fff7aa80000-3fff7aa90000 rw-p 00160000 fd:02 1181158 /usr/lib64/libstdc++.so.6.0.19 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7aa90000-3fff7aaa0000 rw-p 00000000 00:00 0 Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64 kB Referenced: 64 kB Anonymous: 64 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me ac 3fff7aaa0000-3fff7aab0000 r-xp 00000000 fd:02 138001 /usr/lib64/power8/librt-2.17.so Size: 64 kB Rss: 64 kB Pss: 2 kB Shared_Clean: 64 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 64 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd ex mr mw me 3fff7aab0000-3fff7aac0000 r--p 00000000 fd:02 138001 /usr/lib64/power8/librt-2.17.so Size: 64 kB Rss: 64 kB Pss: 64 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB
1,166 lines not shown · raw message
Part of this message is not shown here. View the raw message.
Quoted 4 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 3 lines · Dr. David Alan Gilbert
Quoted 2 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 1 line · Dr. David Alan Gilbert
* Bharata B Rao ([email protected]) wrote:
<snip>
Quoted 4 lines
Quoted 2 lines
Can you :
1) show me the command line you're using./ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -vga none -machine pseries -m 64G,slots=32,maxmem=128G -smp 16 -device virtio-blk-pci,drive=rootdisk -drive file=/home/bharata/F20-snap1,if=none,cache=none,id=rootdisk,format=qcow2 -monitor telnet:127.0.0.1:1234,server,nowait -trace events=my-trace-events
Thanks; looks sensible.
Quoted 5 lines
Quoted 1 line
2) Show me
/proc/pid/smapsfor the destination qemuAttached.
3fef23ff0000-3fff23ff0000 rw-p 00000000 00:00 0
Quoted 16 lines
Size: 67108864 kB Rss: 67108864 kB Pss: 67108864 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 67108864 kB Referenced: 67108864 kB Anonymous: 67108864 kB AnonHugePages: 32768 kB Swap: 0 kB SwapPss: 0 kB KernelPageSize: 64 kB MMUPageSize: 64 kB Locked: 0 kB VmFlags: rd wr mr mw me dc ac hg mg
OK, all dirty - that's not happy.
Quoted 5 lines
Quoted 3 lines
3) Turn on the trace
postcopy_place_page_zero
the theory is that most of your pages should end up as zero pages and not be allocated.No hits for
postcopy_place_page_zeroeither in source or destination QEMU.
OK, that's why it's all dirty then - it should be placing most pages as zero pages; I checked and that is triggering on x86.
Oh, I think I see it; the following is untested, I'll grab a machine to try it on but it will take a little while:
migration/ram.c +1 -1
@@ -2273,6 +2274,7 @@ static int ram_load_postcopy(QEMUFile *f)/* Temporary page that is later 'placed' */ void *postcopy_host_page =postcopy_get_tmp_page (mis);void *last_host =NULL ; +bool all_zero = false;while (!ret && !(flags &RAM_SAVE_FLAG_EOS )) {ram_addr_t addr;
@@ -2280,7 +2282,6 @@ static int ram_load_postcopy(QEMUFile *f)void *page_buffer =NULL ;void *place_source =NULL ;uint8_t ch; -bool all_zero = false; addr =qemu_get_be64 (f); flags = addr & ~TARGET_PAGE_MASK ;
Thanks for spotting this!
Dave
Quoted 3 lines
Regards,
Bharata.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 18 lines · Dr. David Alan Gilbert
<snip>
Oh, I think I see it; the following is untested, I'll grab a machine to try it on but it will take a little while:
migration/ram.c +1
-0@@ -2273,6 +2274,7 @@ static int ram_load_postcopy(QEMUFile *f)/* Temporary page that is later 'placed' */ void *postcopy_host_page =postcopy_get_tmp_page (mis);void *last_host =NULL ; +bool all_zero = false;while (!ret && !(flags &RAM_SAVE_FLAG_EOS )) {ram_addr_t addr;
migration/ram.c static int ram_load_postcopy(QEMUFile *f) · line 2287
Quoted 7 lines
migration/ram.c +0
-1@@ -2280,7 +2282,6 @@ static int ram_load_postcopy(QEMUFile *f)void *page_buffer =NULL ;void *place_source =NULL ;uint8_t ch; -bool all_zero = false; addr =qemu_get_be64 (f); flags = addr & ~TARGET_PAGE_MASK ;
So with the above fix on top of the previous one you sent, I am able to do back and forth postcopy migration of pseries guest within the same host while switching to postcopy mode immediately after starting the migration.
And btw I can see lots of hits to postcopy_place_page_zero now!
Regards,
Bharata.
Quoted 32 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 18 lines · Dr. David Alan Gilbert
<snip>
Oh, I think I see it; the following is untested, I'll grab a machine to try it on but it will take a little while:
migration/ram.c +1
-0@@ -2273,6 +2274,7 @@ static int ram_load_postcopy(QEMUFile *f)/* Temporary page that is later 'placed' */ void *postcopy_host_page =postcopy_get_tmp_page (mis);void *last_host =NULL ; +bool all_zero = false;while (!ret && !(flags &RAM_SAVE_FLAG_EOS )) {ram_addr_t addr;migration/ram.c static int ram_load_postcopy(QEMUFile *f) · line 2287
Quoted 7 lines
migration/ram.c +0
-1@@ -2280,7 +2282,6 @@ static int ram_load_postcopy(QEMUFile *f)void *page_buffer =NULL ;void *place_source =NULL ;uint8_t ch; -bool all_zero = false; addr =qemu_get_be64 (f); flags = addr & ~TARGET_PAGE_MASK ;So with the above fix on top of the previous one you sent, I am able to do back and forth postcopy migration of pseries guest within the same host while switching to postcopy mode immediately after starting the migration.
Quoted 1 line
And btw I can see lots of hits to
postcopy_place_page_zeronow!
Excellent! Thanks for finding that; that misplaced line has been in there for about a year :-)
Dave
Quoted 4 lines
Regards,
Bharata.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 33 lines · Dr. David Alan Gilbert
Quoted 32 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 29 lines · Dr. David Alan Gilbert
Quoted 19 lines · * Bharata B Rao ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
Quoted 7 lines
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tagTesting status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)Tested minimally on ppc64 with back and forth postcopy migration of unloaded pseries guest within the localhost - works as expected.
Quoted 6 lines
However I am seeing a failure in one case. I am not sure if this is a user error or a real issue in postcopy migration. If I switch to postcopy migration immediately after starting the migration, I see the migration failing with error:
qemu-system-ppc64:
qemu_savevm_send_packaged: Unreasonably large packaged state: 25905005I put an arbitrary limit of 16MB (see
MAX_VM_CMD_PACKAGED_SIZEininclude/sysemu/sysemu.h) on the size of the data accepted into the packaged blob. How big is the htab data likely to be?
Quoted 6 lines
Quoted 2 lines
HTAB size is a variable and depends on maxmem size. It will be 1/128 th of maxmem. So for a 32G guest, HTAB will be 256M in size.
OK, that does get a bit big.
Two possible fixes;
1 - postcopy htab (I don't know htab to know how hard that is)
It's.. awkward. We'd need a way to set up the mappings on the destination so that faults on bits of the hash table not yet up to date get flagged and handed to qemu, rather than causing a fatal fault in the guest. I suspect that will need host kernel changes, although maybe there's a way of setting up the htab on destination so that unmapping things look like MMIO (which already goes to qemu).
Quoted 3 lines
2 - do one pass of iterable/non-postcopiable devices before we start the package;
I'm just writing a patch to try that; I'll send it to you to let you try once I get it to not-break normal migration.
Hm. So, depends a bit on what you mean by "one pass". If we've had one complete pass through the hash table, I'd expect that to be enough to get the package down to a reasonable size. But one pass through the full hash table can be multiple calls to the htab iterator.
Which makes me think it's a bit odd that we're not already getting most of the htab data across during the precopy phase. Don't we already delay entering the postcopy phase until precopy is "complete" in the sense that the remaining non-postcopiable data is below the downtime limit? I would have thought that would also ensure we'd only have a reasonable number of remaining htab updates for the package.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Quoted 7 lines · David Gibson
Which makes me think it's a bit odd that we're not already getting most of the htab data across during the precopy phase. Don't we already delay entering the postcopy phase until precopy is "complete" in the sense that the remaining non-postcopiable data is below the downtime limit? I would have thought that would also ensure we'd only have a reasonable number of remaining htab updates for the package.
You can enter postcopy at any time. In this case, Bharata is entering immediately after the beginning of migration, basically without doing any work in precopy mode.
Paolo
Quoted 63 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tagTesting status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)This work has been partially funded by the EU Orbit project:
see http://www.orbitproject.eu/about/v9
Almost all of the changes are changes from review comments, and most are minor.The following are new patches, mostly split out from earlier patches (The exception being adding
userfaultfd.hheader back in - which was in v5
but we took out, but is needed again due to1842bdfd)04/56 Move
page_size_initearlier
08/56qemu_ram_block_by_name
12/56 Factor outhost_from_stream_offsetcall and check 15/56 Add Linuxuserfaultfd.hheader back 21/56migration_is_setup_or_active
30/56migration_completion: Take current state 34/56 Maintain unsentmapThe previous patches 03,10,13,36/54 went in upstream already.
Fix for assert using hotplug (Thanks Bharata for spotting that) Fix for
migrate_cancela second time
Rework formigration_bitmap_rcuafter Denis's deadlock fix
Reworkram_loadinto a separate postcopy loop
The 'sentmap' is now an 'unsentmap' - this saves a complement step at the end
The unsentmap creation is now split into a separate patch
The unsentmap is now stored in the RCU structure (although it can't
really resize during the migrate)
Fix for block migration
still not a suggested combination.
Minor:
trace_savevm_send_open_return_pathadded
splitmigration_is_activefunction into separate patch and made
migration_is_setup_or_active
move file reads intomigrate_handle_adviseandmigrate_handle_packageduse of MIN in send-packaged-chunk
migration_thread_started->migration_thread_running
updatedqemu_get_buffer_in_placewith Juan's version (and size_t'ified it) split thehost_from_stream_offsetchange into a separate patch
'ram_load: Factor outhost_from_stream_offsetcall and check'
split outram_find_block_by_idinto a separate patch and now
calledqemu_ram_block_by_name
postcopy_discard_send_rangeetc now take start/length rather than start/end
(also added another trace)
splitram_save_host_pageinto ram_save_host_page/ram_save_target_page split host page cleanup function into a core that handles both passes
Quoted 1 line
Dave
I need the following patch to make things compile on 32bits, windows, etc.
I guess you are ok with that (I have integrated it in the appropiate places)
Later, Juan.
migration/migration.c +1 -1
@@ -1239,7 +1239,7 @@ static void migrate_handle_rp_req_pages(MigrationState *ms, const char* rbname,if (start & (our_host_ps-1 ) || (len & (our_host_ps-1 ))) {error_report ("%s: Misaligned page request, start: " RAM_ADDR_FMT -" len: " RAM_ADDR_FMT , __func__, start, len); +" len: %zd" , __func__, start, len);mark_source_rp_bad (ms);return ; }
migration/postcopy-ram.c +7 -0
@@ -641,37 +641,44 @@ int postcopy_ram_incoming_init(MigrationIncomingState *mis, size_t ram_pages)int postcopy_ram_incoming_cleanup (MigrationIncomingState *mis) {assert (0 ); +return -1 ; }int postcopy_ram_discard_range (MigrationIncomingState *mis,uint8_t *start,size_t length) {assert (0 ); +return -1 ; }int postcopy_ram_prepare_discard (MigrationIncomingState *mis) {assert (0 ); +return -1 ; }int postcopy_ram_enable_notify (MigrationIncomingState *mis) {assert (0 ); +return -1 ; }int postcopy_place_page (MigrationIncomingState *mis,void *host,void *from) {assert (0 ); +return -1 ; }int postcopy_place_page_zero (MigrationIncomingState *mis,void *host) {assert (0 ); +return -1 ; }void *postcopy_get_tmp_page (MigrationIncomingState *mis) {assert (0 ); +return NULL ; }#endif
Dr. David Alan Gilbert
Reviewed-byQuoted 64 lines · * Juan Quintela ([email protected]) wrote:
Quoted 63 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tagTesting status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)This work has been partially funded by the EU Orbit project:
see http://www.orbitproject.eu/about/v9
Almost all of the changes are changes from review comments, and most are minor.The following are new patches, mostly split out from earlier patches (The exception being adding
userfaultfd.hheader back in - which was in v5
but we took out, but is needed again due to1842bdfd)04/56 Move
page_size_initearlier
08/56qemu_ram_block_by_name
12/56 Factor outhost_from_stream_offsetcall and check 15/56 Add Linuxuserfaultfd.hheader back 21/56migration_is_setup_or_active
30/56migration_completion: Take current state 34/56 Maintain unsentmapThe previous patches 03,10,13,36/54 went in upstream already.
Fix for assert using hotplug (Thanks Bharata for spotting that) Fix for
migrate_cancela second time
Rework formigration_bitmap_rcuafter Denis's deadlock fix
Reworkram_loadinto a separate postcopy loop
The 'sentmap' is now an 'unsentmap' - this saves a complement step at the end
The unsentmap creation is now split into a separate patch
The unsentmap is now stored in the RCU structure (although it can't
really resize during the migrate)
Fix for block migration
still not a suggested combination.
Minor:
trace_savevm_send_open_return_pathadded
splitmigration_is_activefunction into separate patch and made
migration_is_setup_or_active
move file reads intomigrate_handle_adviseandmigrate_handle_packageduse of MIN in send-packaged-chunk
migration_thread_started->migration_thread_running
updatedqemu_get_buffer_in_placewith Juan's version (and size_t'ified it) split thehost_from_stream_offsetchange into a separate patch
'ram_load: Factor outhost_from_stream_offsetcall and check'
split outram_find_block_by_idinto a separate patch and now
calledqemu_ram_block_by_name
postcopy_discard_send_rangeetc now take start/length rather than start/end
(also added another trace)
splitram_save_host_pageinto ram_save_host_page/ram_save_target_page split host page cleanup function into a core that handles both passes
Quoted 6 lines
Quoted 1 line
Dave
I need the following patch to make things compile on 32bits, windows, etc.
I guess you are ok with that (I have integrated it in the appropiate places)
Thanks
Quoted 7 lines
Later, Juan.
migration/migration.c +0
-0
migration/migration.c static void migrate_handle_rp_req_pages(MigrationState *ms, const char* rbname, · line 1245
Quoted 8 lines
migration/migration.c +1
-1@@ -1239,7 +1239,7 @@ static void migrate_handle_rp_req_pages(MigrationState *ms, const char* rbname,if (start & (our_host_ps-1 ) || (len & (our_host_ps-1 ))) {error_report ("%s: Misaligned page request, start: " RAM_ADDR_FMT -" len: " RAM_ADDR_FMT , __func__, start, len); +" len: %zd" , __func__, start, len);mark_source_rp_bad (ms);return ; }
I don't quite get why that doesn't fail in my 32bit test build; but I checked again and my build gives no warnings for it.
migration/postcopy-ram.c int postcopy_ram_incoming_init(MigrationIncomingState *mis, size_t ram_pages) · line 645
Quoted 10 lines
migration/postcopy-ram.c +1
-0@@ -641,37 +641,44 @@ int postcopy_ram_incoming_init(MigrationIncomingState *mis, size_t ram_pages)int postcopy_ram_incoming_cleanup (MigrationIncomingState *mis) {assert (0 ); +return -1 ; }
Sigh, yes, Windows assert/abort's can return.
Quoted 28 lines
migration/postcopy-ram.c +4
-0int postcopy_ram_discard_range (MigrationIncomingState *mis,uint8_t *start,size_t length) {assert (0 ); +return -1 ; }int postcopy_ram_prepare_discard (MigrationIncomingState *mis) {assert (0 ); +return -1 ; }int postcopy_ram_enable_notify (MigrationIncomingState *mis) {assert (0 ); +return -1 ; }int postcopy_place_page (MigrationIncomingState *mis,void *host,void *from) {assert (0 ); +return -1 ; }int postcopy_place_page_zero (MigrationIncomingState *mis,void *host) {assert (0 );
migration/postcopy-ram.c int postcopy_ram_incoming_init(MigrationIncomingState *mis, size_t ram_pages) · line 684
Quoted 10 lines
migration/postcopy-ram.c +2
-0+ return -1; }void *postcopy_get_tmp_page (MigrationIncomingState *mis) {assert (0 ); +return NULL ; }#endif
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Hello, Dr. Gilbert!
As I understand this is not a complete realization of post-copy stage of migration, but only realization of ram post-copy.. I need to implement post-copy migration of block-dirty-bitmaps. And it should work with/without ram post-copy. Did you plan a possibility of post-copying something except ram? If yes, can you please help me with an interface? Should I implement separate thread like postcopy_ram_listen_thread and other things, or something may be reused?
Quoted 164 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tagTesting status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)This work has been partially funded by the EU Orbit project:
see http://www.orbitproject.eu/about/v9
Almost all of the changes are changes from review comments, and most are minor.The following are new patches, mostly split out from earlier patches (The exception being adding
userfaultfd.hheader back in - which was in v5
but we took out, but is needed again due to1842bdfd)04/56 Move
page_size_initearlier
08/56qemu_ram_block_by_name
12/56 Factor outhost_from_stream_offsetcall and check 15/56 Add Linuxuserfaultfd.hheader back 21/56migration_is_setup_or_active
30/56migration_completion: Take current state 34/56 Maintain unsentmapThe previous patches 03,10,13,36/54 went in upstream already.
Fix for assert using hotplug (Thanks Bharata for spotting that) Fix for
migrate_cancela second time
Rework formigration_bitmap_rcuafter Denis's deadlock fix
Reworkram_loadinto a separate postcopy loop
The 'sentmap' is now an 'unsentmap' - this saves a complement step at the end
The unsentmap creation is now split into a separate patch
The unsentmap is now stored in the RCU structure (although it can't
really resize during the migrate)
Fix for block migration
still not a suggested combination.
Minor:
trace_savevm_send_open_return_pathadded
splitmigration_is_activefunction into separate patch and made
migration_is_setup_or_active
move file reads intomigrate_handle_adviseandmigrate_handle_packageduse of MIN in send-packaged-chunk
migration_thread_started->migration_thread_running
updatedqemu_get_buffer_in_placewith Juan's version (and size_t'ified it) split thehost_from_stream_offsetchange into a separate patch
'ram_load: Factor outhost_from_stream_offsetcall and check'
split outram_find_block_by_idinto a separate patch and now
calledqemu_ram_block_by_name
postcopy_discard_send_rangeetc now take start/length rather than start/end
(also added another trace)
splitram_save_host_pageinto ram_save_host_page/ram_save_target_page split host page cleanup function into a core that handles both passesDave
Dr. David Alan Gilbert (56):
Add postcopy documentation
Provide runtime Target page information
Move configuration section writing
Movepage_size_initearlier
qemu_ram_block_from_host
qemu_ram_block_by_name
Renamemis->filetofrom_src_file
Addqemu_get_buffer_in_placeto avoid copies some of the time Add wrapper for setting blocking status on a QEMUFile AddQEMU_MADV_NOHUGEPAGE
ram_debug_dump_bitmap: Dump a migration bitmap as text
ram_load: Factor outhost_from_stream_offsetcall and check
migrate_init: Call from savevm
Renamesave_live_completetosave_live_complete_precopyAdd Linuxuserfaultfd.hheader
Return path: Open a return path on QEMUFile for sockets Return path:socket_writev_buffer: Block even on non-blocking fd's Migration commands
Return path: Control commands
Return path: Send responses from destination to sourcemigration_is_setup_or_active
Return path: Source handling of return path
Rework loadvm path for subloops
Add migration-capability boolean for postcopy-ram.
Add wrappers and handlers for sending/receiving the postcopy-ram
migration messages.
MIG_CMD_PACKAGED: Send a packaged chunk of migration stream Modifysave_live_pendingfor postcopy
postcopy: OS support test
migrate_start_postcopy: Command to trigger transition to postcopy
migration_completion: Take current stateMIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state Avoid sending vmdescription during postcopy
Addqemu_savevm_state_complete_postcopy
Postcopy: Maintain unsentmap
Postcopy: Calculate discard
postcopy: Incoming initialisation
postcopy:ram_enable_notifyto switch on userfault
Postcopy: Postcopy startup in migration thread
Postcopy: End of iteration
Page request: AddMIG_RP_MSG_REQ_PAGESreverse command Page request: Process incoming page request
Page request: Consume pages off the post-copy queuepostcopy_ram.c:place_pageand helpers
Postcopy: Use helpers to map pages during migration
postcopy: Check order of received target pages
Don't sync dirty bitmaps in postcopy
Don't iterate on precopy-only devices during postcopy Host page!=target page: Cleanup bitmaps
Round up RAMBlock sizes to host page sizes
Postcopy; Handle userfault requests
Start up a postcopy/listener thread ready for incoming page data
postcopy: Wire uploadvm_postcopy_handle_commands
Postcopy: Mark nohugepage before discard
End of migration for postcopy
Disable mlock around incoming postcopy
Inhibit ballooning during postcopyballoon.c | 11 + docs/migration.txt | 191 ++++++++ exec.c | 92 +++- hmp-commands.hx | 15 + hmp.c | 7 + hmp.h | 1 + hw/ppc/spapr.c | 2 +- hw/virtio/virtio-balloon.c | 4 +- include/exec/cpu-common.h | 4 + include/exec/exec-all.h | 1 - include/exec/ram_addr.h | 2 - include/migration/migration.h | 121 ++++- include/migration/postcopy-ram.h | 99 ++++ include/migration/qemu-file.h | 10 + include/migration/vmstate.h | 8 +- include/qemu-common.h | 1 + include/qemu/osdep.h | 9 + include/qemu/typedefs.h | 3 + include/sysemu/balloon.h | 2 + include/sysemu/sysemu.h | 46 +- kvm-all.c | 1 - linux-headers/linux/userfaultfd.h | 167 +++++++ migration/Makefile.objs | 2 +- migration/block.c | 9 +- migration/migration.c | 723 +++++++++++++++++++++++++-- migration/postcopy-ram.c | 760 +++++++++++++++++++++++++++++ migration/qemu-file-unix.c | 111 ++++- migration/qemu-file.c | 64 +++ migration/ram.c | 997 +++++++++++++++++++++++++++++++++++--- migration/savevm.c | 823 +++++++++++++++++++++++++++---- qapi-schema.json | 18 +- qmp-commands.hx | 19 + qtest.c | 1 - trace-events | 84 +++- vl.c | 1 + 35 files changed, 4159 insertions(+), 250 deletions(-)create mode 100644
include/migration/postcopy-ram.hcreate mode 100644linux-headers/linux/userfaultfd.hcreate mode 100644migration/postcopy-ram.c
Signature · 3 lines
Best regards, Vladimir * now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.
Quoted 1 line · * Vladimir Sementsov-Ogievskiy ([email protected]) wrote:
Hello, Dr. Gilbert!
Hi Vladimir,
Quoted 2 lines
As I understand this is not a complete realization of post-copy stage of migration, but only realization of ram post-copy.
Yes, that's correct; RAM was the only one we'd had much interest in; but I can understand wanting to help block as well.
Quoted 6 lines
I need to implement
post-copy migration of block-dirty-bitmaps. And it should work with/without ram post-copy. Did you plan a possibility of post-copying something except ram? If yes, can you please help me with an interface? Should I implement separate thread likepostcopy_ram_listen_threadand other things, or something may be reused?
I think with the structure that's in, it should be possible to add another device doing postcopy as well. Although maybe it's worth checking whether these dirty bitmaps behave similarly enough to a non-guest visible RAM block, so then the existing migrate code would handle it.
1) I think you're probably best reusing the postcopy_ram_listen_thread for all postcopy data on the load side; I think that's probably already OK to do that; just rename it. The migration stream once in postcopy mode is just still a normal migration stream, so it can still have a mix of RAM and disk or whatever.
2) The tricky bit I can think of is on the sending side making sure you interleave the requests for RAM and disk requests sensibly; at the moment when in postcopy we drop the bandwidth limit so it'll keep doing RAM transfers and respond to ram requests as soon as possible; you'll need to persuade it to somehow realise there's a disk request and get the RAM code to drop out of it's loop to give the disk code a chance. There's already the opportunity to improve the behaviour with just RAM to stop it flooding the network buffers with background page requests which delay the responses to the postcopy requests.
My knowledge of the block code is pretty minimal, so I've not had a chance to follow much of your dirty-bitmap series.
Dave
Quoted 158 lines · Dr. David Alan Gilbert (git)
Quoted 156 lines
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tagTesting status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)This work has been partially funded by the EU Orbit project:
see http://www.orbitproject.eu/about/v9
Almost all of the changes are changes from review comments, and most are minor.The following are new patches, mostly split out from earlier patches (The exception being adding
userfaultfd.hheader back in - which was in v5
but we took out, but is needed again due to1842bdfd)04/56 Move
page_size_initearlier
08/56qemu_ram_block_by_name
12/56 Factor outhost_from_stream_offsetcall and check 15/56 Add Linuxuserfaultfd.hheader back 21/56migration_is_setup_or_active
30/56migration_completion: Take current state 34/56 Maintain unsentmapThe previous patches 03,10,13,36/54 went in upstream already.
Fix for assert using hotplug (Thanks Bharata for spotting that) Fix for
migrate_cancela second time
Rework formigration_bitmap_rcuafter Denis's deadlock fix
Reworkram_loadinto a separate postcopy loop
The 'sentmap' is now an 'unsentmap' - this saves a complement step at the end
The unsentmap creation is now split into a separate patch
The unsentmap is now stored in the RCU structure (although it can't
really resize during the migrate)
Fix for block migration
still not a suggested combination.
Minor:
trace_savevm_send_open_return_pathadded
splitmigration_is_activefunction into separate patch and made
migration_is_setup_or_active
move file reads intomigrate_handle_adviseandmigrate_handle_packageduse of MIN in send-packaged-chunk
migration_thread_started->migration_thread_running
updatedqemu_get_buffer_in_placewith Juan's version (and size_t'ified it) split thehost_from_stream_offsetchange into a separate patch
'ram_load: Factor outhost_from_stream_offsetcall and check'
split outram_find_block_by_idinto a separate patch and now
calledqemu_ram_block_by_name
postcopy_discard_send_rangeetc now take start/length rather than start/end
(also added another trace)
splitram_save_host_pageinto ram_save_host_page/ram_save_target_page split host page cleanup function into a core that handles both passesDave
Dr. David Alan Gilbert (56):
Add postcopy documentation
Provide runtime Target page information
Move configuration section writing
Movepage_size_initearlier
qemu_ram_block_from_host
qemu_ram_block_by_name
Renamemis->filetofrom_src_file
Addqemu_get_buffer_in_placeto avoid copies some of the time Add wrapper for setting blocking status on a QEMUFile AddQEMU_MADV_NOHUGEPAGE
ram_debug_dump_bitmap: Dump a migration bitmap as text
ram_load: Factor outhost_from_stream_offsetcall and check
migrate_init: Call from savevm
Renamesave_live_completetosave_live_complete_precopyAdd Linuxuserfaultfd.hheader
Return path: Open a return path on QEMUFile for sockets Return path:socket_writev_buffer: Block even on non-blocking fd's Migration commands
Return path: Control commands
Return path: Send responses from destination to sourcemigration_is_setup_or_active
Return path: Source handling of return path
Rework loadvm path for subloops
Add migration-capability boolean for postcopy-ram.
Add wrappers and handlers for sending/receiving the postcopy-ram
migration messages.
MIG_CMD_PACKAGED: Send a packaged chunk of migration stream Modifysave_live_pendingfor postcopy
postcopy: OS support test
migrate_start_postcopy: Command to trigger transition to postcopy
migration_completion: Take current stateMIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state Avoid sending vmdescription during postcopy
Addqemu_savevm_state_complete_postcopy
Postcopy: Maintain unsentmap
Postcopy: Calculate discard
postcopy: Incoming initialisation
postcopy:ram_enable_notifyto switch on userfault
Postcopy: Postcopy startup in migration thread
Postcopy: End of iteration
Page request: AddMIG_RP_MSG_REQ_PAGESreverse command Page request: Process incoming page request
Page request: Consume pages off the post-copy queuepostcopy_ram.c:place_pageand helpers
Postcopy: Use helpers to map pages during migration
postcopy: Check order of received target pages
Don't sync dirty bitmaps in postcopy
Don't iterate on precopy-only devices during postcopy Host page!=target page: Cleanup bitmaps
Round up RAMBlock sizes to host page sizes
Postcopy; Handle userfault requests
Start up a postcopy/listener thread ready for incoming page data
postcopy: Wire uploadvm_postcopy_handle_commands
Postcopy: Mark nohugepage before discard
End of migration for postcopy
Disable mlock around incoming postcopy
Inhibit ballooning during postcopyballoon.c | 11 + docs/migration.txt | 191 ++++++++ exec.c | 92 +++- hmp-commands.hx | 15 + hmp.c | 7 + hmp.h | 1 + hw/ppc/spapr.c | 2 +- hw/virtio/virtio-balloon.c | 4 +- include/exec/cpu-common.h | 4 + include/exec/exec-all.h | 1 - include/exec/ram_addr.h | 2 - include/migration/migration.h | 121 ++++- include/migration/postcopy-ram.h | 99 ++++ include/migration/qemu-file.h | 10 + include/migration/vmstate.h | 8 +- include/qemu-common.h | 1 + include/qemu/osdep.h | 9 + include/qemu/typedefs.h | 3 + include/sysemu/balloon.h | 2 + include/sysemu/sysemu.h | 46 +- kvm-all.c | 1 - linux-headers/linux/userfaultfd.h | 167 +++++++ migration/Makefile.objs | 2 +- migration/block.c | 9 +- migration/migration.c | 723 +++++++++++++++++++++++++-- migration/postcopy-ram.c | 760 +++++++++++++++++++++++++++++ migration/qemu-file-unix.c | 111 ++++- migration/qemu-file.c | 64 +++ migration/ram.c | 997 +++++++++++++++++++++++++++++++++++--- migration/savevm.c | 823 +++++++++++++++++++++++++++---- qapi-schema.json | 18 +- qmp-commands.hx | 19 +
Quoted 15 lines
Quoted 8 lines
qtest.c | 1 - trace-events | 84 +++- vl.c | 1 + 35 files changed, 4159 insertions(+), 250 deletions(-) create mode 100644 include/migration/postcopy-ram.h create mode 100644 linux-headers/linux/userfaultfd.h create mode 100644 migration/postcopy-ram.cSignature · 3 lines
Best regards, Vladimir * now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Ok, thanks, I'll think about it.
Quoted 214 lines · Dr. David Alan Gilbert
Quoted 1 line · * Vladimir Sementsov-Ogievskiy ([email protected]) wrote:
Hello, Dr. Gilbert!
Hi Vladimir,
Quoted 2 lines
As I understand this is not a complete realization of post-copy stage of migration, but only realization of ram post-copy.
Yes, that's correct; RAM was the only one we'd had much interest in; but I can understand wanting to help block as well.
Quoted 6 lines
I need to implement
post-copy migration of block-dirty-bitmaps. And it should work with/without ram post-copy. Did you plan a possibility of post-copying something except ram? If yes, can you please help me with an interface? Should I implement separate thread likepostcopy_ram_listen_threadand other things, or something may be reused?I think with the structure that's in, it should be possible to add another device doing postcopy as well. Although maybe it's worth checking whether these dirty bitmaps behave similarly enough to a non-guest visible RAM block, so then the existing migrate code would handle it.
1) I think you're probably best reusing the
postcopy_ram_listen_threadfor all postcopy data on the load side; I think that's probably already OK to do that; just rename it. The migration stream once in postcopy mode is just still a normal migration stream, so it can still have a mix of RAM and disk or whatever.2) The tricky bit I can think of is on the sending side making sure you interleave the requests for RAM and disk requests sensibly; at the moment when in postcopy we drop the bandwidth limit so it'll keep doing RAM transfers and respond to ram requests as soon as possible; you'll need to persuade it to somehow realise there's a disk request and get the RAM code to drop out of it's loop to give the disk code a chance. There's already the opportunity to improve the behaviour with just RAM to stop it flooding the network buffers with background page requests which delay the responses to the postcopy requests.
My knowledge of the block code is pretty minimal, so I've not had a chance to follow much of your dirty-bitmap series.
Dave
Quoted 157 lines · Dr. David Alan Gilbert (git)
Quoted 156 lines
From: "Dr. David Alan Gilbert" <[email protected]>
This is the 9th cut of my version of postcopy.
The userfaultfd linux kernel code is now in the upstream kernel tree, and so 4.3 can be used without modification.
This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v9 tagTesting status:
* Tested heavily on x86
* Smoke tested on aarch64 (so it does work on different page sizes)This work has been partially funded by the EU Orbit project:
see http://www.orbitproject.eu/about/v9
Almost all of the changes are changes from review comments, and most are minor.The following are new patches, mostly split out from earlier patches (The exception being adding
userfaultfd.hheader back in - which was in v5
but we took out, but is needed again due to1842bdfd)04/56 Move
page_size_initearlier
08/56qemu_ram_block_by_name
12/56 Factor outhost_from_stream_offsetcall and check 15/56 Add Linuxuserfaultfd.hheader back 21/56migration_is_setup_or_active
30/56migration_completion: Take current state 34/56 Maintain unsentmapThe previous patches 03,10,13,36/54 went in upstream already.
Fix for assert using hotplug (Thanks Bharata for spotting that) Fix for
migrate_cancela second time
Rework formigration_bitmap_rcuafter Denis's deadlock fix
Reworkram_loadinto a separate postcopy loop
The 'sentmap' is now an 'unsentmap' - this saves a complement step at the end
The unsentmap creation is now split into a separate patch
The unsentmap is now stored in the RCU structure (although it can't
really resize during the migrate)
Fix for block migration
still not a suggested combination.
Minor:
trace_savevm_send_open_return_pathadded
splitmigration_is_activefunction into separate patch and made
migration_is_setup_or_active
move file reads intomigrate_handle_adviseandmigrate_handle_packageduse of MIN in send-packaged-chunk
migration_thread_started->migration_thread_running
updatedqemu_get_buffer_in_placewith Juan's version (and size_t'ified it) split thehost_from_stream_offsetchange into a separate patch
'ram_load: Factor outhost_from_stream_offsetcall and check'
split outram_find_block_by_idinto a separate patch and now
calledqemu_ram_block_by_name
postcopy_discard_send_rangeetc now take start/length rather than start/end
(also added another trace)
splitram_save_host_pageinto ram_save_host_page/ram_save_target_page split host page cleanup function into a core that handles both passesDave
Dr. David Alan Gilbert (56):
Add postcopy documentation
Provide runtime Target page information
Move configuration section writing
Movepage_size_initearlier
qemu_ram_block_from_host
qemu_ram_block_by_name
Renamemis->filetofrom_src_file
Addqemu_get_buffer_in_placeto avoid copies some of the time Add wrapper for setting blocking status on a QEMUFile AddQEMU_MADV_NOHUGEPAGE
ram_debug_dump_bitmap: Dump a migration bitmap as text
ram_load: Factor outhost_from_stream_offsetcall and check
migrate_init: Call from savevm
Renamesave_live_completetosave_live_complete_precopyAdd Linuxuserfaultfd.hheader
Return path: Open a return path on QEMUFile for sockets Return path:socket_writev_buffer: Block even on non-blocking fd's Migration commands
Return path: Control commands
Return path: Send responses from destination to sourcemigration_is_setup_or_active
Return path: Source handling of return path
Rework loadvm path for subloops
Add migration-capability boolean for postcopy-ram.
Add wrappers and handlers for sending/receiving the postcopy-ram
migration messages.
MIG_CMD_PACKAGED: Send a packaged chunk of migration stream Modifysave_live_pendingfor postcopy
postcopy: OS support test
migrate_start_postcopy: Command to trigger transition to postcopy
migration_completion: Take current stateMIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state Avoid sending vmdescription during postcopy
Addqemu_savevm_state_complete_postcopy
Postcopy: Maintain unsentmap
Postcopy: Calculate discard
postcopy: Incoming initialisation
postcopy:ram_enable_notifyto switch on userfault
Postcopy: Postcopy startup in migration thread
Postcopy: End of iteration
Page request: AddMIG_RP_MSG_REQ_PAGESreverse command Page request: Process incoming page request
Page request: Consume pages off the post-copy queuepostcopy_ram.c:place_pageand helpers
Postcopy: Use helpers to map pages during migration
postcopy: Check order of received target pages
Don't sync dirty bitmaps in postcopy
Don't iterate on precopy-only devices during postcopy Host page!=target page: Cleanup bitmaps
Round up RAMBlock sizes to host page sizes
Postcopy; Handle userfault requests
Start up a postcopy/listener thread ready for incoming page data
postcopy: Wire uploadvm_postcopy_handle_commands
Postcopy: Mark nohugepage before discard
End of migration for postcopy
Disable mlock around incoming postcopy
Inhibit ballooning during postcopyballoon.c | 11 + docs/migration.txt | 191 ++++++++ exec.c | 92 +++- hmp-commands.hx | 15 + hmp.c | 7 + hmp.h | 1 + hw/ppc/spapr.c | 2 +- hw/virtio/virtio-balloon.c | 4 +- include/exec/cpu-common.h | 4 + include/exec/exec-all.h | 1 - include/exec/ram_addr.h | 2 - include/migration/migration.h | 121 ++++- include/migration/postcopy-ram.h | 99 ++++ include/migration/qemu-file.h | 10 + include/migration/vmstate.h | 8 +- include/qemu-common.h | 1 + include/qemu/osdep.h | 9 + include/qemu/typedefs.h | 3 + include/sysemu/balloon.h | 2 + include/sysemu/sysemu.h | 46 +- kvm-all.c | 1 - linux-headers/linux/userfaultfd.h | 167 +++++++ migration/Makefile.objs | 2 +- migration/block.c | 9 +- migration/migration.c | 723 +++++++++++++++++++++++++-- migration/postcopy-ram.c | 760 +++++++++++++++++++++++++++++ migration/qemu-file-unix.c | 111 ++++- migration/qemu-file.c | 64 +++ migration/ram.c | 997 +++++++++++++++++++++++++++++++++++--- migration/savevm.c | 823 +++++++++++++++++++++++++++---- qapi-schema.json | 18 +- qmp-commands.hx | 19 +Quoted 14 lines
Quoted 8 lines
qtest.c | 1 - trace-events | 84 +++- vl.c | 1 + 35 files changed, 4159 insertions(+), 250 deletions(-)create mode 100644
include/migration/postcopy-ram.hcreate mode 100644linux-headers/linux/userfaultfd.hcreate mode 100644migration/postcopy-ram.cSignature · 3 lines
Best regards, Vladimir * now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Signature · 3 lines
Best regards, Vladimir * now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.