Obtain dirty bitmap via VM logging

This patch introduces the mechanism of dumping the dirty bitmap either in an ascii format or binary format. The implementation is almost similar to the migration one. A separate thread is created for the dumping process.

The bitmap is obtained with the help of ramlist blocks. I have used almost similar functions that have been used in the migration mechanism (in arch_init.c file). The mechanism to save the dirty bitmap is based on RamBlock rather than MemoryRegion. After having a discussion with Juan, there are still some issues with MemoryRegion like
* memory regions can overlap as well as
* no port of TCG to MemoryRegion.

+++ b/include/qapi/qmp/qerror.h

+- "frequency": time difference in milliseconds between each epoch
+- "readable": dumps the bitmap in hex format (non-binary)

savevm.c +14 -0

+void qmp_log_dirty_bitmap(const char *filename, bool has_epochs,
+                          int64_t epochs, bool has_frequency,
+                          int64_t frequency, bool has_readable,
+                          bool readable, Error **errp)
+{
+    FILE *f;
+    BitmapLogState *b = logging_current_state();
+
+    if (b->state == LOG_BITMAP_STATE_ACTIVE ||
+            b->state == LOG_BITMAP_STATE_SETUP ||
+            b->state == LOG_BITMAP_STATE_CANCELING) {
+        error_set(errp, QERR_LOG_DIRTY_BITMAP_ACTIVE);
+        return;
+    }

include/qapi/qmp/qerror.h void qerror_report_err(Error *err); · line 123

Quoted 3 lines

include/qapi/qmp/qerror.h +3 -0

+#define QERR_LOG_DIRTY_BITMAP_ACTIVE \
+    ERROR_CLASS_GENERIC_ERROR, "Dirty bitmap dump already in progress"
+

Please don't add new ERROR_CLASS macros. Instead, just use error_setg() and directly output the error message at the call site that produces the error.

qapi-schema.json line 4706

Quoted 9 lines

+++ b/qapi-schema.json

qapi-schema.json +4 -0

@@ -4700,3 +4700,13 @@
               'btn'     : 'InputBtnEvent',
               'rel'     : 'InputMoveEvent',
               'abs'     : 'InputMoveEvent' } }
+##
+# @log-dirty-bitmap
+#
+# provides the dirty bitmap for time t if specified.

Too light on the specification. You should document all 4 parameters, and the default value for when the parameters are omitted.

'time t' isn't mentioned in any of the parameter names, so I'm assuming the docs should mention its significance.

Missing a 'Since: 2.1' designation.

qapi-schema.json line 4712

Quoted 6 lines

qapi-schema.json +6 -0

+##
+{ 'command' : 'log-dirty-bitmap',
+  'data'    : { 'filename'      : 'str',
+                '*epochs'       : 'int',
+                '*frequency'    : 'int',
+                '*readable'     : 'bool' } }

Seems okay. As long as you use qemu_open(), it should also allow management to pass in an fd with the add-fd mechanism.

qmp-commands.hx Example: · line 3588

Quoted 5 lines

qmp-commands.hx +5 -0

+
+Arguments:
+
+- "filename": name of the file, in which the bitmap will be saved
+- "epochs": number of times, the memory will be logged

s/times,/times/

Quoted 2 lines

+- "frequency": time difference in milliseconds between each epoch
+- "readable": dumps the bitmap in hex format (non-binary)

Possibly a poor choice of naming (binary output IS machine-readable; in fact, programs prefer parsing binary output over decoding human-readable text); maybe 'pretty' or 'text' would have been better; or even invert the sense and have it be 'binary' and default to true. I also wonder if we even need it - I'd rather have the QMP command always output raw binary, and then rely on post-processing to turn it into whatever format the user wants to see (od works wonders), than to bloat qemu with having to generate a human-readable variant.

savevm.c void do_savevm(Monitor *mon, const QDict *qdict) · line 1087

Quoted 7 lines

savevm.c +6 -0

+static inline void check_epochs_value(int64_t *epochs, Error **errp)
+{
+    if (*epochs <= 0) {
+        error_setg(errp, "epoch size must be greater than 0, setting"
+                        " a value of 3\n");
+        *epochs = 3;

This error reporting won't work. Since this function returns void, if errp is set at all, the caller must assume failure, rather than hoping that you replaced the user's bad input with a saner default value. If it was worth reporting the error, then it is not worth trying to fix up the value.

savevm.c void do_savevm(Monitor *mon, const QDict *qdict) · line 1126

Quoted 7 lines

savevm.c +7 -0

+/*
+ * copied from arch_init.c file (below function)
+ */
+
+static void dirty_bitmap_logging_sync_range(ram_addr_t start,
+                                            ram_addr_t length)
+{

Rather than copying a function, can you make the original non-static and share it between both callers? Otherwise, the two will get out of sync if a bug is fixed in one.

Quoted 1 line

+ if (readable) {

readable is undefined if has_readable is false. You are missing something like:

if (!has_readable) {
    readable = false;
}

savevm.c void do_savevm(Monitor *mon, const QDict *qdict) · line 1307

Quoted 5 lines

savevm.c +5 -0

+    if (!has_epochs) {
+        epochs = 3;
+    }
+    if (!has_frequency) {
+        frequency = 40;

Magic numbers; use a named constant.

--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org

This patch introduces both hmp and qmp (log-dirty-bitmap-cancel) interface to cancel an already executing dump process.

Signed-off-by Sanidhya Kashyap <[email protected]>

---

+++ b/qapi-schema.json

qapi-schema.json +1 -0

+{ 'command': 'log-dirty-bitmap-cancel' }

qmp-commands.hx +0 -0

index 33e7719..55b2251 100644
@@ -3603,3 +3603,23 @@ value is 3 while that of frequency is 40 and readable defaults to false.
 

This patch introduces both qmp and hmp (log-dirty-bitmap-set-frequency) interface to update the value of the frequency.

Signed-off-by Sanidhya Kashyap <[email protected]>

---

+++ b/qapi-schema.json

This script extracts the bitmap only from a binary file. It takes only a single input as file. This will be useful to users for exploring the writable working set for the VM.

Signed-off-by Sanidhya Kashyap <[email protected]>

---
 scripts/extract-bitmap.py | 68 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
 create mode 100755 scripts/extract-bitmap.py

scripts/extract-bitmap.py new file +0 -0

new file mode 100755
index 0000000..64ce136

Quoted 9 lines · Sanidhya Kashyap

This script extracts the bitmap only from a binary file. It takes only a single input as file. This will be useful to users for exploring the writable working set for the VM.

Signed-off-by Sanidhya Kashyap <[email protected]>

---
 scripts/extract-bitmap.py | 68 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
 create mode 100755 scripts/extract-bitmap.py

Quoted 5 lines

Quoted 5 lines

Signed-off-by Sanidhya Kashyap <[email protected]>

---
 scripts/extract-bitmap.py | 68 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
 create mode 100755 scripts/extract-bitmap.py

Does this file need to be mentioned in any Makefile to ensure that it gets distributed as part of the qemu tarball?

--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Hi,

This is the first part of my gsoc work.

Following patches add the support of dirty bitmap logging and dumping to a specified file. These patches provide the mechanism to get the dirty bitmap if the user wants to log the VM besides migration. This can be useful in case of learning about the writable working set of the VM even when it is not being migrated. I know that it will incur huge degradation but still, a user might want to do that. Besides this, this mechanism can also be useful in case of learning about the pages that can be stored in the cache while performing the xbzrle based live migration or even before the beginning of the hybrid (precopy + postcopy) live migration.

I have tried to provide both of the hmp and qmp commands interface to obtain the dirty bitmap. Besides dumping the dirty bitmap, one can also update the frequency with which the logging is taking place or even can cancel the ongoing logging process. Thus, there are 3 hmp and qmp commands interface provided to perform the task.

Sanidhya Kashyap (6):
split dirty bitmap into four for dumping the bitmaps bitmap dump code via QAPI framework
hmp interface for dirty bitmap dump
cancel mechanism for an already running dump bitmap process set the frequency of the dump bitmap process python script for extracting bitmap from a binary file

 exec.c                    |   4 +
 hmp-commands.hx           |  46 +++++++
 hmp.c                     |  36 +++++
 hmp.h                     |   3 +
 include/exec/memory.h     |   3 +-
 include/exec/ram_addr.h   |  16 ++-
 include/qapi/qmp/qerror.h |   3 +
 qapi-schema.json          |  23 ++++
 qmp-commands.hx           |  77 +++++++++++
 savevm.c                  | 344 ++++++++++++++++++++++++++++++++++++++++++++++
 scripts/extract-bitmap.py |  68 +++++++++
 11 files changed, 621 insertions(+), 2 deletions(-)
 create mode 100755 scripts/extract-bitmap.py

Hi,
Nice job. We should avoid running migration_thread and bitmap_logging_thread simultaneously.

Quoted 3 lines · ChenLiang

Hi,
Nice job. We should avoid running migration_thread and bitmap_logging_thread simultaneously.

Any particular suggestion to avoid running simultaneous execution of the threads?

--
Sanidhya

We can do it like this:
https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg02185.html

Quoted 4 lines · Sanidhya Kashyap

Quoted 3 lines · ChenLiang

Hi,
Nice job. We should avoid running migration_thread and bitmap_logging_thread simultaneously.

Quoted 3 lines

Any particular suggestion to avoid running simultaneous execution of the threads?

We can do it like this:
https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg02185.html

Quoted 6 lines · ChenLiang

Quoted 4 lines · Sanidhya Kashyap

Quoted 3 lines · ChenLiang

Hi,
Nice job. We should avoid running migration_thread and bitmap_logging_thread simultaneously.

Quoted 7 lines

Quoted 3 lines

Any particular suggestion to avoid running simultaneous execution of the threads?

We can do it like this:
https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg02185.html

Hi Chen,

I have some questions which I wanted to get clarified before implementing the above part of avoiding the simultaneous execution.

As the migration process is going on, the RUN_STATE_RUNNING state is being used, which is the same case in bitmap dump process. In order to use the concept of RUN_STATE_* states, should I create two new states as RUN_STATE_LOGBITMAP AND RUN_STATE_OUTMIGRATE? The RUN_STATE_OUTMIGRATE state will have all the transitions that RUN_STATE_RUNNING supports except the ones being used by migration. Similarly, RUN_STATE_MIGRATE will support all the transitions of RUN_STATE_RUNNING except RUN_STATE_LOGBITMAP.

We might also have to update the runstate_is_running function, since RUN_STATE_LOGBITMAP and RUN_STATE_MIGRATE are almost similar in nature to the RUN_STATE_RUNNING. What is your opinion about that?

Is it the only approach to do it or are there any simple and efficient approaches?

Thanks,
Sanidhya

Hmm, I think we need the two flags. Although it is little hard to do it. Because the vm can't do something like hotplug vcpu or device during migrate. But for now qemu don't guarantee it(in my reading). So we should change the vm state RUN_STATE_RUNNING to RUN_STATE_MIGRATE when migrate. BTW, it is not your patches problem.

cc Juan

Best regards
ChenLiang