Skip to content

RFC VM introspection

Adalbert Lazăr qemu-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest26 patches 17 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/26chardev: tcp: allow to change the reconnect timerchardev/char-fe.c chardev/char-socket.c include/chardev/char-fe.h +1 more+33 -00 replies
2/26char-socket: allow vsock parameters (cid, port)chardev/char-socket.c chardev/char.c+27 -32 replies
3/26char-socket: fix the client mode when created through QMPchardev/char-socket.c+1 -15 replies
4/26char-socket: add 'reconnecting' propertychardev/char-socket.c+11 -02 replies
5/26char-socket: add 'fd' propertychardev/char-socket.c+18 -02 replies
6/26E820: extend the table access interfacehw/i386/e820_memory_layout.c hw/i386/e820_memory_layout.h+14 -00 replies
7/26linux-headers: update with VM introspection interfacelinux-headers/linux/kvm.h+20 -00 replies
8/26kvm: add VM introspection usage documentationqemu-options.hx+76 -00 replies
9/26kvm: introduce the VM introspection objectaccel/kvm/Makefile.objs accel/kvm/vmi.c+169 -00 replies
10/26kvm: vmi: add the handshake with the introspection toolaccel/kvm/vmi.c include/sysemu/vmi-handshake.h+335 -00 replies
11/26kvm: vmi: add 'handshake_timeout' propertyaccel/kvm/vmi.c+65 -10 replies
12/26kvm: vmi: add 'key' propertyaccel/kvm/vmi.c+66 -00 replies
13/26kvm: vmi: block the object destruction if the chardev is connectedaccel/kvm/vmi.c+8 -00 replies
14/26kvm: vmi: allow only one instance of the introspection objectaccel/kvm/vmi.c+26 -00 replies
15/26kvm: vmi: reconnect the socket on resetaccel/kvm/vmi.c+18 -00 replies
16/26kvm: vmi: intercept pause/resumeaccel/kvm/vmi.c accel/stubs/Makefile.objs accel/stubs/vmi-stubs.c +2 more+186 -00 replies
17/26kvm: vmi: add 'unhook_timeout' propertyaccel/kvm/vmi.c+33 -00 replies
18/26kvm: vmi: store/restore 'vm_start_time' on migrate/snapshotaccel/kvm/vmi.c+13 -00 replies
19/26kvm: vmi: intercept force-resetaccel/kvm/vmi.c include/sysemu/vmi-intercept.h monitor/qmp-cmds.c+12 -00 replies
20/26kvm: vmi: intercept live migrationaccel/kvm/vmi.c include/sysemu/vmi-intercept.h migration/migration.c +1 more+45 -76 replies
21/26kvm: vmi: postpone the OK response from qmp_stop()accel/kvm/vmi.c accel/stubs/vmi-stubs.c include/monitor/monitor.h +4 more+59 -20 replies
22/26kvm: vmi: add 'async_unhook' propertyaccel/kvm/vmi.c+38 -00 replies
23/26kvm: vmi: intercept shutdownaccel/kvm/vmi.c include/sysemu/vmi-intercept.h monitor/qmp-cmds.c+32 -40 replies
24/26kvm: vmi: add 'unhook_on_shutdown' propertyaccel/kvm/vmi.c+25 -00 replies
25/26kvm: vmi: extend handshake to include the e820 tableaccel/kvm/vmi.c include/sysemu/vmi-handshake.h+82 -90 replies
26/26kvm: vmi: add 'command' and 'event' propertiesaccel/kvm/vmi.c+74 -120 replies

Reviews trailers per patch

1 of 26 patches has a Reviewed-by or Acked-by.

PatchMarc-André Lureau
1/26
2/26
3/26
4/26
5/26R-b
6/26
7/26
8/26
9/26
10/26
11/26
12/26
13/26
14/26
15/26
16/26
17/26
18/26
19/26
20/26
21/26
22/26
23/26
24/26
25/26
26/26

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [RFC PATCH v1 00/26] VM introspection

The KVM introspection subsystem provides a facility for applications running on the host or in a separate VM, to control the execution of other VMs (pause, resume, shutdown), query the state of the vCPUs (GPRs, MSRs etc.), alter the page access bits in the shadow page tables (only for the hardware backed ones, eg. Intel's EPT) and receive notifications when events of interest have taken place (shadow page table level faults, key MSR writes, hypercalls etc.).

This is the userspace part of the KVM introspection API already posted on the KVM list[1]. Thanks to Samuel Laurén and Mathieu Tarral, this new VMI API has been integrated into the KVM-VMI[2] project. The pull request into the libVMI[3] library is under review.

As suggested by Stefan Hajnoczi and Paolo Bonzini, the connection with the introspection tool is initiated by the QEMU process of the introspected VM using a socket. After the handshake, QEMU will hand over the file descriptor to KVM. From this point, the introspection tool will use the socket to send introspection commands (read/write guest memory, set page access, etc.) directly to KVM and to receive introspection events (breakpoint, page fault, etc.). However, for some user actions such as pause, suspend, live migration, etc., we rely on QEMU to notify KVM, that will notify the introspection tool, to remove the changes made to the guest, so that the guest can run when the introspection channel is disconnected.

The patches were tested with QEMU 2.12 (through libvirt 1.3.1) and summarly tested with 5.0.0-rc2, except for the last two patches (25 and 26) which were not tested at all, but still included for the initial feedback.

Patches 01-06 add some extensions to the current code, which may or may not be needed for the next patches, but we're looking forward for your comments about these. Except for patch 6, all are chardev/socket related.

Patch 07 adds the KVM ioctls for VM introspection:
- KVM_INTROSPECTION_HOOK used to hand over the file descriptor
- KVM_INTROSPETION_PREUNHOOK used on pause/suspend/live migration
- KVM_INTROSPECTION_UNHOOK used to clean-up the introspection structures from KVM
- KVM_INTROSPECTION_COMMAND and KVM_INTROSPECTION_EVENT used to mark the the introspection commands/events that are allowed.

Patch 08 and 09 introduce the newly added introspection object. Patch 08 contains the usage documenation of this object with all the parameters that will be added by the next patches. We've tested the creation of this object through QMP/libvirt and we rely on this to start the VM introspection for any running VM.

Patches 10-12 add the handshake, the authentication of the introspection tool and the hand over to KVM.

Patches 13-15 add some safe guards (block the destruction of the introspection object if the introspection socket is connected and allow only one instance of the introspection object) and force the socket reconnection on guest reset. Blocking the destruction of the introspection object might not be enough, because we also want to block the destruction of the introspection socket. Or it might be too much, because this can be done through QMP, and whoever has access to it may crash the guest in multiple ways.

Patches 16-17 add the first intercepted commands (pause/resume) and introduce one of the method we use to delay intercepted commands until the introspection tool has a chance to react.

Patch 18 adds the information we save with the VM snapshot, the VM start time.

Patches 19-20 add the interception of force-reset and live migration commands.

Patch 21 adds an workaround to block the snapshots with memory done by libvirt until the introspection tool has a chance to react. It hasn't been tested with 5.0.0-rc2. For 2.12 the patch is slightly bigger.

Patch 22 adds a second method of delaying the intercepted commands, by running the main loop.

Patches 23-24 add the interception of the shutdown command, which doesn't seems to be done right because the shutdown signal might not be delivered to the guest, not to mention that is desirable to catch all sources that my trigger the shudown.

Patch 25, which is not tested, extends the handshake structures to send the e820 table (for the x86* architectures).

Patch 26, adds the properties to control what introspection commands and what introspection events are allowed for this guest.

[1]: https://lore.kernel.org/kvm/[email protected]/
[2]: https://github.com/KVM-VMI/kvm-vmi
[3]: https://github.com/libvmi/libvmi

Adalbert Lazăr (20):
chardev: tcp: allow to change the reconnect timer
char-socket: allow vsock parameters (cid, port)
char-socket: fix the client mode when created through QMP
char-socket: add 'reconnecting' property
char-socket: add 'fd' property
E820: extend the table access interface
linux-headers: update with VM introspection interface
kvm: introduce the VM introspection object
kvm: vmi: add the handshake with the introspection tool
kvm: vmi: add 'handshake_timeout' property
kvm: vmi: add 'key' property
kvm: vmi: block the object destruction if the chardev is connected
kvm: vmi: allow only one instance of the introspection object
kvm: vmi: add 'unhook_timeout' property
kvm: vmi: store/restore 'vm_start_time' on migrate/snapshot
kvm: vmi: postpone the OK response from qmp_stop()
kvm: vmi: add 'async_unhook' property
kvm: vmi: add 'unhook_on_shutdown' property
kvm: vmi: extend handshake to include the e820 table
kvm: vmi: add 'command' and 'event' properties

Marian Rotariu (6):
kvm: add VM introspection usage documentation
kvm: vmi: reconnect the socket on reset
kvm: vmi: intercept pause/resume
kvm: vmi: intercept force-reset
kvm: vmi: intercept live migration
kvm: vmi: intercept shutdown

 accel/kvm/Makefile.objs        |    1 +
 accel/kvm/vmi.c                | 1091 ++++++++++++++++++++++++++++++++
 accel/stubs/Makefile.objs      |    1 +
 accel/stubs/vmi-stubs.c        |   14 +
 chardev/char-fe.c              |   11 +
 chardev/char-socket.c          |   72 ++-
 chardev/char.c                 |    3 +
 hw/i386/e820_memory_layout.c   |   12 +
 hw/i386/e820_memory_layout.h   |    2 +
 include/chardev/char-fe.h      |    7 +
 include/chardev/char.h         |    1 +
 include/monitor/monitor.h      |    1 +
 include/sysemu/vmi-handshake.h |   66 ++
 include/sysemu/vmi-intercept.h |   25 +
 linux-headers/linux/kvm.h      |   20 +
 migration/migration.c          |   18 +-
 migration/migration.h          |    2 +
 monitor/Makefile.objs          |    2 +-
 monitor/qmp-cmds.c             |   18 +
 monitor/qmp.c                  |   11 +
 monitor/stubs.c                |    9 +
 qemu-options.hx                |   76 +++
 22 files changed, 1455 insertions(+), 8 deletions(-)
 create mode 100644 accel/kvm/vmi.c
 create mode 100644 accel/stubs/vmi-stubs.c
 create mode 100644 include/sysemu/vmi-handshake.h
 create mode 100644 include/sysemu/vmi-intercept.h
 create mode 100644 monitor/stubs.c

base-commit: 14e5526b51910efd62cd31cd95b49baca975c83f
CC: Stefan Hajnoczi <[email protected]>
CC: Paolo Bonzini <[email protected]>
CC: Tamas K Lengyel <[email protected]>
CC: Mathieu Tarral <[email protected]>
CC: Samuel Laurén <[email protected]>
CC: Konrad Rzeszutek Wilk <[email protected]>
CC: Patrick Colp <[email protected]>
CC: Jan Kiszka <[email protected]>
CC: "Marc-André Lureau" <[email protected]>
CC: Richard Henderson <[email protected]>
CC: Eduardo Habkost <[email protected]>
CC: "Michael S. Tsirkin" <[email protected]>
CC: Marcel Apfelbaum <[email protected]>
CC: Markus Armbruster <[email protected]>
CC: Juan Quintela <[email protected]>
CC: "Dr. David Alan Gilbert" <[email protected]>
CC: Marian Rotariu <[email protected]>
CC: Mihai Donțu <[email protected]>