Skip to content

hw/nvme: add basic live migration support

Alexander Mikhalitsyn qemu-devel 10 versions – Applied

Status from this message: Applied to nvme.next!

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v10 Latest8 patches 8 repliesApplied
  2. v98 patches 5 repliesSuperseded
  3. v88 patches 0 repliesSuperseded
Earlier versions · 7
VersionPostedPatchesRepliesStatus
v78 patches4 repliesSuperseded
v68 patches8 repliesSuperseded
v58 patches19 repliesSuperseded
v47 of 8 patches0 repliesSuperseded
v36 patches4 repliesSuperseded
v24 patches0 repliesSuperseded
v1 RFC4 patches7 repliesSuperseded

Patches in v10 Applied

#Subject and filesSizeRepliesApplied
1/8tests/functional/migration: add VM launch/configure hookstests/functional/migration.py+20 -30 replies
2/8hw/nvme: add migration blockers for non-supported caseshw/nvme/ctrl.c hw/nvme/nvme.h include/block/nvme.h+227 -00 replies
3/8hw/nvme: split nvme_init_sq/nvme_init_cq into helpershw/nvme/ctrl.c+37 -220 replies
4/8hw/nvme: set CQE.sq_id earlier in nvme_process_sqhw/nvme/ctrl.c+1 -10 replies
5/8hw/nvme: unmap req->sg earlier in nvme_enqueue_req_completionhw/nvme/ctrl.c+2 -10 replies
6/8hw/nvme: add basic live migration supporthw/nvme/ctrl.c hw/nvme/ns.c hw/nvme/nvme.h +1 more+934 -93 replies
7/8tests/functional/x86_64: add migration test for NVMe deviceMAINTAINERS tests/functional/x86_64/meson.build tests/functional/x86_64/test_nvme_migration.py+174 -02 replies
8/8tests/qtest/nvme-test: add migration test with full CQtests/qtest/nvme-test.c+419 -02 replies

Reviews trailers per patch

All 8 patches have a Reviewed-by or Acked-by.

PatchStefan HajnocziPeter XuKlaus JensenFabiano Rosas
1/8A-b R-b
2/8A-b R-b
3/8A-b R-b
4/8A-b R-b
5/8A-b R-b
6/8A-b R-b R-b
7/8A-b A-b A-b
8/8A-b R-b A-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v10 thread

Cover letter [PATCH v10 0/8] hw/nvme: add basic live migration support

From: Alexander Mikhalitsyn <[email protected]>

Dear friends,

This patchset adds basic live migration support for QEMU emulated NVMe device.

Implementation has some limitations:
- only one NVMe namespace is supported
- SMART counters are not preserved
- CMB is not supported
- PMR is not supported
- SPDM is not supported
- SR-IOV is not supported

I believe this is something I can support in next patchset versions or separately on-demand (when usecase appears).

Testing.

This patch series was manually tested on:
- Debian 13.3 VM (kernel 6.12.69+deb13-amd64) using fio on *non-root* NVMe disk (root disk was virtio-scsi):

time fio --name=nvme-verify \
    --filename=/dev/nvme0n1 \
    --size=5G \
    --rw=randwrite \
    --bs=4k \
    --iodepth=16 \
    --numjobs=1 \
    --direct=0 \
    --ioengine=io_uring \
    --verify=crc32c \
    --verify_fatal=1

- Windows Server 2022 VM (NVMe drive was a *root* disk) with opened browser playing video.

No defects were found.

Git tree:
https://gitlab.com/mihalicyn/qemu/-/commits/nvme-live-migration

Changelog for version 10:
- build fix on Alpine Linux (PAGE_SIZE macro redefinition)
- clang build fix [ thanks to Fabiano ]
- format specifier change from %lx to 0x% PRIx64 in qtest (fix for win64 cross build)
- I run the entire testsuite, (almost) all green now
https://gitlab.com/mihalicyn/qemu/-/pipelines/2594705360
all the failures I can see look unrelated to this patchset
- I kept all RWB/ACK-tags in place (hope it is fine as all changes were trivial)

Changelog for version 9:
- many trivial check-patch fixes here and there
- disabled qtest on ppc64 (spapr bus) by adding qpci_check_buggy_msi() check
[ we had a private discussion with Klaus and he told me that ppc64
tests are failing after debugging I found this old patch which wasn't merged for some reason https://lore.kernel.org/qemu-devel/[email protected]/ ]
- I kept all RWB/ACK-tags in place (hope it is fine, cause really I hasn't changed too much)
- QEMU CI tests results https://gitlab.com/mihalicyn/qemu/-/pipelines/2593676152

Changelog for version 8:
- rebased
- added Acked-by from Stefan
- added RWB tag from Klaus
- added RWB tag from Peter

Changelog for version 7:
- rebased on top of recent main
- addressed review comments from Stefan Hajnoczi:
- better incoming migration stream validation (SQ/CQids correctness)
- endianness bugs are fixed in qtest (validated on s390x)
- added RWB tags from Klaus

Changelog for version 6:
- rebased on top of:
https://gitlab.com/peterx/qemu/-/tree/vmstate-array-null
(see also https://lore.kernel.org/all/[email protected])
- addressed review comments from Stefan Hajnoczi:
- supported "full CQ" case by serializing NvmeRequest state
- added qtest for NVMe device migration with full CQ

Changelog for version 5:
- rebased on top of https://lore.kernel.org/all/[email protected]/
(as Peter has requested)

Changelog for version 4:
- vmstate dynamic array support reworked as suggested by Peter Xu VMS_ARRAY_OF_POINTER_ALLOW_NULL flag was introduced
qtests were added
- NVMe migration blockers were reworked as Klaus has requested earlier Now, instead of having "deny list" approach, we have more strict pattern of NVMe features filtering and it should be harded to break migration when adding new NVMe features.

Changelog for version 3:
- rebased
- simple functional test was added (in accordance with Klaus Jensen's review comment)

  $ meson test 'func-x86_64-nvme_migration' --setup thorough -C build

Changelog for version 2:
- full support for AERs (in-flight requests and queued events too)

Kind regards,
Alex

Alexander Mikhalitsyn (8):
tests/functional/migration: add VM launch/configure hooks hw/nvme: add migration blockers for non-supported cases hw/nvme: split nvme_init_sq/nvme_init_cq into helpers hw/nvme: set CQE.sq_id earlier in nvme_process_sq hw/nvme: unmap req->sg earlier in nvme_enqueue_req_completion hw/nvme: add basic live migration support tests/functional/x86_64: add migration test for NVMe device tests/qtest/nvme-test: add migration test with full CQ

 MAINTAINERS                                   |    1 +
 hw/nvme/ctrl.c                                | 1036 ++++++++++++++++-
 hw/nvme/ns.c                                  |  164 +++
 hw/nvme/nvme.h                                |   12 +
 hw/nvme/trace-events                          |   10 +
 include/block/nvme.h                          |   12 +
 tests/functional/migration.py                 |   23 +-
 tests/functional/x86_64/meson.build           |    1 +
 .../functional/x86_64/test_nvme_migration.py  |  172 +++
 tests/qtest/nvme-test.c                       |  419 +++++++
 10 files changed, 1814 insertions(+), 36 deletions(-)
 create mode 100755 tests/functional/x86_64/test_nvme_migration.py
Signature · 1 line
2.47.3