Skip to contentSkip to conversation

libxl: Enable stubdom cdrom changing

PATCH Superseded Jason Andryuk 2 participants 4 messages May 16, 2024 – Jul 28, 2024

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 libxl: Enable stubdom cdrom changing

From: Jason Andryuk <[email protected]>

From: Jason Andryuk <[email protected]>

To change the cd-rom medium, libxl will:
- QMP eject the medium from QEMU
- block-detach the old PV disk
- block-attach the new PV disk
- QMP change the medium to the new PV disk by fdset-id

The QMP code is reused, and remove and attach are implemented here.

The stubdom must internally handle adding /dev/xvdc to the appropriate fdset. libxl in dom0 doesn't see the result of adding to the fdset as that is internal to the stubdom, but the fdset's opaque fields will be set to stub-devid:$devid, so libxl can identify it. $devid is common between the stubdom and libxl, so it can be identified on both side. The stubdom will name the device xvdY regardless of the guest name hdY, sdY, or xvdY, but the stubdom will be assigned the same devid facilitating lookup. Because the stubdom add-fd call is asynchronous, libxl needs to poll query-fdsets to identify when add-fd has completed.

For cd-eject, we still need to attach the empty vbd. This is necessary since xenstore is used to determine that hdc exists. Otherwise after eject, hdc would be gone and the cd-insert would fail to find the drive to insert new media.

---
QEMU inside the stubdom can be sandboxed disallowing opening the devices by path name, so QMP add-fd is used like with dom0 QEMU depriv.

v1 allowed ERROR_JSON_CONFIG_EMPTY "because a stubdom don't have a json config," but it is re-introduced in v2 since the guest should be acted on. It seems fine in testing. The missing JSON was probably from OpenXT having disabled JSON configs in the past.

v2
Use query-fdsets for removal and addition - stub-devid:$devid Re-use aodev instead of 2nd aodev_del
Rename some functions
Support sdX vdevs
Get stubdomid from cis->disk_domid in stubdom callbacks Use if (rc) not if (rc != 0)
Remove comment about libxl_device_disk_remove
Use EGC_GC not STATE_AO_GC
Re-work and eliminate cdrom_insert_stubdom_query_fdset_retry Change some messages
Allow missing removal fdset in case it wasn't added during startup. Drop LOGD(... rc=%d)

v3:
Remove LOG DEBUGs
s/timeout_retry/retry_timer/
Init stubdom_fdset = -1 and use as flag to pase fdset-id Drop AO_GC
Avoid non-error goto
Expect ERROR_TIMEOUT as non-error
Drop some comments
close payload_fd and set to -1
Remove limit on query-fdset retries

By removing the limit on query-fdset retries, we lose an explicity error message for the stubdom not performing the add-fd, but we give more time which may let more cases succeed.

---
 docs/misc/stubdom.txt         |  10 +
 tools/libs/light/libxl_disk.c | 384 ++++++++++++++++++++++++++++++++--
 2 files changed, 373 insertions(+), 21 deletions(-)
docs/misc/stubdom.txt +10 -0
index c717a95d17..64c220db20 100644
@@ -127,6 +127,16 @@ Limitations:
  - at most 26 emulated disks are supported (more are still available as PV disks)
  - graphics output (VNC/SDL/Spice) not supported
 
+CD-ROM changing:
+
+To change the CD-ROM medium, libxl will:
+ - QMP eject the medium from QEMU
+ - block-detach the old PV disk
+ - block-attach the new PV disk
+ - QMP change the medium to the new PV disk by fdset-id
+
+The stubdom must internally add /dev/xvdc to an fdset in QEMU with opaque set
+to "stub-devid:$devid".  libxl will lookup the fdset with that string.
 
                                    PV-GRUB
                                    =======
tools/libs/light/libxl_disk.c +363 -21

500 lines not on this page · raw message

Signature · 1 line
2.40.1
#2 lore raw
In reply to Jason Andryuk
Quoted 29 lines · Jason Andryuk
tools/libs/light/libxl_disk.c +29 -0
+static void cdrom_insert_stubdom_parse_fdset_rm(libxl__egc *egc,
+                                                libxl__ev_qmp *qmp,
+                                                const libxl__json_object *resp,
+                                                int rc)
+{
+    EGC_GC;
+    libxl__cdrom_insert_state *cis = CONTAINER_OF(qmp, *cis, qmp);
+    int devid;
+    int fdset;
+
+    if (rc) goto out;
+
+    /* Only called for qemu-xen/linux stubdom. */
+    assert(cis->dm_ver == LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN);
+
+    devid = libxl__device_disk_dev_number(cis->disk->vdev, NULL, NULL);
+    fdset = query_fdsets_find_fdset(gc, resp, devid);
+    if (fdset < 0) {
+        rc = fdset;
+        goto out;
+    }
+
+    LOGD(DEBUG, cis->domid, "Found fdset %d", fdset);
+
+    libxl__json_object *args = NULL;
+
+    libxl__qmp_param_add_integer(gc, &args, "fdset-id", fdset);
+
+    cis->qmp.callback = cdrom_insert_stubdom_ejected;

tools/libs/light/libxl_disk.c out: · line 1180

Quoted 10 lines
tools/libs/light/libxl_disk.c +10 -0
+
+    rc = libxl__ev_qmp_send(egc, &cis->qmp, "remove-fd", args);
+    if (rc) goto out;
+
+    return;
+
+ out:
+    if (rc == ERROR_NOTFOUND) {
+        LOGD(DEBUG, cis->domid, "No fdset found - skipping remove-fd");
+        cdrom_insert_stubdom_ejected(egc, qmp, resp, 0);

I think technically, cdrom_insert_stubdom_ejected also "must be last", like cdrom_insert_done, for one thing it actually call cdrom_insert_done in some cases. I think we used "/* must be last */" to indicate that resources used by the current chain of callback could be freed, including egc, gc, ao. There's quite a few more calls in this patch that would benefit from the annotation. But we can live without those.

Quoted 4 lines
tools/libs/light/libxl_disk.c +4 -0
+    } else {
+        cdrom_insert_done(egc, cis, rc); /* must be last */
+    }
+}

[...]

Quoted 20 lines
tools/libs/light/libxl_disk.c +20 -0
+static void cdrom_insert_stubdom_parse_fdset(libxl__egc *egc,
+                                             libxl__ev_qmp *qmp,
+                                             const libxl__json_object *response,
+                                             int rc)
+{
+    EGC_GC;
+    libxl__cdrom_insert_state *cis = CONTAINER_OF(qmp, *cis, qmp);
+    int devid;
+    int fdset;
+
+    if (rc) goto out;
+
+    /* Only called for qemu-xen/linux stubdom. */
+    assert(cis->dm_ver == LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN);
+
+    devid = libxl__device_disk_dev_number(cis->disk->vdev, NULL, NULL);
+    fdset = query_fdsets_find_fdset(gc, response, devid);
+    if (fdset < 0) {
+        rc = fdset;
+        goto out;

tools/libs/light/libxl_disk.c out: · line 1313

Quoted 10 lines
tools/libs/light/libxl_disk.c +10 -0
+    }
+
+    cis->stubdom_fdset = fdset;
+
+    LOGD(DEBUG, cis->domid, "Found fdset %d", cis->stubdom_fdset);
+    cdrom_insert_ejected(egc, &cis->qmp, NULL, rc);
+    return;
+
+ out:
+    if (rc == ERROR_NOTFOUND) {

While in the previous function it seems ok to deal with the NOTFOUND error in the "out:" path, I don't think it's a good idea here as it is an expected part of the workflow of this callback, and not an error.

Could you move setting this timer above again? I guess something like that would work fine:

  fdset = query_fdsets_find_fdset()
  if (fdset == ERROR_NOTFOUND) {
    // doesn't exist yet, wait a bit
    rc = libxl__ev_time_register_rel()
    if (rc) goto out;
    return
  }
  if (fdset < 0) {
  ...

Or also possible to deal with all error from query_fdsets..() first with something like:

  if (fdset < 0 && fdset != ERROR_NOTFOUND) {
    rc = fdset;
    goto out;

...

tools/libs/light/libxl_disk.c out: · line 1317

Quoted 4 lines
tools/libs/light/libxl_disk.c +4 -0
+        rc = libxl__ev_time_register_rel(cis->ao, &cis->retry_timer,
+                                         cdrom_insert_stubdom_query_fdset,
+                                         200);
+        if (rc) goto out;

That "goto out" after the "out" label makes this even stranger and even a potential infinite loop if rc would happen to be set to ERROR_NOTFOUND again, which I don't think can happen right now.

tools/libs/light/libxl_disk.c out: · line 1183

Quoted 5 lines
tools/libs/light/libxl_disk.c +5 -0
+        return;
+    }
+
+    cdrom_insert_done(egc, cis, rc); /* must be last */
+}

Otherwise patch looks good, with just the comment in cdrom_insert_stubdom_parse_fdset
act on: Reviewed-by: Anthony PERARD <[email protected]>

Thanks,

Signature · 5 lines
Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
#3 lore raw
Quoted 42 lines · Anthony PERARD
Quoted 29 lines · Jason Andryuk
tools/libs/light/libxl_disk.c +29 -0
+static void cdrom_insert_stubdom_parse_fdset_rm(libxl__egc *egc,
+                                                libxl__ev_qmp *qmp,
+                                                const libxl__json_object *resp,
+                                                int rc)
+{
+    EGC_GC;
+    libxl__cdrom_insert_state *cis = CONTAINER_OF(qmp, *cis, qmp);
+    int devid;
+    int fdset;
+
+    if (rc) goto out;
+
+    /* Only called for qemu-xen/linux stubdom. */
+    assert(cis->dm_ver == LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN);
+
+    devid = libxl__device_disk_dev_number(cis->disk->vdev, NULL, NULL);
+    fdset = query_fdsets_find_fdset(gc, resp, devid);
+    if (fdset < 0) {
+        rc = fdset;
+        goto out;
+    }
+
+    LOGD(DEBUG, cis->domid, "Found fdset %d", fdset);
+
+    libxl__json_object *args = NULL;
+
+    libxl__qmp_param_add_integer(gc, &args, "fdset-id", fdset);
+
+    cis->qmp.callback = cdrom_insert_stubdom_ejected;

tools/libs/light/libxl_disk.c out: · line 1180

Quoted 10 lines
tools/libs/light/libxl_disk.c +10 -0
+
+    rc = libxl__ev_qmp_send(egc, &cis->qmp, "remove-fd", args);
+    if (rc) goto out;
+
+    return;
+
+ out:
+    if (rc == ERROR_NOTFOUND) {
+        LOGD(DEBUG, cis->domid, "No fdset found - skipping remove-fd");
+        cdrom_insert_stubdom_ejected(egc, qmp, resp, 0);
Quoted 7 lines

I think technically, cdrom_insert_stubdom_ejected also "must be last", like cdrom_insert_done, for one thing it actually call cdrom_insert_done in some cases. I think we used "/* must be last */" to indicate that resources used by the current chain of callback could be freed, including egc, gc, ao. There's quite a few more calls in this patch that would benefit from the annotation. But we can live without those.

Thanks for the explanation. I'll add it here, and also in cdrom_insert_stubdom_disk_add_cb().

Quoted 43 lines
Quoted 4 lines
tools/libs/light/libxl_disk.c +4 -0
+    } else {
+        cdrom_insert_done(egc, cis, rc); /* must be last */
+    }
+}

[...]

Quoted 20 lines
tools/libs/light/libxl_disk.c +20 -0
+static void cdrom_insert_stubdom_parse_fdset(libxl__egc *egc,
+                                             libxl__ev_qmp *qmp,
+                                             const libxl__json_object *response,
+                                             int rc)
+{
+    EGC_GC;
+    libxl__cdrom_insert_state *cis = CONTAINER_OF(qmp, *cis, qmp);
+    int devid;
+    int fdset;
+
+    if (rc) goto out;
+
+    /* Only called for qemu-xen/linux stubdom. */
+    assert(cis->dm_ver == LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN);
+
+    devid = libxl__device_disk_dev_number(cis->disk->vdev, NULL, NULL);
+    fdset = query_fdsets_find_fdset(gc, response, devid);
+    if (fdset < 0) {
+        rc = fdset;
+        goto out;

tools/libs/light/libxl_disk.c out: · line 1313

Quoted 10 lines
tools/libs/light/libxl_disk.c +10 -0
+    }
+
+    cis->stubdom_fdset = fdset;
+
+    LOGD(DEBUG, cis->domid, "Found fdset %d", cis->stubdom_fdset);
+    cdrom_insert_ejected(egc, &cis->qmp, NULL, rc);
+    return;
+
+ out:
+    if (rc == ERROR_NOTFOUND) {

While in the previous function it seems ok to deal with the NOTFOUND error in the "out:" path, I don't think it's a good idea here as it is an expected part of the workflow of this callback, and not an error.

Could you move setting this timer above again? I guess something like that would work fine:

Quoted 8 lines
  fdset = query_fdsets_find_fdset()
  if (fdset == ERROR_NOTFOUND) {
    // doesn't exist yet, wait a bit
    rc = libxl__ev_time_register_rel()
    if (rc) goto out;
    return
  }
  if (fdset < 0) {

I'll go with this approach, and that means ...

Quoted 9 lines

tools/libs/light/libxl_disk.c out: · line 1317

Quoted 5 lines
tools/libs/light/libxl_disk.c +4 -0
+        rc = libxl__ev_time_register_rel(cis->ao, &cis->retry_timer,
+                                         cdrom_insert_stubdom_query_fdset,
+                                         200);
+        if (rc) goto out;

That "goto out" after the "out" label makes this even stranger and even a potential infinite loop if rc would happen to be set to ERROR_NOTFOUND again, which I don't think can happen right now.

...this will go away. I must have re-organized without really thinking through the result.

Quoted 10 lines

tools/libs/light/libxl_disk.c out: · line 1183

Quoted 5 lines
tools/libs/light/libxl_disk.c +5 -0
+        return;
+    }
+
+    cdrom_insert_done(egc, cis, rc); /* must be last */
+}

Otherwise patch looks good, with just the comment in cdrom_insert_stubdom_parse_fdset
act on: Reviewed-by: Anthony PERARD <[email protected]>

I'll refrain from adding this, since I am making a few more changes.

Thanks for the review.

-Jason