Skip to contentSkip to conversation

acpi: pcihp: allow repeating hot-unplug requests

PATCH In review Igor Mammedov 5 participants 7 messages Apr 18, 2023 – May 4, 2023

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 acpi: pcihp: allow repeating hot-unplug requests

with Q35 using ACPI PCI hotplug by default, user's request to unplug device is ignored when it's issued before guest OS has been booted. And any additional attempt to request device hot-unplug afterwards results in following error:

"Device XYZ is already in the process of unplug"

arguably it can be considered as a regression introduced by [2], before which it was possible to issue unplug request multiple times.

Accept new uplug requests after timeout (1ms). This brings ACPI PCI hotplug on par with native PCIe unplug behavior [1] and allows user to repeat unplug requests at propper times.
Set expire timeout to arbitrary 1msec so user won't be able to flood guest with SCI interrupts by calling device_del in tight loop.

PS:
ACPI spec doesn't mandate what OSPM can do with GPEx.status bits set before it's booted => it's impl. depended. Status bits may be retained (I tested with one Windows version) or cleared (Linux since 2.6 kernel times) during guest's ACPI subsystem initialization.
Clearing status bits (though not wrong per se) hides the unplug event from guest, and it's upto user to repeat device_del later when guest is able to handle unplug requests.

1) 18416c62e3 ("pcie: expire pending delete")
2)

Fixes cce8944cc9ef ("qdev-monitor: Forbid repeated device_del")

---
v4:
* massage commit message some more (Kashyap Chamarthy <[email protected]>)
* pickup Gerd's ACK

---
 hw/acpi/pcihp.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
hw/acpi/pcihp.c +10 -0
index dcfb779a7a..cdd6f775a1 100644
@@ -357,6 +357,16 @@ void acpi_pcihp_device_unplug_request_cb(HotplugHandler *hotplug_dev,
      * acpi_pcihp_eject_slot() when the operation is completed.
      */
     pdev->qdev.pending_deleted_event = true;
+    /* if unplug was requested before OSPM is initialized,
+     * linux kernel will clear GPE0.sts[] bits during boot, which effectively
+     * hides unplug event. And than followup qmp_device_del() calls remain
+     * blocked by above flag permanently.
+     * Unblock qmp_device_del() by setting expire limit, so user can
+     * repeat unplug request later when OSPM has been booted.
+     */
+    pdev->qdev.pending_deleted_expires_ms =
+        qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL); /* 1 msec */
+
     s->acpi_pcihp_pci_status[bsel].down |= (1U << slot);
     acpi_send_event(DEVICE(hotplug_dev), ACPI_PCI_HOTPLUG_STATUS);
 }
Signature · 1 line
2.39.1
Reviewed-by on v4 1/1
#1 lore raw
Quoted 27 lines · Igor Mammedov

with Q35 using ACPI PCI hotplug by default, user's request to unplug device is ignored when it's issued before guest OS has been booted. And any additional attempt to request device hot-unplug afterwards results in following error:

"Device XYZ is already in the process of unplug"

arguably it can be considered as a regression introduced by [2], before which it was possible to issue unplug request multiple times.

Accept new uplug requests after timeout (1ms). This brings ACPI PCI hotplug on par with native PCIe unplug behavior [1] and allows user to repeat unplug requests at propper times.
Set expire timeout to arbitrary 1msec so user won't be able to flood guest with SCI interrupts by calling device_del in tight loop.

PS:
ACPI spec doesn't mandate what OSPM can do with GPEx.status bits set before it's booted => it's impl. depended. Status bits may be retained (I tested with one Windows version) or cleared (Linux since 2.6 kernel times) during guest's ACPI subsystem initialization.
Clearing status bits (though not wrong per se) hides the unplug event from guest, and it's upto user to repeat device_del later when guest is able to handle unplug requests.

Quoted 5 lines

1) 18416c62e3 ("pcie: expire pending delete")
2)

Fixes cce8944cc9ef ("qdev-monitor: Forbid repeated device_del")
Quoted 37 lines

---
v4:
* massage commit message some more (Kashyap Chamarthy <[email protected]>)
* pickup Gerd's ACK

---
 hw/acpi/pcihp.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
hw/acpi/pcihp.c +10 -1
index dcfb779a7a..cdd6f775a1 100644
@@ -357,6 +357,16 @@ void acpi_pcihp_device_unplug_request_cb(HotplugHandler *hotplug_dev,
      * acpi_pcihp_eject_slot() when the operation is completed.
      */
     pdev->qdev.pending_deleted_event = true;
+    /* if unplug was requested before OSPM is initialized,
+     * linux kernel will clear GPE0.sts[] bits during boot, which effectively
+     * hides unplug event. And than followup qmp_device_del() calls remain
+     * blocked by above flag permanently.
+     * Unblock qmp_device_del() by setting expire limit, so user can
+     * repeat unplug request later when OSPM has been booted.
+     */
+    pdev->qdev.pending_deleted_expires_ms =
+        qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL); /* 1 msec */
+
     s->acpi_pcihp_pci_status[bsel].down |= (1U << slot);
     acpi_send_event(DEVICE(hotplug_dev), ACPI_PCI_HOTPLUG_STATUS);
 }
--

2.39.1

#2 lore raw
In reply to Igor Mammedov
Quoted 10 lines · Igor Mammedov

with Q35 using ACPI PCI hotplug by default, user's request to unplug device is ignored when it's issued before guest OS has been booted. And any additional attempt to request device hot-unplug afterwards results in following error:

"Device XYZ is already in the process of unplug"

arguably it can be considered as a regression introduced by [2], before which it was possible to issue unplug request multiple times.

Stable-8.0 material?

Thanks,

/mjt

#3 lore raw
Quoted 6 lines · Michael Tokarev
Quoted 5 lines · Igor Mammedov

with Q35 using ACPI PCI hotplug by default, user's request to unplug device is ignored when it's issued before guest OS has been booted. And any additional attempt to request device hot-unplug afterwards results in following error:

Quoted 7 lines
Quoted 5 lines

"Device XYZ is already in the process of unplug"

arguably it can be considered as a regression introduced by [2], before which it was possible to issue unplug request multiple times.

Stable-8.0 material?

FWIW, I'd say, yes. This fix is useful for stable releases. As this solves a real problem for upper-management tools.

I have tested this fix; and it works. I'll post my testing notes / reproducer in a follow-up email. In short, I followed the reproducer steps from here[1].

[1] https://gitlab.com/libvirt/libvirt/-/issues/309

Signature · 1 line
/kashyap
#4 lore raw
Quoted 15 lines · Kashyap Chamarthy
Quoted 6 lines · Michael Tokarev
Quoted 5 lines · Igor Mammedov

with Q35 using ACPI PCI hotplug by default, user's request to unplug device is ignored when it's issued before guest OS has been booted. And any additional attempt to request device hot-unplug afterwards results in following error:

Quoted 7 lines
Quoted 5 lines

"Device XYZ is already in the process of unplug"

arguably it can be considered as a regression introduced by [2], before which it was possible to issue unplug request multiple times.

Stable-8.0 material?

Quoted 9 lines

FWIW, I'd say, yes. This fix is useful for stable releases. As this solves a real problem for upper-management tools.

I have tested this fix; and it works. I'll post my testing notes / reproducer in a follow-up email. In short, I followed the reproducer steps from here[1].

[1] https://gitlab.com/libvirt/libvirt/-/issues/309

You can CC stable then.

Quoted 2 lines
Signature · 1 line
/kashyap
#5 lore raw

27.04.2023 10:01, Michael S. Tsirkin пишет:
..

Quoted 1 line

You can CC stable then.

Yes, please do in the future.
I picked up this one already.

Thanks!

/mjt

#6 lore raw
In reply to Kashyap Chamarthy
Quoted 13 lines · Kashyap Chamarthy
Quoted 12 lines · Michael Tokarev
Quoted 10 lines · Igor Mammedov

with Q35 using ACPI PCI hotplug by default, user's request to unplug device is ignored when it's issued before guest OS has been booted. And any additional attempt to request device hot-unplug afterwards results in following error:

"Device XYZ is already in the process of unplug"

arguably it can be considered as a regression introduced by [2], before which it was possible to issue unplug request multiple times.

Quoted 8 lines
Quoted 1 line

Stable-8.0 material?

FWIW, I'd say, yes. This fix is useful for stable releases. As this solves a real problem for upper-management tools.

I have tested this fix; and it works. I'll post my testing notes / reproducer in a follow-up email. In short, I followed the reproducer steps from here[1].

It solves the device-detach bug noted here[1]. As promised, here are my reproducer notes (expanded from[1]):

Disk image prep
---------------

(1) Download an Ubuntu "Jammy" guest image from here:
https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img

(2) Update the above disk image's kernel command-line to have the guest-boot slowed down by 100 seconds; use "boot_delay=100".

(3) Have an additional image ("disk2.img") ready for hot-plug/un-plug.

Test
----

(1) Build QEMU with the patch in question:

      $ git describe
      7.2.94v8.0.0-rc4-1-gfa6650df6d7

(2) Use the above QEMU binary to launch the Ubuntu "Jammy" guest:

      $ virsh dumpxml jammy1 | grep emulator
      <emulator>/home/kashyapc/tinker-space/qemu-upstream/build/qemu-system-x86_64</emulator>

(3) Have a split tmux ready; start the guest in the first pane, with the serial console logs rolling:

$ virsh start jammy1 --console

(4) Wait until the guest consoles messages start rolling. Once they do, on the other tmux pane, issue the below command (it's a live attach, followed by a detach):

$ virsh attach-disk jammy1 ./disk2.img vdb --live --persistent \
&& sleep 1 \
&& virsh detach-disk jammy1 --live ./disk2.img

Disk attached successfully

Disk detached successfully

(5) Enumerate the attached block devices to the guest. We still see the second disk, "disk2.img":

$> virsh domblklist jammy1
Target Source
-------------------------------------------
vda /data/images/jammy-ubuntu.qcow2
vdb /data/images/disk2.img

(6) Now detach the disk from the inactive guest XML (that affects next boot) by using "--persistent" flag; and enumerate the live block devices (we still see the second disk)

$> virsh detach-disk jammy1 --persistent /data/images/disk2.img Disk detached successfully

$> virsh domblklist jammy1
Target Source
-------------------------------------------
vda /data/images/jammy-ubuntu.qcow2
vdb /data/images/disk2.img

(NOTE: We're using two separate calls to virsh detach-disk, one with "--live" and the other with "--persistent" based on upstream libvirt recommendation in [1].)

(7) Again, re-issue the detach command with just "--live" flag:

$> virsh detach-disk jammy1 --live /data/images/disk2.img Disk detached successfully

(8) Re-enumerate the attached block devices:

$> virsh domblklist jammy1
Target Source
-------------------------------------------
vda /data/images/jammy-ubuntu.qcow2

Now we see the second device is detached "for real". Overall, we were able to successfully re-issue device detach while the guest is still booting, and see through the actual detach to its logical conclusion.

[1] https://gitlab.com/libvirt/libvirt/-/issues/309 -- Disk detach is unsuccessfull while the guest is still booting

Signature · 1 line
/kashyap