Skip to contentSkip to conversation

s390x/vfio-ap: Implement hot plug/unplug of vfio-ap device

PATCH Superseded Tony Krowiak 4 participants 14 messages Jan 8, 2019 – Jan 28, 2019

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 s390x/vfio-ap: Implement hot plug/unplug of vfio-ap device

Introduces hot plug/unplug support for the vfio-ap device. Note that only one vfio-ap device can be attached to the ap-bus, so a vfio-ap device can only be hot plugged if the '-device vfio-ap,sysfsdev=$path_to_mdev' option is not specified on the QEMU command line.

---
 hw/s390x/ap-bridge.c | 12 +++++++++++-
 hw/vfio/ap.c         |  2 +-
 2 files changed, 12 insertions(+), 2 deletions(-)
hw/s390x/ap-bridge.c +11 -1
index 3795d30dd7c9..25a03412fcb9 100644
@@ -39,6 +39,7 @@ static const TypeInfo ap_bus_info = {
 void s390_init_ap(void)
 {
     DeviceState *dev;
+    BusState *bus;
 
     /* If no AP instructions then no need for AP bridge */
     if (!s390_has_feat(S390_FEAT_AP)) {
@@ -52,13 +53,18 @@ void s390_init_ap(void)
     qdev_init_nofail(dev);
 
     /* Create bus on bridge device */
-    qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+    bus = qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+
+    /* Enable hotplugging */
+    qbus_set_hotplug_handler(bus, dev, &error_abort);
  }
 
 static void ap_bridge_class_init(ObjectClass *oc, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
+    hc->unplug = qdev_simple_device_unplug_cb;
     set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
 }
 
@@ -67,6 +73,10 @@ static const TypeInfo ap_bridge_info = {
     .parent        = TYPE_SYS_BUS_DEVICE,
     .instance_size = 0,
     .class_init    = ap_bridge_class_init,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_HOTPLUG_HANDLER },
+        { }
+    }
 };
 
 static void ap_register(void)
hw/vfio/ap.c +1 -1
index 6166ccd47a4a..d8b79ebe53ae 100644
@@ -169,7 +169,7 @@ static void vfio_ap_class_init(ObjectClass *klass, void *data)
     set_bit(DEVICE_CATEGORY_MISC, dc->categories);
     dc->realize = vfio_ap_realize;
     dc->unrealize = vfio_ap_unrealize;
-    dc->hotpluggable = false;
+    dc->hotpluggable = true;
     dc->reset = vfio_ap_reset;
     dc->bus_type = TYPE_AP_BUS;
 }
Signature · 1 line
2.7.4
#1 lore raw
Quoted 32 lines · Tony Krowiak

Introduces hot plug/unplug support for the vfio-ap device. Note that only one vfio-ap device can be attached to the ap-bus, so a vfio-ap device can only be hot plugged if the '-device vfio-ap,sysfsdev=$path_to_mdev' option is not specified on the QEMU command line.

---
 hw/s390x/ap-bridge.c | 12 +++++++++++-
 hw/vfio/ap.c         |  2 +-
 2 files changed, 12 insertions(+), 2 deletions(-)
hw/s390x/ap-bridge.c +3 -1
index 3795d30dd7c9..25a03412fcb9 100644
@@ -39,6 +39,7 @@ static const TypeInfo ap_bus_info = {
 void s390_init_ap(void)
 {
     DeviceState *dev;
+    BusState *bus;
 
     /* If no AP instructions then no need for AP bridge */
     if (!s390_has_feat(S390_FEAT_AP)) {
@@ -52,13 +53,18 @@ void s390_init_ap(void)
     qdev_init_nofail(dev);
 
     /* Create bus on bridge device */
-    qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+    bus = qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+

hw/s390x/ap-bridge.c void s390_init_ap(void) · line 67

Quoted 10 lines
hw/s390x/ap-bridge.c +4 -0
+    /* Enable hotplugging */
+    qbus_set_hotplug_handler(bus, dev, &error_abort);
  }
 
 static void ap_bridge_class_init(ObjectClass *oc, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
+    hc->unplug = qdev_simple_device_unplug_cb;

confused, why is there no plug action?

Quoted 28 lines
hw/s390x/ap-bridge.c +4 -0
     set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
 }
 
@@ -67,6 +73,10 @@ static const TypeInfo ap_bridge_info = {
     .parent        = TYPE_SYS_BUS_DEVICE,
     .instance_size = 0,
     .class_init    = ap_bridge_class_init,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_HOTPLUG_HANDLER },
+        { }
+    }
 };
 
 static void ap_register(void)
hw/vfio/ap.c +1 -1
index 6166ccd47a4a..d8b79ebe53ae 100644
@@ -169,7 +169,7 @@ static void vfio_ap_class_init(ObjectClass *klass, void *data)
     set_bit(DEVICE_CATEGORY_MISC, dc->categories);
     dc->realize = vfio_ap_realize;
     dc->unrealize = vfio_ap_unrealize;
-    dc->hotpluggable = false;
+    dc->hotpluggable = true;
     dc->reset = vfio_ap_reset;
     dc->bus_type = TYPE_AP_BUS;
 }
Signature · 3 lines
Thanks,

David / dhildenb
#2 lore raw
Quoted 37 lines · David Hildenbrand
Quoted 36 lines · Tony Krowiak

Introduces hot plug/unplug support for the vfio-ap device. Note that only one vfio-ap device can be attached to the ap-bus, so a vfio-ap device can only be hot plugged if the '-device vfio-ap,sysfsdev=$path_to_mdev' option is not specified on the QEMU command line.

---
  hw/s390x/ap-bridge.c | 12 +++++++++++-
  hw/vfio/ap.c         |  2 +-
  2 files changed, 12 insertions(+), 2 deletions(-)
hw/s390x/ap-bridge.c +5 -1
index 3795d30dd7c9..25a03412fcb9 100644
@@ -39,6 +39,7 @@ static const TypeInfo ap_bus_info = {
  void s390_init_ap(void)
  {
      DeviceState *dev;
+    BusState *bus;
 
      /* If no AP instructions then no need for AP bridge */
      if (!s390_has_feat(S390_FEAT_AP)) {
@@ -52,13 +53,18 @@ void s390_init_ap(void)
      qdev_init_nofail(dev);
 
      /* Create bus on bridge device */
-    qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+    bus = qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+
+    /* Enable hotplugging */
+    qbus_set_hotplug_handler(bus, dev, &error_abort);
   }
 
Quoted 8 lines

hw/s390x/ap-bridge.c void s390_init_ap(void) · line 67

Quoted 6 lines
hw/s390x/ap-bridge.c +2 -0
  static void ap_bridge_class_init(ObjectClass *oc, void *data)
  {
      DeviceClass *dc = DEVICE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
+    hc->unplug = qdev_simple_device_unplug_cb;

confused, why is there no plug action?

You will find this is the case for several devices that are hot pluggable. The plug callback is invoked after the device is attached to the bus and after the device is realized. Not having a hot plug callback does not preclude hot plugging of a device. I presume the purpose of the callback is to provide an opportunity to do perform any additional processing that may be required to prepare the device for use. In the case of the vfio-ap device, there is nothing to do once the device is plugged.

Quoted 31 lines
Quoted 29 lines
hw/s390x/ap-bridge.c +4 -0
      set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
  }
 
@@ -67,6 +73,10 @@ static const TypeInfo ap_bridge_info = {
      .parent        = TYPE_SYS_BUS_DEVICE,
      .instance_size = 0,
      .class_init    = ap_bridge_class_init,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_HOTPLUG_HANDLER },
+        { }
+    }
  };
 
  static void ap_register(void)
hw/vfio/ap.c +1 -1
index 6166ccd47a4a..d8b79ebe53ae 100644
@@ -169,7 +169,7 @@ static void vfio_ap_class_init(ObjectClass *klass, void *data)
      set_bit(DEVICE_CATEGORY_MISC, dc->categories);
      dc->realize = vfio_ap_realize;
      dc->unrealize = vfio_ap_unrealize;
-    dc->hotpluggable = false;
+    dc->hotpluggable = true;
      dc->reset = vfio_ap_reset;
      dc->bus_type = TYPE_AP_BUS;
  }
#3 lore raw
Quoted 43 lines · Tony Krowiak
Quoted 42 lines · David Hildenbrand
Quoted 41 lines · Tony Krowiak

Introduces hot plug/unplug support for the vfio-ap device. Note that only one vfio-ap device can be attached to the ap-bus, so a vfio-ap device can only be hot plugged if the '-device vfio-ap,sysfsdev=$path_to_mdev' option is not specified on the QEMU command line.

---
  hw/s390x/ap-bridge.c | 12 +++++++++++-
  hw/vfio/ap.c         |  2 +-
  2 files changed, 12 insertions(+), 2 deletions(-)
hw/s390x/ap-bridge.c +6 -1
index 3795d30dd7c9..25a03412fcb9 100644
@@ -39,6 +39,7 @@ static const TypeInfo ap_bus_info = {
  void s390_init_ap(void)
  {
      DeviceState *dev;
+    BusState *bus;
 
      /* If no AP instructions then no need for AP bridge */
      if (!s390_has_feat(S390_FEAT_AP)) {
@@ -52,13 +53,18 @@ void s390_init_ap(void)
      qdev_init_nofail(dev);
 
      /* Create bus on bridge device */
-    qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+    bus = qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+
+    /* Enable hotplugging */
+    qbus_set_hotplug_handler(bus, dev, &error_abort);
   }
 
  static void ap_bridge_class_init(ObjectClass *oc, void *data)
  {
      DeviceClass *dc = DEVICE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
Quoted 6 lines
Quoted 3 lines

hw/s390x/ap-bridge.c void s390_init_ap(void) · line 67

Quoted 1 line
hw/s390x/ap-bridge.c +1 -0
+    hc->unplug = qdev_simple_device_unplug_cb;

confused, why is there no plug action?

You will find this is the case for several devices that are hot pluggable.

Usually missing hotplug handlers are an indication of legacy code ;)

Quoted 3 lines

The plug callback is invoked after the device is attached to the bus and after the device is realized. Not having a hot plug callback does not preclude hot plugging of a device.

The hotplug handler is there to

1. Assign resources (e.g. ids etc)
2. Notify the system (e.g. hotplug interrupt)

In legacy code (e.g. PCI) such stuff is usually still located in the realize function (where it doesn't belong anymore but factoring out is hard ...)

Looking at hw/vfio/ap.c:realize(), there isn't really anything in there.

So I assume that

1. No resources have to be assigned (for vfio-ap, I guess the IDs and such are implicit)
2. No notification will happen. How will the guest know that a new AP adapter is available?

Quoted 4 lines

I presume the purpose of the callback is to provide an opportunity to do perform any additional processing that may be required to prepare the device for use. In the case of the vfio-ap device, there is nothing to do once the device is plugged.

When removing the device, is it really a forced removal? ("simply rip it out without telling the guest")

Signature · 3 lines
Thanks,

David / dhildenb
#4 lore raw
Quoted 66 lines · On Tue, 8 Jan 2019 23:13:39 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 43 lines · Tony Krowiak
Quoted 42 lines · David Hildenbrand
Quoted 41 lines · Tony Krowiak

Introduces hot plug/unplug support for the vfio-ap device. Note that only one vfio-ap device can be attached to the ap-bus, so a vfio-ap device can only be hot plugged if the '-device vfio-ap,sysfsdev=$path_to_mdev' option is not specified on the QEMU command line.

---
  hw/s390x/ap-bridge.c | 12 +++++++++++-
  hw/vfio/ap.c         |  2 +-
  2 files changed, 12 insertions(+), 2 deletions(-)
hw/s390x/ap-bridge.c +6 -1
index 3795d30dd7c9..25a03412fcb9 100644
@@ -39,6 +39,7 @@ static const TypeInfo ap_bus_info = {
  void s390_init_ap(void)
  {
      DeviceState *dev;
+    BusState *bus;
 
      /* If no AP instructions then no need for AP bridge */
      if (!s390_has_feat(S390_FEAT_AP)) {
@@ -52,13 +53,18 @@ void s390_init_ap(void)
      qdev_init_nofail(dev);
 
      /* Create bus on bridge device */
-    qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+    bus = qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+
+    /* Enable hotplugging */
+    qbus_set_hotplug_handler(bus, dev, &error_abort);
   }
 
  static void ap_bridge_class_init(ObjectClass *oc, void *data)
  {
      DeviceClass *dc = DEVICE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
Quoted 6 lines
Quoted 3 lines

hw/s390x/ap-bridge.c void s390_init_ap(void) · line 67

Quoted 1 line
hw/s390x/ap-bridge.c +1 -0
+    hc->unplug = qdev_simple_device_unplug_cb;  

confused, why is there no plug action?

You will find this is the case for several devices that are hot pluggable.

Usually missing hotplug handlers are an indication of legacy code ;)

Quoted 3 lines

The plug callback is invoked after the device is attached to the bus and after the device is realized. Not having a hot plug callback does not preclude hot plugging of a device.

The hotplug handler is there to

1. Assign resources (e.g. ids etc)
2. Notify the system (e.g. hotplug interrupt)

In legacy code (e.g. PCI) such stuff is usually still located in the realize function (where it doesn't belong anymore but factoring out is hard ...)

Quoted 6 lines

Looking at hw/vfio/ap.c:realize(), there isn't really anything in there.

So I assume that

1. No resources have to be assigned (for vfio-ap, I guess the IDs and such are implicit)

That's my understanding as well. The interesting stuff will be configured on kernel level before the device is even handed to QEMU for consumption.

(Would be nice to hint at that in the patch description.)

Quoted 2 lines

2. No notification will happen. How will the guest know that a new AP adapter is available?

My understanding is that hotplugging the matrix device will make the guest go from "no adapters/domains are available" to "some adapters/domains are available" (and unplug will do the reverse). I.e., no hot(un)plugging of individual queues (which would need to be done on the kernel level, and is tricky IIRC.)

I'm not sure what the architectured options for notifying the guest are (I dimly recall some kind of "AP configuration has changed event").

IIRC, the Linux guest driver scans for new queues periodically. Does it even do that if no queues are available to start with?

Quoted 8 lines
Quoted 5 lines

I presume the purpose of the callback is to provide an opportunity to do perform any additional processing that may be required to prepare the device for use. In the case of the vfio-ap device, there is nothing to do once the device is plugged.

When removing the device, is it really a forced removal? ("simply rip it out without telling the guest")

I hope that severing the connection between QEMU and the host kernel for AP takes care of cleanup. It's all a bit confusing :(

#5 lore raw
Quoted 71 lines · Cornelia Huck
Quoted 68 lines · On Tue, 8 Jan 2019 23:13:39 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 43 lines · Tony Krowiak
Quoted 42 lines · David Hildenbrand
Quoted 41 lines · Tony Krowiak

Introduces hot plug/unplug support for the vfio-ap device. Note that only one vfio-ap device can be attached to the ap-bus, so a vfio-ap device can only be hot plugged if the '-device vfio-ap,sysfsdev=$path_to_mdev' option is not specified on the QEMU command line.

---
   hw/s390x/ap-bridge.c | 12 +++++++++++-
   hw/vfio/ap.c         |  2 +-
   2 files changed, 12 insertions(+), 2 deletions(-)
hw/s390x/ap-bridge.c +6 -1
index 3795d30dd7c9..25a03412fcb9 100644
@@ -39,6 +39,7 @@ static const TypeInfo ap_bus_info = {
   void s390_init_ap(void)
   {
       DeviceState *dev;
+    BusState *bus;
 
       /* If no AP instructions then no need for AP bridge */
       if (!s390_has_feat(S390_FEAT_AP)) {
@@ -52,13 +53,18 @@ void s390_init_ap(void)
       qdev_init_nofail(dev);
 
       /* Create bus on bridge device */
-    qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+    bus = qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+
+    /* Enable hotplugging */
+    qbus_set_hotplug_handler(bus, dev, &error_abort);
    }
 
   static void ap_bridge_class_init(ObjectClass *oc, void *data)
   {
       DeviceClass *dc = DEVICE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
Quoted 6 lines
Quoted 3 lines

hw/s390x/ap-bridge.c void s390_init_ap(void) · line 67

Quoted 1 line
hw/s390x/ap-bridge.c +1 -0
+    hc->unplug = qdev_simple_device_unplug_cb;

confused, why is there no plug action?

You will find this is the case for several devices that are hot pluggable.

Usually missing hotplug handlers are an indication of legacy code ;)

Quoted 3 lines

The plug callback is invoked after the device is attached to the bus and after the device is realized. Not having a hot plug callback does not preclude hot plugging of a device.

The hotplug handler is there to

1. Assign resources (e.g. ids etc)
2. Notify the system (e.g. hotplug interrupt)

In legacy code (e.g. PCI) such stuff is usually still located in the realize function (where it doesn't belong anymore but factoring out is hard ...)

Looking at hw/vfio/ap.c:realize(), there isn't really anything in there.

Quoted 8 lines
Quoted 4 lines

So I assume that

1. No resources have to be assigned (for vfio-ap, I guess the IDs and such are implicit)

That's my understanding as well. The interesting stuff will be configured on kernel level before the device is even handed to QEMU for consumption.

The vfio-ap device represents a VFIO mdev device. AP resources - i.e.,
adapters, domains and control domains - are assigned to the mdev device
via its sysfs interfaces. This is all handled by the vfio_ap kernel
driver before a guest can use the mdev device. As part of vfio-ap device
realization, a file descriptor is opened on the mdev device. When the mdev device's fd is opened, a callback is invoked on the vfio_ap kernel device driver. The device driver then updates the guest's AP matrix
configuration based on the configuration specified via the mdev
device's sysfs interfaces.

Quoted 12 lines

(Would be nice to hint at that in the patch description.)

Quoted 2 lines

2. No notification will happen. How will the guest know that a new AP adapter is available?

My understanding is that hotplugging the matrix device will make the guest go from "no adapters/domains are available" to "some adapters/domains are available" (and unplug will do the reverse). I.e., no hot(un)plugging of individual queues (which would need to be done on the kernel level, and is tricky IIRC.)

Quoted 5 lines

I'm not sure what the architectured options for notifying the guest are (I dimly recall some kind of "AP configuration has changed event").

IIRC, the Linux guest driver scans for new queues periodically. Does it even do that if no queues are available to start with?

The AP bus - in this case, running in the guest - does a periodic scan for AP devices. The bus relies on an AP instruction that queries the AP configuration information. When the guest's AP matrix is updated - see description of mdev device fd open processing above - the query will provide the newly configured AP matrix and the bus will create the adapter and queue devices on the guest. Consequently, there is nothing to do in a hot plug handler. If you'd like, I'd be more than happy to include a hot plug handler that does some logging (or nothing at all) so it doesn't look like legacy code ;)

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

I presume the purpose of the callback is to provide an opportunity to do perform any additional processing that may be required to prepare the device for use. In the case of the vfio-ap device, there is nothing to do once the device is plugged.

When removing the device, is it really a forced removal? ("simply rip it out without telling the guest")

When the vfio-ap device is unrealized, the mdev device's file descriptor is closed. When the fd is closed, a callback is invoked on the vfio_ap kernel device driver. This callback clears the guest's AP matrix configuration and resets all of the AP queues affected. When the AP device scan is subsequently run by the AP bus on the guest, the AP instruction for querying the AP configuration will indicate that there are no AP devices configured. The bus will then remove the AP devices from the sysfs for the guest.

Quoted 3 lines

I hope that severing the connection between QEMU and the host kernel for AP takes care of cleanup. It's all a bit confusing :(

I agree, the AP architecture is complicated and confusing. As I said above, when the mdev device fd is closed, the vfio_ap kernel device driver will reset all AP queues affected by the de-configuration.

Quoted 1 line
Reviewed-by
#6 lore raw
Quoted 109 lines · Tony Krowiak
Quoted 71 lines · Cornelia Huck
Quoted 68 lines · On Tue, 8 Jan 2019 23:13:39 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 43 lines · Tony Krowiak
Quoted 42 lines · David Hildenbrand
Quoted 41 lines · Tony Krowiak

Introduces hot plug/unplug support for the vfio-ap device. Note that only one vfio-ap device can be attached to the ap-bus, so a vfio-ap device can only be hot plugged if the '-device vfio-ap,sysfsdev=$path_to_mdev' option is not specified on the QEMU command line.

---
   hw/s390x/ap-bridge.c | 12 +++++++++++-
   hw/vfio/ap.c         |  2 +-
   2 files changed, 12 insertions(+), 2 deletions(-)
hw/s390x/ap-bridge.c +6 -1
index 3795d30dd7c9..25a03412fcb9 100644
@@ -39,6 +39,7 @@ static const TypeInfo ap_bus_info = {
   void s390_init_ap(void)
   {
       DeviceState *dev;
+    BusState *bus;
 
       /* If no AP instructions then no need for AP bridge */
       if (!s390_has_feat(S390_FEAT_AP)) {
@@ -52,13 +53,18 @@ void s390_init_ap(void)
       qdev_init_nofail(dev);
 
       /* Create bus on bridge device */
-    qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+    bus = qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+
+    /* Enable hotplugging */
+    qbus_set_hotplug_handler(bus, dev, &error_abort);
    }
 
   static void ap_bridge_class_init(ObjectClass *oc, void *data)
   {
       DeviceClass *dc = DEVICE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
Quoted 6 lines
Quoted 3 lines

hw/s390x/ap-bridge.c void s390_init_ap(void) · line 67

Quoted 1 line
hw/s390x/ap-bridge.c +1 -0
+    hc->unplug = qdev_simple_device_unplug_cb;

confused, why is there no plug action?

You will find this is the case for several devices that are hot pluggable.

Usually missing hotplug handlers are an indication of legacy code ;)

Quoted 3 lines

The plug callback is invoked after the device is attached to the bus and after the device is realized. Not having a hot plug callback does not preclude hot plugging of a device.

The hotplug handler is there to

1. Assign resources (e.g. ids etc)
2. Notify the system (e.g. hotplug interrupt)

In legacy code (e.g. PCI) such stuff is usually still located in the realize function (where it doesn't belong anymore but factoring out is hard ...)

Looking at hw/vfio/ap.c:realize(), there isn't really anything in there.

Quoted 8 lines
Quoted 4 lines

So I assume that

1. No resources have to be assigned (for vfio-ap, I guess the IDs and such are implicit)

That's my understanding as well. The interesting stuff will be configured on kernel level before the device is even handed to QEMU for consumption.

The vfio-ap device represents a VFIO mdev device. AP resources - i.e., adapters, domains and control domains - are assigned to the mdev device via its sysfs interfaces. This is all handled by the vfio_ap kernel driver before a guest can use the mdev device. As part of vfio-ap device realization, a file descriptor is opened on the mdev device. When the mdev device's fd is opened, a callback is invoked on the vfio_ap kernel device driver. The device driver then updates the guest's AP matrix configuration based on the configuration specified via the mdev device's sysfs interfaces.

Quoted 12 lines

(Would be nice to hint at that in the patch description.)

Quoted 2 lines

2. No notification will happen. How will the guest know that a new AP adapter is available?

My understanding is that hotplugging the matrix device will make the guest go from "no adapters/domains are available" to "some adapters/domains are available" (and unplug will do the reverse). I.e., no hot(un)plugging of individual queues (which would need to be done on the kernel level, and is tricky IIRC.)

Quoted 5 lines

I'm not sure what the architectured options for notifying the guest are (I dimly recall some kind of "AP configuration has changed event").

IIRC, the Linux guest driver scans for new queues periodically. Does it even do that if no queues are available to start with?

Quoted 9 lines

The AP bus - in this case, running in the guest - does a periodic scan for AP devices. The bus relies on an AP instruction that queries the AP configuration information. When the guest's AP matrix is updated - see description of mdev device fd open processing above - the query will provide the newly configured AP matrix and the bus will create the adapter and queue devices on the guest. Consequently, there is nothing to do in a hot plug handler. If you'd like, I'd be more than happy to include a hot plug handler that does some logging (or nothing at all) so it doesn't look like legacy code ;)

Hehe, no it's fine for me.

Can you extend this patch description a little bit, including what we discussed here? With that

Quoted 31 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

I presume the purpose of the callback is to provide an opportunity to do perform any additional processing that may be required to prepare the device for use. In the case of the vfio-ap device, there is nothing to do once the device is plugged.

When removing the device, is it really a forced removal? ("simply rip it out without telling the guest")

When the vfio-ap device is unrealized, the mdev device's file descriptor is closed. When the fd is closed, a callback is invoked on the vfio_ap kernel device driver. This callback clears the guest's AP matrix configuration and resets all of the AP queues affected. When the AP device scan is subsequently run by the AP bus on the guest, the AP instruction for querying the AP configuration will indicate that there are no AP devices configured. The bus will then remove the AP devices from the sysfs for the guest.

Quoted 3 lines

I hope that severing the connection between QEMU and the host kernel for AP takes care of cleanup. It's all a bit confusing :(

I agree, the AP architecture is complicated and confusing. As I said above, when the mdev device fd is closed, the vfio_ap kernel device driver will reset all AP queues affected by the de-configuration.

Quoted 1 line
Signature · 3 lines
Thanks,

David / dhildenb
#7 lore raw
Quoted 120 lines · On Wed, 9 Jan 2019 17:37:49 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 109 lines · Tony Krowiak
Quoted 71 lines · Cornelia Huck
Quoted 68 lines · On Tue, 8 Jan 2019 23:13:39 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 43 lines · Tony Krowiak
Quoted 42 lines · David Hildenbrand
Quoted 41 lines · Tony Krowiak

Introduces hot plug/unplug support for the vfio-ap device. Note that only one vfio-ap device can be attached to the ap-bus, so a vfio-ap device can only be hot plugged if the '-device vfio-ap,sysfsdev=$path_to_mdev' option is not specified on the QEMU command line.

---
   hw/s390x/ap-bridge.c | 12 +++++++++++-
   hw/vfio/ap.c         |  2 +-
   2 files changed, 12 insertions(+), 2 deletions(-)
hw/s390x/ap-bridge.c +6 -1
index 3795d30dd7c9..25a03412fcb9 100644
@@ -39,6 +39,7 @@ static const TypeInfo ap_bus_info = {
   void s390_init_ap(void)
   {
       DeviceState *dev;
+    BusState *bus;
 
       /* If no AP instructions then no need for AP bridge */
       if (!s390_has_feat(S390_FEAT_AP)) {
@@ -52,13 +53,18 @@ void s390_init_ap(void)
       qdev_init_nofail(dev);
 
       /* Create bus on bridge device */
-    qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+    bus = qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+
+    /* Enable hotplugging */
+    qbus_set_hotplug_handler(bus, dev, &error_abort);
    }
 
   static void ap_bridge_class_init(ObjectClass *oc, void *data)
   {
       DeviceClass *dc = DEVICE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
Quoted 6 lines
Quoted 3 lines

hw/s390x/ap-bridge.c void s390_init_ap(void) · line 67

Quoted 1 line
hw/s390x/ap-bridge.c +1 -0
+    hc->unplug = qdev_simple_device_unplug_cb;

confused, why is there no plug action?

You will find this is the case for several devices that are hot pluggable.

Usually missing hotplug handlers are an indication of legacy code ;)

Quoted 3 lines

The plug callback is invoked after the device is attached to the bus and after the device is realized. Not having a hot plug callback does not preclude hot plugging of a device.

The hotplug handler is there to

1. Assign resources (e.g. ids etc)
2. Notify the system (e.g. hotplug interrupt)

In legacy code (e.g. PCI) such stuff is usually still located in the realize function (where it doesn't belong anymore but factoring out is hard ...)

Looking at hw/vfio/ap.c:realize(), there isn't really anything in there.

Quoted 8 lines
Quoted 4 lines

So I assume that

1. No resources have to be assigned (for vfio-ap, I guess the IDs and such are implicit)

That's my understanding as well. The interesting stuff will be configured on kernel level before the device is even handed to QEMU for consumption.

The vfio-ap device represents a VFIO mdev device. AP resources - i.e., adapters, domains and control domains - are assigned to the mdev device via its sysfs interfaces. This is all handled by the vfio_ap kernel driver before a guest can use the mdev device. As part of vfio-ap device realization, a file descriptor is opened on the mdev device. When the mdev device's fd is opened, a callback is invoked on the vfio_ap kernel device driver. The device driver then updates the guest's AP matrix configuration based on the configuration specified via the mdev device's sysfs interfaces.

Quoted 12 lines

(Would be nice to hint at that in the patch description.)

Quoted 2 lines

2. No notification will happen. How will the guest know that a new AP adapter is available?

My understanding is that hotplugging the matrix device will make the guest go from "no adapters/domains are available" to "some adapters/domains are available" (and unplug will do the reverse). I.e., no hot(un)plugging of individual queues (which would need to be done on the kernel level, and is tricky IIRC.)

Quoted 5 lines

I'm not sure what the architectured options for notifying the guest are (I dimly recall some kind of "AP configuration has changed event").

IIRC, the Linux guest driver scans for new queues periodically. Does it even do that if no queues are available to start with?

Quoted 9 lines

The AP bus - in this case, running in the guest - does a periodic scan for AP devices. The bus relies on an AP instruction that queries the AP configuration information. When the guest's AP matrix is updated - see description of mdev device fd open processing above - the query will provide the newly configured AP matrix and the bus will create the adapter and queue devices on the guest. Consequently, there is nothing to do in a hot plug handler. If you'd like, I'd be more than happy to include a hot plug handler that does some logging (or nothing at all) so it doesn't look like legacy code ;)

Quoted 4 lines

Hehe, no it's fine for me.

Can you extend this patch description a little bit, including what we discussed here?

Maybe a short comment that explains why qdev_simple_device_unplug_cb() is appropriate as well (i.e. hits that closing the mdev's fd is what triggers the cleanup of the actual resources)? I personally go log digging only once I get desperate.

Regards,
Halil

#8 lore raw
Quoted 125 lines · Halil Pasic
Quoted 122 lines · On Wed, 9 Jan 2019 17:37:49 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 109 lines · Tony Krowiak
Quoted 71 lines · Cornelia Huck
Quoted 68 lines · On Tue, 8 Jan 2019 23:13:39 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 43 lines · Tony Krowiak
Quoted 42 lines · David Hildenbrand
Quoted 41 lines · Tony Krowiak

Introduces hot plug/unplug support for the vfio-ap device. Note that only one vfio-ap device can be attached to the ap-bus, so a vfio-ap device can only be hot plugged if the '-device vfio-ap,sysfsdev=$path_to_mdev' option is not specified on the QEMU command line.

---
   hw/s390x/ap-bridge.c | 12 +++++++++++-
   hw/vfio/ap.c         |  2 +-
   2 files changed, 12 insertions(+), 2 deletions(-)
hw/s390x/ap-bridge.c +6 -1
index 3795d30dd7c9..25a03412fcb9 100644
@@ -39,6 +39,7 @@ static const TypeInfo ap_bus_info = {
   void s390_init_ap(void)
   {
       DeviceState *dev;
+    BusState *bus;
 
       /* If no AP instructions then no need for AP bridge */
       if (!s390_has_feat(S390_FEAT_AP)) {
@@ -52,13 +53,18 @@ void s390_init_ap(void)
       qdev_init_nofail(dev);
 
       /* Create bus on bridge device */
-    qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+    bus = qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+
+    /* Enable hotplugging */
+    qbus_set_hotplug_handler(bus, dev, &error_abort);
    }
 
   static void ap_bridge_class_init(ObjectClass *oc, void *data)
   {
       DeviceClass *dc = DEVICE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
Quoted 6 lines
Quoted 3 lines

hw/s390x/ap-bridge.c void s390_init_ap(void) · line 67

Quoted 1 line
hw/s390x/ap-bridge.c +1 -0
+    hc->unplug = qdev_simple_device_unplug_cb;

confused, why is there no plug action?

You will find this is the case for several devices that are hot pluggable.

Usually missing hotplug handlers are an indication of legacy code ;)

Quoted 3 lines

The plug callback is invoked after the device is attached to the bus and after the device is realized. Not having a hot plug callback does not preclude hot plugging of a device.

The hotplug handler is there to

1. Assign resources (e.g. ids etc)
2. Notify the system (e.g. hotplug interrupt)

In legacy code (e.g. PCI) such stuff is usually still located in the realize function (where it doesn't belong anymore but factoring out is hard ...)

Looking at hw/vfio/ap.c:realize(), there isn't really anything in there.

Quoted 8 lines
Quoted 4 lines

So I assume that

1. No resources have to be assigned (for vfio-ap, I guess the IDs and such are implicit)

That's my understanding as well. The interesting stuff will be configured on kernel level before the device is even handed to QEMU for consumption.

The vfio-ap device represents a VFIO mdev device. AP resources - i.e., adapters, domains and control domains - are assigned to the mdev device via its sysfs interfaces. This is all handled by the vfio_ap kernel driver before a guest can use the mdev device. As part of vfio-ap device realization, a file descriptor is opened on the mdev device. When the mdev device's fd is opened, a callback is invoked on the vfio_ap kernel device driver. The device driver then updates the guest's AP matrix configuration based on the configuration specified via the mdev device's sysfs interfaces.

Quoted 12 lines

(Would be nice to hint at that in the patch description.)

Quoted 2 lines

2. No notification will happen. How will the guest know that a new AP adapter is available?

My understanding is that hotplugging the matrix device will make the guest go from "no adapters/domains are available" to "some adapters/domains are available" (and unplug will do the reverse). I.e., no hot(un)plugging of individual queues (which would need to be done on the kernel level, and is tricky IIRC.)

Quoted 5 lines

I'm not sure what the architectured options for notifying the guest are (I dimly recall some kind of "AP configuration has changed event").

IIRC, the Linux guest driver scans for new queues periodically. Does it even do that if no queues are available to start with?

Quoted 9 lines

The AP bus - in this case, running in the guest - does a periodic scan for AP devices. The bus relies on an AP instruction that queries the AP configuration information. When the guest's AP matrix is updated - see description of mdev device fd open processing above - the query will provide the newly configured AP matrix and the bus will create the adapter and queue devices on the guest. Consequently, there is nothing to do in a hot plug handler. If you'd like, I'd be more than happy to include a hot plug handler that does some logging (or nothing at all) so it doesn't look like legacy code ;)

Hehe, no it's fine for me.

Quoted 7 lines
Quoted 2 lines

Can you extend this patch description a little bit, including what we discussed here?

Maybe a short comment that explains why qdev_simple_device_unplug_cb() is appropriate as well (i.e. hits that closing the mdev's fd is what triggers the cleanup of the actual resources)? I personally go log digging only once I get desperate.

I go digging if I can't find a public document on how it works ;)

Quoted 4 lines

Regards,
Halil

Signature · 3 lines
Thanks,

David / dhildenb
#9 lore raw
Quoted 129 lines · David Hildenbrand
Quoted 128 lines · Halil Pasic
Quoted 124 lines · On Wed, 9 Jan 2019 17:37:49 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 109 lines · Tony Krowiak
Quoted 71 lines · Cornelia Huck
Quoted 68 lines · On Tue, 8 Jan 2019 23:13:39 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 43 lines · Tony Krowiak
Quoted 42 lines · David Hildenbrand
Quoted 41 lines · Tony Krowiak

Introduces hot plug/unplug support for the vfio-ap device. Note that only one vfio-ap device can be attached to the ap-bus, so a vfio-ap device can only be hot plugged if the '-device vfio-ap,sysfsdev=$path_to_mdev' option is not specified on the QEMU command line.

---
    hw/s390x/ap-bridge.c | 12 +++++++++++-
    hw/vfio/ap.c         |  2 +-
    2 files changed, 12 insertions(+), 2 deletions(-)
hw/s390x/ap-bridge.c +6 -1
index 3795d30dd7c9..25a03412fcb9 100644
@@ -39,6 +39,7 @@ static const TypeInfo ap_bus_info = {
    void s390_init_ap(void)
    {
        DeviceState *dev;
+    BusState *bus;
 
        /* If no AP instructions then no need for AP bridge */
        if (!s390_has_feat(S390_FEAT_AP)) {
@@ -52,13 +53,18 @@ void s390_init_ap(void)
        qdev_init_nofail(dev);
 
        /* Create bus on bridge device */
-    qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+    bus = qbus_create(TYPE_AP_BUS, dev, TYPE_AP_BUS);
+
+    /* Enable hotplugging */
+    qbus_set_hotplug_handler(bus, dev, &error_abort);
     }
 
    static void ap_bridge_class_init(ObjectClass *oc, void *data)
    {
        DeviceClass *dc = DEVICE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
Quoted 6 lines
Quoted 3 lines

hw/s390x/ap-bridge.c void s390_init_ap(void) · line 67

Quoted 1 line
hw/s390x/ap-bridge.c +1 -0
+    hc->unplug = qdev_simple_device_unplug_cb;

confused, why is there no plug action?

You will find this is the case for several devices that are hot pluggable.

Usually missing hotplug handlers are an indication of legacy code ;)

Quoted 3 lines

The plug callback is invoked after the device is attached to the bus and after the device is realized. Not having a hot plug callback does not preclude hot plugging of a device.

The hotplug handler is there to

1. Assign resources (e.g. ids etc)
2. Notify the system (e.g. hotplug interrupt)

In legacy code (e.g. PCI) such stuff is usually still located in the realize function (where it doesn't belong anymore but factoring out is hard ...)

Looking at hw/vfio/ap.c:realize(), there isn't really anything in there.

Quoted 8 lines
Quoted 4 lines

So I assume that

1. No resources have to be assigned (for vfio-ap, I guess the IDs and such are implicit)

That's my understanding as well. The interesting stuff will be configured on kernel level before the device is even handed to QEMU for consumption.

The vfio-ap device represents a VFIO mdev device. AP resources - i.e., adapters, domains and control domains - are assigned to the mdev device via its sysfs interfaces. This is all handled by the vfio_ap kernel driver before a guest can use the mdev device. As part of vfio-ap device realization, a file descriptor is opened on the mdev device. When the mdev device's fd is opened, a callback is invoked on the vfio_ap kernel device driver. The device driver then updates the guest's AP matrix configuration based on the configuration specified via the mdev device's sysfs interfaces.

Quoted 12 lines

(Would be nice to hint at that in the patch description.)

Quoted 2 lines

2. No notification will happen. How will the guest know that a new AP adapter is available?

My understanding is that hotplugging the matrix device will make the guest go from "no adapters/domains are available" to "some adapters/domains are available" (and unplug will do the reverse). I.e., no hot(un)plugging of individual queues (which would need to be done on the kernel level, and is tricky IIRC.)

Quoted 5 lines

I'm not sure what the architectured options for notifying the guest are (I dimly recall some kind of "AP configuration has changed event").

IIRC, the Linux guest driver scans for new queues periodically. Does it even do that if no queues are available to start with?

Quoted 9 lines

The AP bus - in this case, running in the guest - does a periodic scan for AP devices. The bus relies on an AP instruction that queries the AP configuration information. When the guest's AP matrix is updated - see description of mdev device fd open processing above - the query will provide the newly configured AP matrix and the bus will create the adapter and queue devices on the guest. Consequently, there is nothing to do in a hot plug handler. If you'd like, I'd be more than happy to include a hot plug handler that does some logging (or nothing at all) so it doesn't look like legacy code ;)

Hehe, no it's fine for me.

Can you extend this patch description a little bit, including what we discussed here?

Quoted 6 lines
Quoted 4 lines

Maybe a short comment that explains why qdev_simple_device_unplug_cb() is appropriate as well (i.e. hits that closing the mdev's fd is what triggers the cleanup of the actual resources)? I personally go log digging only once I get desperate.

I go digging if I can't find a public document on how it works ;)

Which reminds me, I will also need to update the docs/vfio-ap.txt document.

Quoted 7 lines
Quoted 5 lines

Regards,
Halil

#10 lore raw
Quoted 8 lines · On Thu, 10 Jan 2019 11:22:48 -0500 Tony Krowiak <[email protected]> wrote:
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Can you extend this patch description a little bit, including what we discussed here?

Maybe a short comment that explains why qdev_simple_device_unplug_cb() is appropriate as well (i.e. hits that closing the mdev's fd is what triggers the cleanup of the actual resources)? I personally go log digging only once I get desperate.

Quoted 4 lines
Quoted 1 line

I go digging if I can't find a public document on how it works ;)

Which reminds me, I will also need to update the docs/vfio-ap.txt document.

So, I assume there will be a v2 with some documentation changes?

#11 lore raw
Quoted 11 lines · Cornelia Huck
Quoted 8 lines · On Thu, 10 Jan 2019 11:22:48 -0500 Tony Krowiak <[email protected]> wrote:
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Can you extend this patch description a little bit, including what we discussed here?

Maybe a short comment that explains why qdev_simple_device_unplug_cb() is appropriate as well (i.e. hits that closing the mdev's fd is what triggers the cleanup of the actual resources)? I personally go log digging only once I get desperate.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I go digging if I can't find a public document on how it works ;)

Which reminds me, I will also need to update the docs/vfio-ap.txt document.

So, I assume there will be a v2 with some documentation changes?

I'm sorry Connie, I got sidetracked with some other priorities and missed this response. I will post a v2 with doc changes.

Quoted 1 line
#12 lore raw
In reply to Tony Krowiak
Quoted 5 lines · Tony Krowiak

Introduces hot plug/unplug support for the vfio-ap device. Note that only one vfio-ap device can be attached to the ap-bus, so a vfio-ap device can only be hot plugged if the '-device vfio-ap,sysfsdev=$path_to_mdev' option is not specified on the QEMU command line.

Quoted 8 lines
---
  hw/s390x/ap-bridge.c | 12 +++++++++++-
  hw/vfio/ap.c         |  2 +-
  2 files changed, 12 insertions(+), 2 deletions(-)

Please note: During testing of this patch I discovered a bug whereby an attempt to plug in a vfio-ap device subsequent to an unplug of the device will result in the following error:

error: Failed to attach device from /root/conf/vfio-ap.xml
error: internal error: unable to execute QEMU command 'device_add': No 'ap-bus' bus found for device 'vfio-ap'

I wrote a patch for this bug:
Message ID: <[email protected]>

Reviewed-by on v1 1/1
#13 lore raw
In reply to Tony Krowiak
Quoted 8 lines · On Tue, 8 Jan 2019 11:01:50 -0500 Tony Krowiak <[email protected]> wrote:

Introduces hot plug/unplug support for the vfio-ap device. Note that only one vfio-ap device can be attached to the ap-bus, so a vfio-ap device can only be hot plugged if the '-device vfio-ap,sysfsdev=$path_to_mdev' option is not specified on the QEMU command line.