Skip to contentSkip to conversation

pci: Refuse to hotplug PCI Devices when the Guest OS is not ready

PATCH In review Marcel Apfelbaum 6 participants 33 messages Oct 22, 2020 – Nov 15, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pci: Refuse to hotplug PCI Devices when the Guest OS is not ready

From: Marcel Apfelbaum <[email protected]>

During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) the "Slot Control Register" has the "Power Indicator Control"
set to "Blinking" expressing a "power transition" mode.

Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.

Detect and refuse hotplug operations in such case.

---
 hw/pci/pcie.c | 7 +++++++
 1 file changed, 7 insertions(+)
hw/pci/pcie.c +7 -0
index 5b48bae0f6..2fe5c1473f 100644
@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
     uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;
     uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
+    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL);
 
     /* Check if hot-plug is disabled on the slot */
     if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) {
@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         return;
     }
 
+    if ((sltctl & PCI_EXP_SLTCTL_PIC) == PCI_EXP_SLTCTL_PWR_IND_BLINK) {
+        error_setg(errp, "Hot-plug failed: %s is in Power Transition",
+                   DEVICE(hotplug_pdev)->id);
+        return;
+    }
+
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
 }
 
Signature · 1 line
2.17.2
#1 lore raw
Quoted 30 lines · Marcel Apfelbaum

From: Marcel Apfelbaum <[email protected]>

During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) the "Slot Control Register" has the "Power Indicator Control"
set to "Blinking" expressing a "power transition" mode.

Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.

Detect and refuse hotplug operations in such case.

---
 hw/pci/pcie.c | 7 +++++++
 1 file changed, 7 insertions(+)
hw/pci/pcie.c +1 -0
index 5b48bae0f6..2fe5c1473f 100644
@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
     uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;
     uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
+    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL);
 
     /* Check if hot-plug is disabled on the slot */
     if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) {
@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         return;

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 429

Quoted 10 lines
hw/pci/pcie.c +6 -0
     }
 
+    if ((sltctl & PCI_EXP_SLTCTL_PIC) == PCI_EXP_SLTCTL_PWR_IND_BLINK) {
+        error_setg(errp, "Hot-plug failed: %s is in Power Transition",
+                   DEVICE(hotplug_pdev)->id);
+        return;
+    }
+
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
 }

Probably the only way to handle for existing machine types. For new ones, can't we queue it in host memory somewhere?

Quoted 2 lines
Signature · 1 line
2.17.2
#2 lore raw
Quoted 33 lines · On Thu, 22 Oct 2020 08:06:55 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 32 lines · Marcel Apfelbaum

From: Marcel Apfelbaum <[email protected]>

During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) the "Slot Control Register" has the "Power Indicator Control"
set to "Blinking" expressing a "power transition" mode.

Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.

Detect and refuse hotplug operations in such case.

---
 hw/pci/pcie.c | 7 +++++++
 1 file changed, 7 insertions(+)
hw/pci/pcie.c +1 -0
index 5b48bae0f6..2fe5c1473f 100644
@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
     uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;
     uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
+    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL);
 
     /* Check if hot-plug is disabled on the slot */
     if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) {
@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         return;
     }
 
Quoted 11 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 429

Quoted 8 lines
hw/pci/pcie.c +6 -0
+    if ((sltctl & PCI_EXP_SLTCTL_PIC) == PCI_EXP_SLTCTL_PWR_IND_BLINK) {
+        error_setg(errp, "Hot-plug failed: %s is in Power Transition",
+                   DEVICE(hotplug_pdev)->id);
+        return;
+    }
+
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
 }  

Probably the only way to handle for existing machine types. For new ones, can't we queue it in host memory somewhere?

I'm not actually convinced we can't do that even for existing machine types. So I'm a bit hesitant to suggest going ahead with this without looking a bit closer at whether we can implement a wait-for-ready in qemu, rather than forcing every user of qemu (human or machine) to do so.

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
#3 lore raw
Quoted 48 lines · David Gibson
Quoted 33 lines · On Thu, 22 Oct 2020 08:06:55 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 32 lines · Marcel Apfelbaum

From: Marcel Apfelbaum <[email protected]>

During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) the "Slot Control Register" has the "Power Indicator Control"
set to "Blinking" expressing a "power transition" mode.

Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.

Detect and refuse hotplug operations in such case.

---
 hw/pci/pcie.c | 7 +++++++
 1 file changed, 7 insertions(+)
hw/pci/pcie.c +1 -0
index 5b48bae0f6..2fe5c1473f 100644
@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
     uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;
     uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
+    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL);
 
     /* Check if hot-plug is disabled on the slot */
     if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) {
@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         return;
     }
 
Quoted 11 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 429

Quoted 8 lines
hw/pci/pcie.c +6 -0
+    if ((sltctl & PCI_EXP_SLTCTL_PIC) == PCI_EXP_SLTCTL_PWR_IND_BLINK) {
+        error_setg(errp, "Hot-plug failed: %s is in Power Transition",
+                   DEVICE(hotplug_pdev)->id);
+        return;
+    }
+
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
 }  

Probably the only way to handle for existing machine types. For new ones, can't we queue it in host memory somewhere?

Quoted 2 lines

I'm not actually convinced we can't do that even for existing machine types.

The difficulty would be in migrating the extra "reuested but defferred" state.

Quoted 9 lines

So I'm a bit hesitant to suggest going ahead with this without
looking a bit closer at whether we can implement a wait-for-ready in qemu, rather than forcing every user of qemu (human or machine) to do so.

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
#4 lore raw
Quoted 5 lines · On Thu, 22 Oct 2020 09:15:28 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

On Thu, Oct 22, 2020 at 11:56:32PM +1100, David Gibson wrote:
[...]
[...]
[...]

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Probably the only way to handle for existing machine types. For new ones, can't we queue it in host memory somewhere?

I'm not actually convinced we can't do that even for existing machine types.

The difficulty would be in migrating the extra "reuested but defferred" state.

Ah, true. Although we could block migration for the duration instead.

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
#5 lore raw
In reply to David Gibson

Hi David, Michael,

Quoted 38 lines · David Gibson
Quoted 4 lines · On Thu, 22 Oct 2020 08:06:55 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 3 lines · Marcel Apfelbaum

From: Marcel Apfelbaum <[email protected]>

During PCIe Root Port's transition from Power-Off to Power-ON (or

vice-versa)

Quoted 4 lines
Quoted 4 lines

the "Slot Control Register" has the "Power Indicator Control" set to "Blinking" expressing a "power transition" mode.

Any hotplug operation during the "power transition" mode is not

permitted

Quoted 9 lines
Quoted 9 lines

or at least not expected by the Guest OS leading to strange failures.

Detect and refuse hotplug operations in such case.

---
 hw/pci/pcie.c | 7 +++++++
 1 file changed, 7 insertions(+)
Quoted 5 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler

Quoted 5 lines
hw/pci/pcie.c +0 -0
index 5b48bae0f6..2fe5c1473f 100644
@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler

*hotplug_dev, DeviceState *dev,

Quoted 2 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler · line 411

Quoted 2 lines
hw/pci/pcie.c +0 -0
     PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
     uint8_t *exp_cap = hotplug_pdev->config +

hotplug_pdev->exp.exp_cap;

Quoted 6 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler

Quoted 6 lines
hw/pci/pcie.c +1 -0
     uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
+    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL);
 
     /* Check if hot-plug is disabled on the slot */
     if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) {
@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler

*hotplug_dev, DeviceState *dev,

Quoted 15 lines
Quoted 4 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler · line 422

Quoted 4 lines
hw/pci/pcie.c +1 -0
         return;
     }
 
+    if ((sltctl & PCI_EXP_SLTCTL_PIC) ==

PCI_EXP_SLTCTL_PWR_IND_BLINK) {

Quoted 9 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 429

Quoted 7 lines
hw/pci/pcie.c +5 -0
+        error_setg(errp, "Hot-plug failed: %s is in Power Transition",
+                   DEVICE(hotplug_pdev)->id);
+        return;
+    }
+
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
 }

Probably the only way to handle for existing machine types.

I agree

Quoted 3 lines
Quoted 1 line

For new ones, can't we queue it in host memory somewhere?

I am not sure I understand what will be the flow.
- The user asks for a hotplug operation.
- QEMU deferred operation.
After that the operation may still fail, how would the user know if the operation
succeeded or not?

Quoted 2 lines

I'm not actually convinced we can't do that even for existing machine types.

Is a Guest visible change, I don't think we can do it.

Quoted 5 lines

So I'm a bit hesitant to suggest going ahead with this without looking a bit closer at whether we can implement a wait-for-ready in qemu, rather than forcing every user of qemu (human or machine) to do so.

While I agree it is a pain from the usability point of view, hotplug operations
are allowed to fail. This is not more than a corner case, ensuring the right response (gracefully erroring out) may be enough.

Thanks,
Marcel

Quoted 6 lines

--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat

#6 lore raw
Quoted 65 lines · Marcel Apfelbaum

Hi David, Michael,

On Thu, Oct 22, 2020 at 3:56 PM David Gibson <[email protected]> wrote:

    On Thu, 22 Oct 2020 08:06:55 -0400
    "Michael S. Tsirkin" <[email protected]> wrote:

    > On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel Apfelbaum wrote:
    > > From: Marcel Apfelbaum <[email protected]>
    > >
    > > During PCIe Root Port's transition from Power-Off to Power-ON (or
    vice-versa)
    > > the "Slot Control Register" has the "Power Indicator Control"
    > > set to "Blinking" expressing a "power transition" mode.
    > >
    > > Any hotplug operation during the "power transition" mode is not
    permitted
    > > or at least not expected by the Guest OS leading to strange failures.
    > >
    > > Detect and refuse hotplug operations in such case.
    > >
    > > Signed-off-by: Marcel Apfelbaum <[email protected]>
    > > ---
    > >  hw/pci/pcie.c | 7 +++++++
    > >  1 file changed, 7 insertions(+)
    > >
    > > diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
    > > index 5b48bae0f6..2fe5c1473f 100644
    > > --- a/hw/pci/pcie.c
    > > +++ b/hw/pci/pcie.c
    > > @@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler
    *hotplug_dev, DeviceState *dev,
    > >      PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
    > >      uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->
    exp.exp_cap;
    > >      uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
    > > +    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL);
    > > 
    > >      /* Check if hot-plug is disabled on the slot */
    > >      if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) {
    > > @@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler
    *hotplug_dev, DeviceState *dev,
    > >          return;
    > >      }
    > > 
    > > +    if ((sltctl & PCI_EXP_SLTCTL_PIC) == PCI_EXP_SLTCTL_PWR_IND_BLINK)
    {
    > > +        error_setg(errp, "Hot-plug failed: %s is in Power Transition",
    > > +                   DEVICE(hotplug_pdev)->id);
    > > +        return;
    > > +    }
    > > +
    > >      pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
    > >  } 
    >
    > Probably the only way to handle for existing machine types.

I agree
Â

Quoted 1 line

For new ones, can't we queue it in host memory somewhere?

Quoted 6 lines

I am not sure I understand what will be the flow.
  - The user asks for a hotplug operation.
  -  QEMU deferred operation.
After that the operation may still fail, how would the user know if the operation
succeeded or not?

How can it fail? It's just a button press ...

Quoted 15 lines

Â

I'm not actually convinced we can't do that even for existing machine types.Â

Is a Guest visible change, I don't think we can do it. Â

So I'm a bit hesitant to suggest going ahead with this without looking a bit closer at whether we can implement a wait-for-ready in qemu, rather than forcing every user of qemu (human or machine) to do so.

Quoted 7 lines

While I agree it is a pain from the usability point of view, hotplug operations are allowed to fail. This is not more than a corner case, ensuring the right response (gracefully erroring out) may be enough.

Thanks,
Marcel

I don't think they ever failed in the past so management is unlikely to handle the failure by retrying ...

Quoted 7 lines

--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat

#7 lore raw
Quoted 79 lines · Michael S. Tsirkin
Quoted 10 lines · Marcel Apfelbaum

Hi David, Michael,

On Thu, Oct 22, 2020 at 3:56 PM David Gibson <[email protected]> wrote:

On Thu, 22 Oct 2020 08:06:55 -0400
"Michael S. Tsirkin" <[email protected]> wrote:

> On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel Apfelbaum wrote:
> > From: Marcel Apfelbaum <[email protected]>
> >

Quoted 8 lines

> > During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa)
> > the "Slot Control Register" has the "Power Indicator Control" > > set to "Blinking" expressing a "power transition" mode. > >
> > Any hotplug operation during the "power transition" mode is not permitted
> > or at least not expected by the Guest OS leading to strange

failures.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Detect and refuse hotplug operations in such case.

---
 hw/pci/pcie.c | 7 +++++++
 1 file changed, 7 insertions(+)
Quoted 15 lines
Quoted 5 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler

Quoted 5 lines
hw/pci/pcie.c +0 -0
index 5b48bae0f6..2fe5c1473f 100644
@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler
    *hotplug_dev, DeviceState *dev,
    > >      PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
    > >      uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->
    exp.exp_cap;
    > >      uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
    > > +    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL);
    > >
    > >      /* Check if hot-plug is disabled on the slot */
    > >      if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) {
    > > @@ -418,6 +419,12 @@ void

pcie_cap_slot_pre_plug_cb(HotplugHandler

Quoted 5 lines
    *hotplug_dev, DeviceState *dev,
    > >          return;
    > >      }
    > >
    > > +    if ((sltctl & PCI_EXP_SLTCTL_PIC) ==

PCI_EXP_SLTCTL_PWR_IND_BLINK)

Quoted 2 lines

{
> > + error_setg(errp, "Hot-plug failed: %s is in Power

Transition",

Quoted 5 lines
Quoted 5 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,

Quoted 5 lines
hw/pci/pcie.c +4 -0
+                   DEVICE(hotplug_pdev)->id);
+        return;
+    }
+
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev,

errp);

Quoted 12 lines
Quoted 3 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler · line 415

Quoted 1 line
hw/pci/pcie.c +0 -0
 }

Probably the only way to handle for existing machine types.

I agree

Quoted 1 line

For new ones, can't we queue it in host memory somewhere?

Quoted 6 lines

I am not sure I understand what will be the flow.
- The user asks for a hotplug operation.
- QEMU deferred operation.
After that the operation may still fail, how would the user know if the operation
succeeded or not?

Quoted 3 lines

How can it fail? It's just a button press ...

Currently we have "Hotplug unsupported."
With this change we have "Guest/System not ready"

Quoted 21 lines
Quoted 9 lines

I'm not actually convinced we can't do that even for existing machine types.

Is a Guest visible change, I don't think we can do it.

Quoted 7 lines

So I'm a bit hesitant to suggest going ahead with this without looking a bit closer at whether we can implement a wait-for-ready in qemu, rather than forcing every user of qemu (human or machine) to do so.

While I agree it is a pain from the usability point of view, hotplug

operations

Quoted 1 line

are allowed to fail. This is not more than a corner case, ensuring the

right

Quoted 2 lines

response (gracefully erroring out) may be enough.

Quoted 8 lines
Quoted 3 lines

Thanks,
Marcel

I don't think they ever failed in the past so management is unlikely to handle the failure by retrying ...

That would require some management handling, yes.
But even without a "retry", failing is better than strange OS behavior.

Trying a better alternative like deferring the operation for new machines would make sense, however is out of the scope of this patch that simply detects the error leaving us in a slightly better state than today.

Thanks,
Marcel

Quoted 10 lines
Quoted 8 lines

--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat

#8 lore raw
Quoted 83 lines · Marcel Apfelbaum

On Thu, Oct 22, 2020 at 5:01 PM Michael S. Tsirkin <[email protected]> wrote:

On Thu, Oct 22, 2020 at 04:55:10PM +0300, Marcel Apfelbaum wrote:
> Hi David, Michael,
>
> On Thu, Oct 22, 2020 at 3:56 PM David Gibson <[email protected]> wrote:
>
>     On Thu, 22 Oct 2020 08:06:55 -0400
>     "Michael S. Tsirkin" <[email protected]> wrote:
>
>     > On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel Apfelbaum wrote:
>     > > From: Marcel Apfelbaum <[email protected]>
>     > >
>     > > During PCIe Root Port's transition from Power-Off to Power-ON (or >     vice-versa)
>     > > the "Slot Control Register" has the "Power Indicator Control" >     > > set to "Blinking" expressing a "power transition" mode. >     > >
>     > > Any hotplug operation during the "power transition" mode is not >     permitted
>     > > or at least not expected by the Guest OS leading to strange failures.
>     > >
>     > > Detect and refuse hotplug operations in such case.
>     > >
>     > > Signed-off-by: Marcel Apfelbaum <[email protected]> >     > > ---
>     > >  hw/pci/pcie.c | 7 +++++++
>     > >  1 file changed, 7 insertions(+)
>     > >
>     > > diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
>     > > index 5b48bae0f6..2fe5c1473f 100644
>     > > --- a/hw/pci/pcie.c
>     > > +++ b/hw/pci/pcie.c
>     > > @@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler >     *hotplug_dev, DeviceState *dev,
>     > >      PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev); >     > >      uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev-> >     exp.exp_cap;
>     > >      uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP); >     > > +    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL); >     > >Â
>     > >      /* Check if hot-plug is disabled on the slot */ >     > >      if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) { >     > > @@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb (HotplugHandler
>     *hotplug_dev, DeviceState *dev,
>     > >          return;
>     > >      }
>     > >Â
>     > > +    if ((sltctl & PCI_EXP_SLTCTL_PIC) == PCI_EXP_SLTCTL_PWR_IND_BLINK)
>     {
>     > > +        error_setg(errp, "Hot-plug failed: %s is in Power Transition",
>     > > +                   DEVICE(hotplug_pdev)->id); >     > > +        return;
>     > > +    }
>     > > +
>     > >      pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
>     > >  }Â
>     >
>     > Probably the only way to handle for existing machine types. >
>
> I agree
> Â
>
>     > For new ones, can't we queue it in host memory somewhere? >
>
>
> I am not sure I understand what will be the flow.
>   - The user asks for a hotplug operation.
>   -  QEMU deferred operation.
> After that the operation may still fail, how would the user know if the > operation
> succeeded or not?

Quoted 6 lines

How can it fail? It's just a button press ...

Currently we have "Hotplug unsupported."
With this change we have "Guest/System not ready"

Hotplug unsupported is not an error that can trigger with a well behaved management such as libvirt.

Quoted 32 lines

Â

Quoted 9 lines

Â

     I'm not actually convinced we can't do that even for existing machine      types.Â

Is a Guest visible change, I don't think we can do it. Â

Quoted 7 lines

     So I'm a bit hesitant to suggest going ahead with this without      looking a bit closer at whether we can implement a wait-for-ready in      qemu, rather than forcing every user of qemu (human or machine) to do      so.

While I agree it is a pain from the usability point of view, hotplug

operations
> are allowed to fail. This is not more than a corner case, ensuring the right
> response (gracefully erroring out) may be enough.
>
> Thanks,
> Marcel
>

I don't think they ever failed in the past so management is unlikely to handle the failure by retrying ...

Quoted 5 lines

That would require some management handling, yes.
But even without a "retry", failing is better than strange OS behavior.

Trying a better alternative like deferring the operation for new machines would make sense, however is out of the scope of this patch

Expand the scope please. The scope should be "solve a problem xx" not "solve a problem xx by doing abc".

Quoted 5 lines

that simply
detects the error leaving us in a slightly better state than today.

Thanks,
Marcel

Not applying a patch is the only tool we maintainers have to influence people to solve the problem fully. That's why I'm not inclined to apply "slightly better" patches generally.

Quoted 11 lines
Quoted 9 lines

     --
     David Gibson <[email protected]>
     Principal Software Engineer, Virtualization, Red Hat

#9 lore raw
Quoted 147 lines · Michael S. Tsirkin
Quoted 3 lines · Marcel Apfelbaum

On Thu, Oct 22, 2020 at 5:01 PM Michael S. Tsirkin <[email protected]>

Quoted 5 lines · wrote:
    On Thu, Oct 22, 2020 at 04:55:10PM +0300, Marcel Apfelbaum wrote:
    > Hi David, Michael,
    >
    > On Thu, Oct 22, 2020 at 3:56 PM David Gibson <[email protected]>
Quoted 5 lines · wrote:
Quoted 5 lines
    On Thu, 22 Oct 2020 08:06:55 -0400
    "Michael S. Tsirkin" <[email protected]> wrote:

    > On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel Apfelbaum
Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

From: Marcel Apfelbaum <[email protected]>

During PCIe Root Port's transition from Power-Off to

Power-ON (or

Quoted 2 lines
Quoted 2 lines

vice-versa)
> > the "Slot Control Register" has the "Power Indicator

Control"

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

set to "Blinking" expressing a "power transition" mode.

Any hotplug operation during the "power transition" mode

is not

Quoted 7 lines
Quoted 2 lines

permitted
> > or at least not expected by the Guest OS leading to strange

failures.
> > >
> > > Detect and refuse hotplug operations in such case.
> > >
> > > Signed-off-by: Marcel Apfelbaum <

[email protected]>

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

hw/pci/pcie.c void

Quoted 9 lines
---
 hw/pci/pcie.c | 7 +++++++
 1 file changed, 7 insertions(+)
hw/pci/pcie.c +0 -0
index 5b48bae0f6..2fe5c1473f 100644
@@ -410,6 +410,7 @@ void

pcie_cap_slot_pre_plug_cb(HotplugHandler

Quoted 3 lines
Quoted 3 lines
    *hotplug_dev, DeviceState *dev,
    > >      PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
    > >      uint8_t *exp_cap = hotplug_pdev->config +

hotplug_pdev->

Quoted 2 lines
Quoted 2 lines

exp.exp_cap;
> > uint32_t sltcap = pci_get_word(exp_cap +

PCI_EXP_SLTCAP);

Quoted 1 line
Quoted 1 line
Quoted 1 line

hw/pci/pcie.c void · line 410

Quoted 1 line
hw/pci/pcie.c +1 -0
+    uint32_t sltctl = pci_get_word(exp_cap +

PCI_EXP_SLTCTL);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

hw/pci/pcie.c void · line 412

Quoted 3 lines
hw/pci/pcie.c +0 -0
     /* Check if hot-plug is disabled on the slot */
     if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC)

== 0) {

Quoted 16 lines
Quoted 1 line
Quoted 1 line

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb

Quoted 1 line
hw/pci/pcie.c +0 -0
@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb
    (HotplugHandler
    >     *hotplug_dev, DeviceState *dev,
    >     > >          return;
    >     > >      }
    >     > >
    >     > > +    if ((sltctl & PCI_EXP_SLTCTL_PIC) ==
    PCI_EXP_SLTCTL_PWR_IND_BLINK)
    >     {
    >     > > +        error_setg(errp, "Hot-plug failed: %s is in Power
    Transition",
    >     > > +                   DEVICE(hotplug_pdev)->id);
    >     > > +        return;
    >     > > +    }
    >     > > +
    >     > >      pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev),

dev,

Quoted 9 lines
    errp);
    >     > >  }
    >     >
    >     > Probably the only way to handle for existing machine types.
    >
    >
    > I agree
    >
    >
Quoted 8 lines
    >     > For new ones, can't we queue it in host memory somewhere?
    >
    >
    >
    > I am not sure I understand what will be the flow.
    >   - The user asks for a hotplug operation.
    >   -  QEMU deferred operation.
    > After that the operation may still fail, how would the user know

if the

Quoted 10 lines
Quoted 2 lines

operation
succeeded or not?

How can it fail? It's just a button press ...

Currently we have "Hotplug unsupported."
With this change we have "Guest/System not ready"

Hotplug unsupported is not an error that can trigger with a well behaved management such as libvirt.

Quoted 5 lines
Quoted 5 lines

I'm not actually convinced we can't do that even for existing

machine

Quoted 8 lines
Quoted 8 lines

types.

Is a Guest visible change, I don't think we can do it.

So I'm a bit hesitant to suggest going ahead with this without looking a bit closer at whether we can implement a

wait-for-ready in

Quoted 1 line
Quoted 1 line

qemu, rather than forcing every user of qemu (human or

machine) to do

Quoted 4 lines
Quoted 4 lines
    so.

While I agree it is a pain from the usability point of view,

hotplug

Quoted 2 lines

operations
> are allowed to fail. This is not more than a corner case, ensuring

the

Quoted 12 lines

right
> response (gracefully erroring out) may be enough. >
> Thanks,
> Marcel
>

I don't think they ever failed in the past so management is unlikely to handle the failure by retrying ...

Quoted 10 lines
Quoted 5 lines

That would require some management handling, yes.
But even without a "retry", failing is better than strange OS behavior.

Trying a better alternative like deferring the operation for new machines would make sense, however is out of the scope of this patch

Expand the scope please. The scope should be "solve a problem xx" not "solve a problem xx by doing abc".

The scope is detecting a hotplug error early instead passing to the Guest OS a hotplug operation that we know it will fail.

Quoted 8 lines
Quoted 5 lines

that simply
detects the error leaving us in a slightly better state than today.

Thanks,
Marcel

Not applying a patch is the only tool we maintainers have to influence people to solve the problem fully.

That's why I'm not inclined to apply

Quoted 3 lines

"slightly better" patches generally.

The patch is a proposal following some offline discussions on this matter. I personally see the value of it versus what we have today.

Thanks,
Marcel

Quoted 14 lines
Quoted 12 lines
Quoted 10 lines

--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat

#10 lore raw
Quoted 157 lines · Marcel Apfelbaum

On Thu, Oct 22, 2020 at 5:33 PM Michael S. Tsirkin <[email protected]> wrote:

On Thu, Oct 22, 2020 at 05:10:43PM +0300, Marcel Apfelbaum wrote:
>
>
> On Thu, Oct 22, 2020 at 5:01 PM Michael S. Tsirkin <[email protected]> wrote:
>
>     On Thu, Oct 22, 2020 at 04:55:10PM +0300, Marcel Apfelbaum wrote:
>     > Hi David, Michael,
>     >
>     > On Thu, Oct 22, 2020 at 3:56 PM David Gibson <[email protected]> wrote:
>     >
>     >     On Thu, 22 Oct 2020 08:06:55 -0400
>     >     "Michael S. Tsirkin" <[email protected]> wrote:
>     >
>     >     > On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel Apfelbaum wrote:
>     >     > > From: Marcel Apfelbaum <[email protected]>
>     >     > >
>     >     > > During PCIe Root Port's transition from Power-Off to Power-ON (or
>     >     vice-versa)
>     >     > > the "Slot Control Register" has the "Power Indicator Control"
>     >     > > set to "Blinking" expressing a "power transition" mode. >     >     > >
>     >     > > Any hotplug operation during the "power transition" mode is not
>     >     permitted
>     >     > > or at least not expected by the Guest OS leading to strange >     failures.
>     >     > >
>     >     > > Detect and refuse hotplug operations in such case. >     >     > >
>     >     > > Signed-off-by: Marcel Apfelbaum <[email protected] >
>     >     > > ---
>     >     > >  hw/pci/pcie.c | 7 +++++++
>     >     > >  1 file changed, 7 insertions(+)
>     >     > >
>     >     > > diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
>     >     > > index 5b48bae0f6..2fe5c1473f 100644
>     >     > > --- a/hw/pci/pcie.c
>     >     > > +++ b/hw/pci/pcie.c
>     >     > > @@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb (HotplugHandler
>     >     *hotplug_dev, DeviceState *dev,
>     >     > >      PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev); >     >     > >      uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->
>     >     exp.exp_cap;
>     >     > >      uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
>     >     > > +    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL);
>     >     > >Â
>     >     > >      /* Check if hot-plug is disabled on the slot */ >     >     > >      if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) = = 0) {
>     >     > > @@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb >     (HotplugHandler
>     >     *hotplug_dev, DeviceState *dev,
>     >     > >          return;
>     >     > >      }
>     >     > >Â
>     >     > > +    if ((sltctl & PCI_EXP_SLTCTL_PIC) ==
>     PCI_EXP_SLTCTL_PWR_IND_BLINK)
>     >     {
>     >     > > +        error_setg(errp, "Hot-plug failed: %s is in Power >     Transition",
>     >     > > +                   DEVICE(hotplug_pdev)->id); >     >     > > +        return;
>     >     > > +    }
>     >     > > +
>     >     > >      pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev,
>     errp);
>     >     > >  }Â
>     >     >
>     >     > Probably the only way to handle for existing machine types. >     >
>     >
>     > I agree
>     > Â
>     >
>     >     > For new ones, can't we queue it in host memory somewhere? >     >
>     >
>     >
>     > I am not sure I understand what will be the flow.
>     >   - The user asks for a hotplug operation.
>     >   -  QEMU deferred operation.
>     > After that the operation may still fail, how would the user know if the
>     > operation
>     > succeeded or not?
>
>
>     How can it fail? It's just a button press ...
>
>
>
> Currently we have "Hotplug unsupported."
> With this change we have "Guest/System not ready"

Hotplug unsupported is not an error that can trigger with a well behaved management such as libvirt.

> Â
>
>     > Â
>     >
>     >     I'm not actually convinced we can't do that even for existing machine
>     >     types.Â
>     >
>     >
>     > Is a Guest visible change, I don't think we can do it. >     > Â
>     >
>     >     So I'm a bit hesitant to suggest going ahead with this without >     >     looking a bit closer at whether we can implement a wait-for-ready in
>     >     qemu, rather than forcing every user of qemu (human or machine) to do
>     >     so.
>     >
>     >
>     > While I agree it is a pain from the usability point of view, hotplug
>     operations
>     > are allowed to fail. This is not more than a corner case, ensuring the
>     right
>     > response (gracefully erroring out) may be enough.
>     >
>     > Thanks,
>     > Marcel
>     >
>
>
>     I don't think they ever failed in the past so management is unlikely >     to handle the failure by retrying ...
>
>
> That would require some management handling, yes.
> But even without a "retry", failing is better than strange OS behavior. >
> Trying a better alternative like deferring the operation for new machines > would make sense, however is out of the scope of this patch

Quoted 8 lines

Expand the scope please. The scope should be "solve a problem xx" not "solve a problem xx by doing abc".

The scope is detecting a hotplug error early instead passing to the Guest OS a hotplug operation that we know it will fail.

Right. After detecting just failing unconditionally it a bit too simplistic IMHO.

Quoted 35 lines
Quoted 6 lines

that simply
detects the error leaving us in a slightly better state than today.

Thanks,
Marcel

Not applying a patch is the only tool we maintainers have to influence people to solve the problem fully.Â

That's why I'm not inclined to apply
"slightly better" patches generally.

The patch is a proposal following some offline discussions on this matter. I personally see the value of it versus what we have today.

Thanks,
Marcel

Quoted 11 lines

     >
     >
     >
     >     --
     >     David Gibson <[email protected]>
     >     Principal Software Engineer, Virtualization, Red Hat      >

#11 lore raw
Quoted 5 lines · On Thu, 22 Oct 2020 11:01:04 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
[...]

Right. After detecting just failing unconditionally it a bit too simplistic IMHO.

There's also another factor here, which I thought I'd mentioned already, but looks like I didn't: I think we're still missing some details in what's going on.

The premise for this patch is that plugging while the indicator is in transition state is allowed to fail in any way on the guest side. I don't think that's a reasonable interpretation, because it's unworkable for physical hotplug. If the indicator starts blinking while you're in the middle of shoving a card in, you'd be in trouble.

So, what I'm assuming here is that while "don't plug while blinking" is the instruction for the operator to obey as best they can, on the guest side the rule has to be "start blinking, wait a while and by the time you leave blinking state again, you can be confident any plugs or unplugs have completed". Obviously still racy in the strict computer science sense, but about the best you can do with slow humans in the mix.

So, qemu should of course endeavour to follow that rule as though it was a human operator on a physical machine and not plug when the indicator is blinking. *But* the qemu plug will in practice be fast enough that if we're hitting real problems here, it suggests the guest is still doing something wrong.

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
#12 lore raw

Hi David,

Quoted 27 lines · David Gibson
Quoted 5 lines · On Thu, 22 Oct 2020 11:01:04 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
[...]

Right. After detecting just failing unconditionally it a bit too simplistic IMHO.

There's also another factor here, which I thought I'd mentioned already, but looks like I didn't: I think we're still missing some details in what's going on.

The premise for this patch is that plugging while the indicator is in transition state is allowed to fail in any way on the guest side. I don't think that's a reasonable interpretation, because it's unworkable for physical hotplug. If the indicator starts blinking while you're in the middle of shoving a card in, you'd be in trouble.

So, what I'm assuming here is that while "don't plug while blinking" is the instruction for the operator to obey as best they can, on the guest side the rule has to be "start blinking, wait a while and by the time you leave blinking state again, you can be confident any plugs or unplugs have completed". Obviously still racy in the strict computer science sense, but about the best you can do with slow humans in the mix.

Quoted 6 lines

So, qemu should of course endeavour to follow that rule as though it was a human operator on a physical machine and not plug when the indicator is blinking. *But* the qemu plug will in practice be fast enough that if we're hitting real problems here, it suggests the guest is still doing something wrong.

I personally think there is a little bit of over-engineering here. Let's start with the spec:

Power Indicator Blinking
A blinking Power Indicator indicates that the slot is powering up or
powering down and that
insertion or removal of the adapter is not permitted.

What exactly is an interpretation here?
As you stated, the races are theoretical, the whole point of the indicator is to let the operator know he can't plug the device just yet.

I understand it would be more user friendly if the QEMU would wait internally for the
blinking to end, but the whole point of the indicator is to let the operator (human or machine)
know they can't plug the device at a specific time.
Should QEMU take the responsibility of the operator? Is it even correct?

Even if we would want such a feature, how is it related to this patch? The patch simply refuses to start a hotplug operation when it knows it will not succeed.

Another way that would make sense to me would be is a new QEMU interface other than
"add_device", let's say "adding_device_allowed", that would return true if the hotplug is allowed
at this point of time. (I am aware of the theoretical races)

The above will at least mimic the mechanics of the pyhs world. The operator looks at the indicator,
the management software checks if adding the device is allowed. Since it is a corner case I would prefer the device_add to fail rather than introducing a new interface,
but that's just me.

Thanks,
Marcel

Quoted 4 lines

--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat

#13 lore raw
Quoted 57 lines · Marcel Apfelbaum

Hi David,

On Fri, Oct 23, 2020 at 6:49 AM David Gibson <[email protected]> wrote:

On Thu, 22 Oct 2020 11:01:04 -0400
"Michael S. Tsirkin" <[email protected]> wrote:

> On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
>  [...]Â
>
> Right. After detecting just failing unconditionally it a bit too > simplistic IMHO.

There's also another factor here, which I thought I'd mentioned already, but looks like I didn't: I think we're still missing some details in what's going on.

The premise for this patch is that plugging while the indicator is in transition state is allowed to fail in any way on the guest side.  I don't think that's a reasonable interpretation, because it's unworkable for physical hotplug.  If the indicator starts blinking while you're in the middle of shoving a card in, you'd be in trouble.

So, what I'm assuming here is that while "don't plug while blinking" is the instruction for the operator to obey as best they can, on the guest side the rule has to be "start blinking, wait a while and by the time you leave blinking state again, you can be confident any plugs or unplugs have completed".  Obviously still racy in the strict computer science sense, but about the best you can do with slow humans in the mix.

So, qemu should of course endeavour to follow that rule as though it was a human operator on a physical machine and not plug when the indicator is blinking.  *But* the qemu plug will in practice be fast enough that if we're hitting real problems here, it suggests the guest is still doing something wrong.

I personally think there is a little bit of over-engineering here. Let's start with the spec:

    Power Indicator Blinking
    A blinking Power Indicator indicates that the slot is powering up or powering down and that
    insertion or removal of the adapter is not permitted.

What exactly is an interpretation here?
As you stated, the races are theoretical, the whole point of the indicator is to let the operator know he can't plug the device just yet.

I understand it would be more user friendly if the QEMU would wait internally for the
blinking to end, but the whole point of the indicator is to let the operator (human or machine)
know they can't plug the device at a specific time.
Should QEMU take the responsibility of the operator? Is it even correct?

Quoted 9 lines

Even if we would want such a feature, how is it related to this patch? The patch simply refuses to start a hotplug operation when it knows it will not succeed.Â
Â
Another way that would make sense to me would be  is a new QEMU interface other than
"add_device", let's say "adding_device_allowed", that would return true if the hotplug is allowed
at this point of time. (I am aware of the theoretical races)Â

Rather than adding_device_allowed, something like "query slot" might be helpful for debugging. That would help user figure out e.g. why isn't device visible without any races.

Quoted 10 lines

The above will at least mimic the mechanics of the pyhs world.  The operator looks at the indicator,
the management software checks if adding the device is allowed. Since it is a corner case I would prefer the device_add to fail rather than introducing a new interface,
but that's just me.

Thanks,
Marcel

I think we want QEMU management interface to be reasonably abstract and agnostic if possible. Pushing knowledge of hardware detail to management will just lead to pain IMHO.
We supported device_add which practically never fails for years, at this point it's easier to keep supporting it than change all users ...

Quoted 5 lines

--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat

#14 lore raw
Quoted 68 lines · On Fri, 23 Oct 2020 11:54:40 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 57 lines · Marcel Apfelbaum

Hi David,

On Fri, Oct 23, 2020 at 6:49 AM David Gibson <[email protected]> wrote:

On Thu, 22 Oct 2020 11:01:04 -0400
"Michael S. Tsirkin" <[email protected]> wrote:

> On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
>  [...]ÂÂ
>
> Right. After detecting just failing unconditionally it a bit too > simplistic IMHO.

There's also another factor here, which I thought I'd mentioned already, but looks like I didn't: I think we're still missing some details in what's going on.

The premise for this patch is that plugging while the indicator is in transition state is allowed to fail in any way on the guest side.  I don't think that's a reasonable interpretation, because it's unworkable for physical hotplug.  If the indicator starts blinking while you're in the middle of shoving a card in, you'd be in trouble.

So, what I'm assuming here is that while "don't plug while blinking" is the instruction for the operator to obey as best they can, on the guest side the rule has to be "start blinking, wait a while and by the time you leave blinking state again, you can be confident any plugs or unplugs have completed".  Obviously still racy in the strict computer science sense, but about the best you can do with slow humans in the mix.

So, qemu should of course endeavour to follow that rule as though it was a human operator on a physical machine and not plug when the indicator is blinking.  *But* the qemu plug will in practice be fast enough that if we're hitting real problems here, it suggests the guest is still doing something wrong.

I personally think there is a little bit of over-engineering here. Let's start with the spec:

    Power Indicator Blinking
    A blinking Power Indicator indicates that the slot is powering up or powering down and that
    insertion or removal of the adapter is not permitted.

What exactly is an interpretation here?
As you stated, the races are theoretical, the whole point of the indicator is to let the operator know he can't plug the device just yet.

I understand it would be more user friendly if the QEMU would wait internally for the
blinking to end, but the whole point of the indicator is to let the operator (human or machine)
know they can't plug the device at a specific time.
Should QEMU take the responsibility of the operator? Is it even correct?

Quoted 9 lines

Even if we would want such a feature, how is it related to this patch? The patch simply refuses to start a hotplug operation when it knows it will not succeed.ÂÂ
ÂÂ
Another way that would make sense to me would be  is a new QEMU interface other than
"add_device", let's say "adding_device_allowed", that would return true if the hotplug is allowed
at this point of time. (I am aware of the theoretical races)ÂÂ

Quoted 3 lines

Rather than adding_device_allowed, something like "query slot" might be helpful for debugging. That would help user figure out e.g. why isn't device visible without any races.

Would be new command useful tough? What we end up is broken guest (if I read commit message right) and a user who has no idea if device_add was successful or not.
So what user should do in this case
- wait till it explodes?
- can user remove it or it would be stuck there forever?
- poll slot before hotplug, manually?

(if this is the case then failing device_add cleanly doesn't sound bad,
it looks similar to another error we have "/* Check if hot-plug is disabled on the slot */" in pcie_cap_slot_pre_plug_cb)

CCing libvirt, as it concerns not only QEMU.

Quoted 8 lines
Quoted 8 lines

The above will at least mimic the mechanics of the pyhs world.  The operator looks at the indicator,
the management software checks if adding the device is allowed.
Since it is a corner case I would prefer the device_add to fail rather than introducing a new interface,
but that's just me.

Quoted 8 lines
Quoted 3 lines

Thanks,
Marcel

I think we want QEMU management interface to be reasonably abstract and agnostic if possible. Pushing knowledge of hardware detail to management will just lead to pain IMHO.
We supported device_add which practically never fails for years,

For CPUs and RAM, device_add can fail so maybe management is also prepared to handle errors on PCI hotplug path.

Quoted 11 lines

at this point it's easier to keep supporting it than change all users ...

Quoted 5 lines

--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat

#15 lore raw
Quoted 29 lines · On Fri, 23 Oct 2020 19:27:55 +0200 Igor Mammedov <[email protected]> wrote:

On Fri, 23 Oct 2020 11:54:40 -0400
"Michael S. Tsirkin" <[email protected]> wrote:

[...]
[...]
[...]
[...]

Quoted 4 lines

Rather than adding_device_allowed, something like "query slot" might be helpful for debugging. That would help user figure out e.g. why isn't device visible without any races.

Would be new command useful tough? What we end up is broken guest (if I read commit message right) and a user who has no idea if device_add was successful or not.
So what user should do in this case
- wait till it explodes?
- can user remove it or it would be stuck there forever?
- poll slot before hotplug, manually?

(if this is the case then failing device_add cleanly doesn't sound bad,
it looks similar to another error we have "/* Check if hot-plug is disabled on the slot */" in pcie_cap_slot_pre_plug_cb)

CCing libvirt, as it concerns not only QEMU.

[...]
[...]

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

I think we want QEMU management interface to be reasonably abstract and agnostic if possible. Pushing knowledge of hardware detail to management will just lead to pain IMHO.
We supported device_add which practically never fails for years,

For CPUs and RAM, device_add can fail so maybe management is also prepared to handle errors on PCI hotplug path.

There can be unarguable reasons for PCI hotplug to fail as well (attempting to plug to a bus that can't support it for one). The difference here is that it's a failure that we expect to be transitory.

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
#16 lore raw
In reply to Igor Mammedov
Quoted 83 lines · Igor Mammedov
Quoted 68 lines · On Fri, 23 Oct 2020 11:54:40 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 57 lines · Marcel Apfelbaum

Hi David,

On Fri, Oct 23, 2020 at 6:49 AM David Gibson <[email protected]> wrote:

On Thu, 22 Oct 2020 11:01:04 -0400
"Michael S. Tsirkin" <[email protected]> wrote:

> On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
>  [...]ÂÂ
>
> Right. After detecting just failing unconditionally it a bit too > simplistic IMHO.

There's also another factor here, which I thought I'd mentioned already, but looks like I didn't: I think we're still missing some details in what's going on.

The premise for this patch is that plugging while the indicator is in transition state is allowed to fail in any way on the guest side.  I don't think that's a reasonable interpretation, because it's unworkable for physical hotplug.  If the indicator starts blinking while you're in the middle of shoving a card in, you'd be in trouble.

So, what I'm assuming here is that while "don't plug while blinking" is the instruction for the operator to obey as best they can, on the guest side the rule has to be "start blinking, wait a while and by the time you leave blinking state again, you can be confident any plugs or unplugs have completed".  Obviously still racy in the strict computer science sense, but about the best you can do with slow humans in the mix.

So, qemu should of course endeavour to follow that rule as though it was a human operator on a physical machine and not plug when the indicator is blinking.  *But* the qemu plug will in practice be fast enough that if we're hitting real problems here, it suggests the guest is still doing something wrong.

I personally think there is a little bit of over-engineering here. Let's start with the spec:

    Power Indicator Blinking
    A blinking Power Indicator indicates that the slot is powering up or powering down and that
    insertion or removal of the adapter is not permitted.

What exactly is an interpretation here?
As you stated, the races are theoretical, the whole point of the indicator is to let the operator know he can't plug the device just yet.

I understand it would be more user friendly if the QEMU would wait internally for the
blinking to end, but the whole point of the indicator is to let the operator (human or machine)
know they can't plug the device at a specific time.
Should QEMU take the responsibility of the operator? Is it even correct?

Quoted 9 lines

Even if we would want such a feature, how is it related to this patch? The patch simply refuses to start a hotplug operation when it knows it will not succeed.ÂÂ
ÂÂ
Another way that would make sense to me would be  is a new QEMU interface other than
"add_device", let's say "adding_device_allowed", that would return true if the hotplug is allowed
at this point of time. (I am aware of the theoretical races)ÂÂ

Quoted 3 lines

Rather than adding_device_allowed, something like "query slot" might be helpful for debugging. That would help user figure out e.g. why isn't device visible without any races.

Would be new command useful tough? What we end up is broken guest (if I read commit message right) and a user who has no idea if device_add was successful or not.
So what user should do in this case
- wait till it explodes?
- can user remove it or it would be stuck there forever?
- poll slot before hotplug, manually?

Quoted 5 lines

(if this is the case then failing device_add cleanly doesn't sound bad,
it looks similar to another error we have "/* Check if hot-plug is disabled on the slot */" in pcie_cap_slot_pre_plug_cb)

CCing libvirt, as it concerns not only QEMU.

The only reason a separate command might make sense is if libvirt would want to provide a specific error to the user/upper management layer that the operation failed due to a transient failure (so that it can be retried later).

We don't really want to go to a policy decision of how long to wait in such case, so unless qemu waits libvirt will plainly want to report an error.

That said IMO 'device_add' should still fail if it's certain that the device won't be plugged in. This will fix any client which is currently in use. Adding a separate command is worth only for pre-checking for saner error handling.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

The above will at least mimic the mechanics of the pyhs world.  The operator looks at the indicator,
the management software checks if adding the device is allowed.
Since it is a corner case I would prefer the device_add to fail rather than introducing a new interface,
but that's just me.

Thanks,
Marcel

Quoted 7 lines
Quoted 4 lines

I think we want QEMU management interface to be reasonably abstract and agnostic if possible. Pushing knowledge of hardware detail to management will just lead to pain IMHO.
We supported device_add which practically never fails for years,

For CPUs and RAM, device_add can fail so maybe management is also prepared to handle errors on PCI hotplug path.

While it was me who implemented device_add for cpu/memory I don't remmeber any more whether we are really prepared for it.

We certainly want to do it if there's a possibility to do it.

#17 lore raw
In reply to Marcel Apfelbaum
Quoted 38 lines · On Fri, 23 Oct 2020 09:47:14 +0300 Marcel Apfelbaum…

Hi David,

Quoted 27 lines · David Gibson
Quoted 5 lines · On Thu, 22 Oct 2020 11:01:04 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
[...]

Right. After detecting just failing unconditionally it a bit too simplistic IMHO.

There's also another factor here, which I thought I'd mentioned already, but looks like I didn't: I think we're still missing some details in what's going on.

The premise for this patch is that plugging while the indicator is in transition state is allowed to fail in any way on the guest side. I don't think that's a reasonable interpretation, because it's unworkable for physical hotplug. If the indicator starts blinking while you're in the middle of shoving a card in, you'd be in trouble.

So, what I'm assuming here is that while "don't plug while blinking" is the instruction for the operator to obey as best they can, on the guest side the rule has to be "start blinking, wait a while and by the time you leave blinking state again, you can be confident any plugs or unplugs have completed". Obviously still racy in the strict computer science sense, but about the best you can do with slow humans in the mix.

Quoted 6 lines

So, qemu should of course endeavour to follow that rule as though it was a human operator on a physical machine and not plug when the indicator is blinking. *But* the qemu plug will in practice be fast enough that if we're hitting real problems here, it suggests the guest is still doing something wrong.

Quoted 7 lines

I personally think there is a little bit of over-engineering here. Let's start with the spec:

Power Indicator Blinking
A blinking Power Indicator indicates that the slot is powering up or
powering down and that
insertion or removal of the adapter is not permitted.

Right, but the weird bit here is that IIUC, the kernel during general probe is switching the indicator from off -> blinking -> off without it ever going to "on" state. And it seems to do the power on and presence check with the indicator still in blinking state. This is different from the normal sequence on a hotplug:
press button
indicator goes to blinking
...wait...
indicator goes to full on
slot powers on
presence detect

Or am I missing something?

Quoted 10 lines

What exactly is an interpretation here?
As you stated, the races are theoretical, the whole point of the indicator is to let the operator know he can't plug the device just yet.

I understand it would be more user friendly if the QEMU would wait internally for the
blinking to end, but the whole point of the indicator is to let the operator (human or machine)
know they can't plug the device at a specific time.
Should QEMU take the responsibility of the operator? Is it even correct?

I don't think there's an unambiguous correct answer here. But I think it's reasonable to interpret a general "device_add" as "instruct the virtual operator to plug in the card ASAP" as easily as "shove the virtual card in right now". device_add already covers a bunch of different pluggable interfaces, so I don't think precisely aligning to pci-e semantics is really a priority.

Quoted 3 lines

Even if we would want such a feature, how is it related to this patch? The patch simply refuses to start a hotplug operation when it knows it will not succeed.

I don't think I was clear. There are two separate and unrelated things I'm bringing up here. The first is that having the device_add fail for transitory reasons that the management layer doesn't really care about is really bad UX.

But the second point I'm raising here is that "don't plug when blinking" is not, strictly speaking an implementable strategy for a human operator. That makes me conclude that the idea here is plugs started at basically the same time as the blinking starts (which could be a little before or a little after, humans being fuzzy) should actually be acceptable, even if they finish after the indicator is blinking. It's not clear to me that the kernel's current behaviour actually permits that.

Quoted 6 lines

Another way that would make sense to me would be is a new QEMU interface other than
"add_device", let's say "adding_device_allowed", that would return true if the hotplug is allowed
at this point of time. (I am aware of the theoretical races)

Quoted 6 lines

The above will at least mimic the mechanics of the pyhs world. The operator looks at the indicator,
the management software checks if adding the device is allowed. Since it is a corner case I would prefer the device_add to fail rather than introducing a new interface,
but that's just me.

Oh, no, that's not what I'm suggesting. Adding an "is allowed" command is even worse. It makes the management's task *more* complex, which making any possible race here even wider.

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
#18 lore raw
In reply to Michael S. Tsirkin

Hi Michael,

Quoted 186 lines · Michael S. Tsirkin
Quoted 3 lines · Marcel Apfelbaum

On Thu, Oct 22, 2020 at 5:33 PM Michael S. Tsirkin <[email protected]>

Quoted 9 lines · wrote:
    On Thu, Oct 22, 2020 at 05:10:43PM +0300, Marcel Apfelbaum wrote:
    >
    >
    > On Thu, Oct 22, 2020 at 5:01 PM Michael S. Tsirkin <[email protected]

    wrote:
    >
    >     On Thu, Oct 22, 2020 at 04:55:10PM +0300, Marcel Apfelbaum
Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines

Hi David, Michael,

On Thu, Oct 22, 2020 at 3:56 PM David Gibson <

[email protected]>

Quoted 6 lines
    wrote:
    >     >
    >     >     On Thu, 22 Oct 2020 08:06:55 -0400
    >     >     "Michael S. Tsirkin" <[email protected]> wrote:
    >     >
    >     >     > On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel

Apfelbaum

Quoted 9 lines

wrote:
> > > > From: Marcel Apfelbaum <[email protected]>
> > > >
> > > > During PCIe Root Port's transition from Power-Off to
Power-ON (or
> > vice-versa)
> > > > the "Slot Control Register" has the "Power Indicator
Control"
> > > > set to "Blinking" expressing a "power transition"

mode.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Any hotplug operation during the "power transition"

mode is

Quoted 3 lines

not
> > permitted
> > > > or at least not expected by the Guest OS leading to

strange

Quoted 5 lines
Quoted 5 lines

failures.
> > >
> > > Detect and refuse hotplug operations in such case.
> > >
> > > Signed-off-by: Marcel Apfelbaum <

[email protected]

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
---
 hw/pci/pcie.c | 7 +++++++
 1 file changed, 7 insertions(+)
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb

Quoted 5 lines
hw/pci/pcie.c +0 -0
index 5b48bae0f6..2fe5c1473f 100644
@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb

(HotplugHandler
> > *hotplug_dev, DeviceState *dev,
> > > > PCIDevice *hotplug_pdev =

PCI_DEVICE(hotplug_dev);

Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb · line 410

Quoted 1 line
hw/pci/pcie.c +0 -0
     uint8_t *exp_cap = hotplug_pdev->config +
    hotplug_pdev->
    >     >     exp.exp_cap;
    >     >     > >      uint32_t sltcap = pci_get_word(exp_cap +
    PCI_EXP_SLTCAP);
    >     >     > > +    uint32_t sltctl = pci_get_word(exp_cap +
    PCI_EXP_SLTCTL);
    >     >     > >
    >     >     > >      /* Check if hot-plug is disabled on the slot */
    >     >     > >      if (dev->hotplugged && (sltcap &

PCI_EXP_SLTCAP_HPC) =

Quoted 11 lines
    = 0) {
    >     >     > > @@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb
    >     (HotplugHandler
    >     >     *hotplug_dev, DeviceState *dev,
    >     >     > >          return;
    >     >     > >      }
    >     >     > >
    >     >     > > +    if ((sltctl & PCI_EXP_SLTCTL_PIC) ==
    >     PCI_EXP_SLTCTL_PWR_IND_BLINK)
    >     >     {
    >     >     > > +        error_setg(errp, "Hot-plug failed: %s is in

Power

Quoted 6 lines
Quoted 6 lines
    Transition",
    >     > > +                   DEVICE(hotplug_pdev)->id);
    >     > > +        return;
    >     > > +    }
    >     > > +
    >     > >

pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev),

Quoted 5 lines

dev,
> errp);
> > > > }
> > >
> > > Probably the only way to handle for existing machine

types.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I agree

Quoted 1 line

For new ones, can't we queue it in host memory

somewhere?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I am not sure I understand what will be the flow.
- The user asks for a hotplug operation.
- QEMU deferred operation.
After that the operation may still fail, how would the user

know if

Quoted 14 lines
    the
    >     > operation
    >     > succeeded or not?
    >
    >
    >     How can it fail? It's just a button press ...
    >
    >
    >
    > Currently we have "Hotplug unsupported."
    > With this change we have "Guest/System not ready"


    Hotplug unsupported is not an error that can trigger with
Quoted 8 lines
    a well behaved management such as libvirt.


    >
    >
    >     >
    >     >
    >     >     I'm not actually convinced we can't do that even for

existing

Quoted 8 lines
    machine
    >     >     types.
    >     >
    >     >
    >     > Is a Guest visible change, I don't think we can do it.
    >     >
    >     >
    >     >     So I'm a bit hesitant to suggest going ahead with this

without

Quoted 3 lines
Quoted 1 line
Quoted 1 line

looking a bit closer at whether we can implement a

wait-for-ready in
> > qemu, rather than forcing every user of qemu (human or

machine)

Quoted 8 lines

to do
> > so.
> >
> >
> > While I agree it is a pain from the usability point of view,
hotplug
> operations
> > are allowed to fail. This is not more than a corner case,

ensuring

Quoted 10 lines

the
> right
> > response (gracefully erroring out) may be enough.
> >
> > Thanks,
> > Marcel
> >
>
>
> I don't think they ever failed in the past so management is

unlikely

Quoted 5 lines
Quoted 5 lines

to handle the failure by retrying ...

That would require some management handling, yes. But even without a "retry", failing is better than strange OS

behavior.

Quoted 2 lines
Quoted 2 lines

Trying a better alternative like deferring the operation for new

machines

Quoted 7 lines
Quoted 1 line

would make sense, however is out of the scope of this patch

Expand the scope please. The scope should be "solve a problem xx" not "solve a problem xx by doing abc".

Quoted 8 lines
Quoted 3 lines

The scope is detecting a hotplug error early instead passing to the Guest OS a hotplug operation that we know it will fail.

Right. After detecting just failing unconditionally it a bit too simplistic IMHO.

Simplistic does not mean wrong or incorrect.
I fail to see why it is not enough.

What QEMU can do better? Wait an unbounded time for the blinking to finish? What if we have a buggy guest with a kernel stuck in blinking? Is QEMU's responsibility to emulate the operator itself? Because the operator
is the one who is supposed to wait.

Thanks,
Marcel

[...]

#19 lore raw
Quoted 106 lines · On Fri, 23 Oct 2020 09:26:48 +0300 Marcel Apfelbaum…

Hi Michael,

On Thu, Oct 22, 2020 at 6:01 PM Michael S. Tsirkin <[email protected]> wrote:

[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
Simplistic does not mean wrong or incorrect.
I fail to see why it is not enough.

Quoted 1 line

What QEMU can do better? Wait an unbounded time for the blinking to finish?

It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.

Quoted 10 lines

What if we have a buggy guest with a kernel stuck in blinking? Is QEMU's responsibility to emulate the operator itself? Because the operator
is the one who is supposed to wait.

Thanks,
Marcel

[...]

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
#20 lore raw
Quoted 109 lines · David Gibson
Quoted 106 lines · On Fri, 23 Oct 2020 09:26:48 +0300 Marcel Apfelbaum…

Hi Michael,

On Thu, Oct 22, 2020 at 6:01 PM Michael S. Tsirkin <[email protected]> wrote:

[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
Simplistic does not mean wrong or incorrect.
I fail to see why it is not enough.

Quoted 4 lines
Quoted 1 line

What QEMU can do better? Wait an unbounded time for the blinking to finish?

It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.

If it's helpful, I'd add a query to check state
so management can figure out why doesn't guest see device yet. But otherwise just buffer the request until such time as we can deliver it to guest ...

Quoted 15 lines
Quoted 10 lines

What if we have a buggy guest with a kernel stuck in blinking? Is QEMU's responsibility to emulate the operator itself? Because the operator
is the one who is supposed to wait.

Thanks,
Marcel

[...]

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
#21 lore raw
Quoted 9 lines · On Tue, 27 Oct 2020 07:26:44 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 8 lines · David Gibson
Quoted 5 lines · On Fri, 23 Oct 2020 09:26:48 +0300 Marcel Apfelbaum…

Hi Michael,

On Thu, Oct 22, 2020 at 6:01 PM Michael S. Tsirkin <[email protected]> wrote:

[...]

[...]

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Simplistic does not mean wrong or incorrect.
I fail to see why it is not enough.

What QEMU can do better? Wait an unbounded time for the blinking to finish?

It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.

racy, timeout is bound to break once it's in overcommited env.

Quoted 2 lines

If it's helpful, I'd add a query to check state
so management can figure out why doesn't guest see device yet.

that means mgmt would have to poll it and forward it to user somehow.

Quoted 2 lines

But otherwise just buffer the request until such time as we can deliver it to guest ...

I have more questions wrt the suggestion/workflow:
* at what place would you suggest buffering it?
* what would be the request in this case, i.e. create PCI device anyways and try to signal hotplug event later?
* what would baremethal do in such case?
* what to do in case guest is never ready, what user should do in such case?
* can be such device be removed?

not sure that all of this is worth of the effort and added complexity.

alternatively:
maybe ports can send QMP events about it's state changes, which end user would be able to see + error like in this patch.

On top of it, mgmt could build a better UIx, like retry/notify logic if that's what user really wishes for and configures (it would be up to user to define behaviour).

Quoted 18 lines
Quoted 15 lines
Quoted 10 lines

What if we have a buggy guest with a kernel stuck in blinking? Is QEMU's responsibility to emulate the operator itself? Because the operator
is the one who is supposed to wait.

Thanks,
Marcel

[...]

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat  
#22 lore raw
Quoted 24 lines · Igor Mammedov
Quoted 9 lines · On Tue, 27 Oct 2020 07:26:44 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 8 lines · David Gibson
Quoted 5 lines · On Fri, 23 Oct 2020 09:26:48 +0300 Marcel Apfelbaum…

Hi Michael,

On Thu, Oct 22, 2020 at 6:01 PM Michael S. Tsirkin <[email protected]> wrote:

[...]

[...]

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Simplistic does not mean wrong or incorrect.
I fail to see why it is not enough.

What QEMU can do better? Wait an unbounded time for the blinking to finish?

It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.

racy, timeout is bound to break once it's in overcommited env.

Quoted 2 lines

If it's helpful, I'd add a query to check state
so management can figure out why doesn't guest see device yet.

Quoted 7 lines

that means mgmt would have to poll it and forward it to user somehow.

Quoted 2 lines

But otherwise just buffer the request until such time as we can deliver it to guest ...

I have more questions wrt the suggestion/workflow:
* at what place would you suggest buffering it?

PCIESlot maybe?

Quoted 2 lines

* what would be the request in this case, i.e. create PCI device anyways and try to signal hotplug event later?

that was my idea, yes.

Quoted 1 line

* what would baremethal do in such case?

exactly the same, human would wait until blinking stops.

Quoted 1 line

* what to do in case guest is never ready, what user should do in such case?

As in guest is stuck? Do we care? It can't use the device.

Quoted 1 line

* can be such device be removed?

why not? device_del could complete immediately ...

Quoted 9 lines

not sure that all of this is worth of the effort and added complexity.

alternatively:
maybe ports can send QMP events about it's state changes, which end user would be able to see + error like in this patch.

On top of it, mgmt could build a better UIx, like retry/notify logic if that's what user really wishes for and configures (it would be up to user to define behaviour).

I'd say let's get expected behaviour for existing commands first. We can add events and stuff on top.

Quoted 18 lines
Quoted 18 lines
Quoted 15 lines
Quoted 10 lines

What if we have a buggy guest with a kernel stuck in blinking? Is QEMU's responsibility to emulate the operator itself? Because the operator
is the one who is supposed to wait.

Thanks,
Marcel

[...]

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat  
#23 lore raw
Quoted 15 lines · On Tue, 27 Oct 2020 09:02:06 -0400 "Michael S. Tsirkin" <[email protected]> wrote:

On Tue, Oct 27, 2020 at 01:54:26PM +0100, Igor Mammedov wrote:
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]

Quoted 2 lines

I have more questions wrt the suggestion/workflow:
* at what place would you suggest buffering it?

Quoted 7 lines

PCIESlot maybe?

Quoted 2 lines

* what would be the request in this case, i.e. create PCI device anyways and try to signal hotplug event later?

that was my idea, yes.

Actually, I don't think that will quite work. A whole chunk of the problem here is that because the device is realized, the guest sees it as part of its general scan *before* the hotplug event (ABP and PDC interrupts) appears. That makes the guest misinterpret the ABP as an *unplug* request.

So delaying the interrupt without delaying the realize (or at least filtering config space access to it based on.. something) will just trigger the same problem AFAICT.

Quoted 21 lines
Quoted 1 line

* what would baremethal do in such case?

exactly the same, human would wait until blinking stops.

Quoted 1 line

* what to do in case guest is never ready, what user should do in such case?

As in guest is stuck? Do we care? It can't use the device.

Quoted 1 line

* can be such device be removed?

why not? device_del could complete immediately ...

[...]

I'd say let's get expected behaviour for existing commands first. We can add events and stuff on top.

[...]
[...]
[...]

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
#24 lore raw
In reply to Igor Mammedov
Quoted 9 lines · On Tue, 27 Oct 2020 13:54:26 +0100 Igor Mammedov <[email protected]> wrote:

On Tue, 27 Oct 2020 07:26:44 -0400
"Michael S. Tsirkin" <[email protected]> wrote:

[...]
[...]
[...]
[...]
[...]

Quoted 1 line
Quoted 1 line
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.

racy, timeout is bound to break once it's in overcommited env.

Hm. That's no less true at the management layer than it is at the qemu layer.

Quoted 4 lines
Quoted 2 lines

If it's helpful, I'd add a query to check state
so management can figure out why doesn't guest see device yet.

that means mgmt would have to poll it and forward it to user somehow.

If that even makes sense. In the case of Kata, it's supposed to be autonomously creating the VM, so there's nothing meaningful it can forward to the user other than "failed to create the container because of some hotplug problem that means nothing to you".

Quoted 11 lines

[...]
I have more questions wrt the suggestion/workflow:
* at what place would you suggest buffering it?
* what would be the request in this case, i.e. create PCI device anyways and try to signal hotplug event later?
* what would baremethal do in such case?
* what to do in case guest is never ready, what user should do in such case?
* can be such device be removed?

not sure that all of this is worth of the effort and added complexity.

Quoted 7 lines

alternatively:
maybe ports can send QMP events about it's state changes, which end user would be able to see + error like in this patch.

On top of it, mgmt could build a better UIx, like retry/notify logic if that's what user really wishes for and configures (it would be up to user to define behaviour).

That kind of makes sense if the user is explicitly requesting hotplugs, but that's not necessarily the case.

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
#25 lore raw
Quoted 12 lines · On Wed, 28 Oct 2020 14:31:35 +1100 David Gibson <[email protected]> wrote:
Quoted 9 lines · On Tue, 27 Oct 2020 13:54:26 +0100 Igor Mammedov <[email protected]> wrote:

On Tue, 27 Oct 2020 07:26:44 -0400
"Michael S. Tsirkin" <[email protected]> wrote:

[...]
[...]
[...]
[...]
[...]

Quoted 1 line
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.

racy, timeout is bound to break once it's in overcommited env.

Hm. That's no less true at the management layer than it is at the qemu layer.

true, but it's user policy which is defined by user not by QEMU.

Quoted 26 lines
Quoted 5 lines
Quoted 3 lines

If it's helpful, I'd add a query to check state
so management can figure out why doesn't guest see device yet.

that means mgmt would have to poll it and forward it to user somehow.

If that even makes sense. In the case of Kata, it's supposed to be autonomously creating the VM, so there's nothing meaningful it can forward to the user other than "failed to create the container because of some hotplug problem that means nothing to you".

Quoted 15 lines

[...]
I have more questions wrt the suggestion/workflow:
* at what place would you suggest buffering it?
* what would be the request in this case, i.e. create PCI device anyways and try to signal hotplug event later?
* what would baremethal do in such case?
* what to do in case guest is never ready, what user should do in such case?
* can be such device be removed?

not sure that all of this is worth of the effort and added complexity.

alternatively:
maybe ports can send QMP events about it's state changes, which end user would be able to see + error like in this patch.

Quoted 6 lines
Quoted 3 lines

On top of it, mgmt could build a better UIx, like retry/notify logic if that's what user really wishes for and configures (it would be up to user to define behaviour).

That kind of makes sense if the user is explicitly requesting hotplugs, but that's not necessarily the case.

user doesn't have to be a human, it could be some mgmt layer that would automate retry logic, depending on what actually user needs for particular task (i.e. fail immediately, retry N time then fail, retry with time out - then fail, don't care - succeed, ...). The point is for QEMU to provide means for mgmt to implement whatever policy user would need.

PS:
but then, I know close to nothing about PCI, so all of above might be nonsense.

#26 lore raw
Quoted 53 lines · Igor Mammedov
Quoted 12 lines · On Wed, 28 Oct 2020 14:31:35 +1100 David Gibson <[email protected]> wrote:
Quoted 9 lines · On Tue, 27 Oct 2020 13:54:26 +0100 Igor Mammedov <[email protected]> wrote:

On Tue, 27 Oct 2020 07:26:44 -0400
"Michael S. Tsirkin" <[email protected]> wrote:

[...]
[...]
[...]
[...]
[...]

Quoted 1 line
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.

racy, timeout is bound to break once it's in overcommited env.

Hm. That's no less true at the management layer than it is at the qemu layer.

true, but it's user policy which is defined by user not by QEMU.

Quoted 30 lines
Quoted 5 lines
Quoted 3 lines

If it's helpful, I'd add a query to check state
so management can figure out why doesn't guest see device yet.

that means mgmt would have to poll it and forward it to user somehow.

If that even makes sense. In the case of Kata, it's supposed to be autonomously creating the VM, so there's nothing meaningful it can forward to the user other than "failed to create the container because of some hotplug problem that means nothing to you".

Quoted 18 lines

[...]
I have more questions wrt the suggestion/workflow:
* at what place would you suggest buffering it?
* what would be the request in this case, i.e. create PCI device anyways and try to signal hotplug event later?
* what would baremethal do in such case?
* what to do in case guest is never ready, what user should do in such case?
* can be such device be removed?

not sure that all of this is worth of the effort and added complexity.

alternatively:
maybe ports can send QMP events about it's state changes, which end user would be able to see + error like in this patch.

On top of it, mgmt could build a better UIx, like retry/notify logic if that's what user really wishes for and configures (it would be up to user to define behaviour).

Quoted 7 lines
Quoted 2 lines

That kind of makes sense if the user is explicitly requesting hotplugs, but that's not necessarily the case.

user doesn't have to be a human, it could be some mgmt layer that would automate retry logic, depending on what actually user needs for particular task (i.e. fail immediately, retry N time then fail, retry with time out - then fail, don't care - succeed, ...). The point is for QEMU to provide means for mgmt to implement whatever policy user would need.

We are not coming up with new APIs here. Let's make existing ones work reliably first. We can talk about a flag where it fails instead of deferring hotplug, separately.

Quoted 2 lines

PS:
but then, I know close to nothing about PCI, so all of above might be nonsense.

#27 lore raw
In reply to Marcel Apfelbaum
Quoted 193 lines · Marcel Apfelbaum

Hi Michael,

On Thu, Oct 22, 2020 at 6:01 PM Michael S. Tsirkin <[email protected]> wrote:

On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
>
>
> On Thu, Oct 22, 2020 at 5:33 PM Michael S. Tsirkin <[email protected]> wrote:
>
>     On Thu, Oct 22, 2020 at 05:10:43PM +0300, Marcel Apfelbaum wrote:
>     >
>     >
>     > On Thu, Oct 22, 2020 at 5:01 PM Michael S. Tsirkin <[email protected]> >     wrote:
>     >
>     >     On Thu, Oct 22, 2020 at 04:55:10PM +0300, Marcel Apfelbaum wrote:
>     >     > Hi David, Michael,
>     >     >
>     >     > On Thu, Oct 22, 2020 at 3:56 PM David Gibson < [email protected]>
>     wrote:
>     >     >
>     >     >     On Thu, 22 Oct 2020 08:06:55 -0400
>     >     >     "Michael S. Tsirkin" <[email protected]> wrote:
>     >     >
>     >     >     > On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel Apfelbaum
>     wrote:
>     >     >     > > From: Marcel Apfelbaum <[email protected]> >     >     >     > >
>     >     >     > > During PCIe Root Port's transition from Power-Off to >     Power-ON (or
>     >     >     vice-versa)
>     >     >     > > the "Slot Control Register" has the "Power Indicator >     Control"
>     >     >     > > set to "Blinking" expressing a "power transition" mode.
>     >     >     > >
>     >     >     > > Any hotplug operation during the "power transition" mode is
>     not
>     >     >     permitted
>     >     >     > > or at least not expected by the Guest OS leading to strange
>     >     failures.
>     >     >     > >
>     >     >     > > Detect and refuse hotplug operations in such case. >     >     >     > >
>     >     >     > > Signed-off-by: Marcel Apfelbaum < [email protected]
>     >
>     >     >     > > ---
>     >     >     > >  hw/pci/pcie.c | 7 +++++++
>     >     >     > >  1 file changed, 7 insertions(+)
>     >     >     > >
>     >     >     > > diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c >     >     >     > > index 5b48bae0f6..2fe5c1473f 100644
>     >     >     > > --- a/hw/pci/pcie.c
>     >     >     > > +++ b/hw/pci/pcie.c
>     >     >     > > @@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb >     (HotplugHandler
>     >     >     *hotplug_dev, DeviceState *dev,
>     >     >     > >      PCIDevice *hotplug_pdev = PCI_DEVICE (hotplug_dev);
>     >     >     > >      uint8_t *exp_cap = hotplug_pdev->config + >     hotplug_pdev->
>     >     >     exp.exp_cap;
>     >     >     > >      uint32_t sltcap = pci_get_word(exp_cap + >     PCI_EXP_SLTCAP);
>     >     >     > > +    uint32_t sltctl = pci_get_word(exp_cap + >     PCI_EXP_SLTCTL);
>     >     >     > >Â
>     >     >     > >      /* Check if hot-plug is disabled on the slot */ >     >     >     > >      if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) =
>     = 0) {
>     >     >     > > @@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb >     >     (HotplugHandler
>     >     >     *hotplug_dev, DeviceState *dev,
>     >     >     > >          return;
>     >     >     > >      }
>     >     >     > >Â
>     >     >     > > +    if ((sltctl & PCI_EXP_SLTCTL_PIC) == >     >     PCI_EXP_SLTCTL_PWR_IND_BLINK)
>     >     >     {
>     >     >     > > +        error_setg(errp, "Hot-plug failed: %s is in Power
>     >     Transition",
>     >     >     > > +                   DEVICE(hotplug_pdev)->id); >     >     >     > > +        return;
>     >     >     > > +    }
>     >     >     > > +
>     >     >     > >      pcie_cap_slot_plug_common(PCI_DEVICE (hotplug_dev),
>     dev,
>     >     errp);
>     >     >     > >  }Â
>     >     >     >
>     >     >     > Probably the only way to handle for existing machine types.
>     >     >
>     >     >
>     >     > I agree
>     >     > Â
>     >     >
>     >     >     > For new ones, can't we queue it in host memory somewhere?
>     >     >
>     >     >
>     >     >
>     >     > I am not sure I understand what will be the flow.
>     >     >   - The user asks for a hotplug operation.
>     >     >   -  QEMU deferred operation.
>     >     > After that the operation may still fail, how would the user know if
>     the
>     >     > operation
>     >     > succeeded or not?
>     >
>     >
>     >     How can it fail? It's just a button press ...
>     >
>     >
>     >
>     > Currently we have "Hotplug unsupported."
>     > With this change we have "Guest/System not ready"
>
>
>     Hotplug unsupported is not an error that can trigger with
>     a well behaved management such as libvirt.
>
>
>     > Â
>     >
>     >     > Â
>     >     >
>     >     >     I'm not actually convinced we can't do that even for existing
>     machine
>     >     >     types.Â
>     >     >
>     >     >
>     >     > Is a Guest visible change, I don't think we can do it. >     >     > Â
>     >     >
>     >     >     So I'm a bit hesitant to suggest going ahead with this without
>     >     >     looking a bit closer at whether we can implement a >     wait-for-ready in
>     >     >     qemu, rather than forcing every user of qemu (human or machine)
>     to do
>     >     >     so.
>     >     >
>     >     >
>     >     > While I agree it is a pain from the usability point of view, >     hotplug
>     >     operations
>     >     > are allowed to fail. This is not more than a corner case, ensuring
>     the
>     >     right
>     >     > response (gracefully erroring out) may be enough.
>     >     >
>     >     > Thanks,
>     >     > Marcel
>     >     >
>     >
>     >
>     >     I don't think they ever failed in the past so management is unlikely
>     >     to handle the failure by retrying ...
>     >
>     >
>     > That would require some management handling, yes.
>     > But even without a "retry", failing is better than strange OS behavior.
>     >
>     > Trying a better alternative like deferring the operation for new machines
>     > would make sense, however is out of the scope of this patch
>
>     Expand the scope please. The scope should be "solve a problem xx" not >     "solve a problem xx by doing abc".
>
>
>
> The scope is detecting a hotplug error early instead
> passing to the Guest OS a hotplug operation that we know it will fail. >

Quoted 7 lines

Right. After detecting just failing unconditionally it a bit too simplistic IMHO.

Simplistic does not mean wrong or incorrect.
I fail to see why it is not enough.

The failure patch requires management to retry later.
A more elaborate scheme will fix the bug without need for management changes.

Quoted 2 lines

What QEMU can do better? Wait an unbounded time for the blinking to finish? What if we have a buggy guest with a kernel stuck in blinking?

Then it won't see the new device ever but does it even matter? It's stuck ... I'd ack adding a query command to see what is going on with the device. Can be generic, implementable on top of ACPI too.

Quoted 2 lines

Is QEMU's responsibility to emulate the operator itself? Because the operator is the one who is supposed to wait.

I think these details are immaterial for users. They don't read pci spec.

Quoted 5 lines

Thanks,
Marcel

[...]Â

#28 lore raw
In reply to Marcel Apfelbaum
Quoted 38 lines · On Thu, 22 Oct 2020 16:55:10 +0300 Marcel Apfelbaum…

Hi David, Michael,

On Thu, Oct 22, 2020 at 3:56 PM David Gibson <[email protected]> wrote:

[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]

Quoted 3 lines
Quoted 2 lines

Probably the only way to handle for existing machine types.

I agree

Quoted 3 lines
Quoted 1 line

For new ones, can't we queue it in host memory somewhere?

I am not sure I understand what will be the flow.
- The user asks for a hotplug operation.
- QEMU deferred operation.
After that the operation may still fail, how would the user know if the operation
succeeded or not?

Quoted 5 lines
Quoted 2 lines

I'm not actually convinced we can't do that even for existing machine types.

Is a Guest visible change, I don't think we can do it.

How is it a guest visible change?

Quoted 16 lines
Quoted 4 lines

So I'm a bit hesitant to suggest going ahead with this without looking a bit closer at whether we can implement a wait-for-ready in qemu, rather than forcing every user of qemu (human or machine) to do so.

While I agree it is a pain from the usability point of view, hotplug operations
are allowed to fail. This is not more than a corner case, ensuring the right response (gracefully erroring out) may be enough.

Thanks,
Marcel

[...]

Signature · 2 lines
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
#29 lore raw
In reply to Marcel Apfelbaum
Quoted 6 lines · Marcel Apfelbaum

From: Marcel Apfelbaum <[email protected]>

During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) the "Slot Control Register" has the "Power Indicator Control"
set to "Blinking" expressing a "power transition" mode.

Quoted 6 lines

Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.

Detect and refuse hotplug operations in such case.

Going back to this I have another question: could we get a bit more detail on when do we get into this situation? When does guest start blinking the indicator without us first starting a hotplug operation?

Quoted 31 lines
---
 hw/pci/pcie.c | 7 +++++++
 1 file changed, 7 insertions(+)
hw/pci/pcie.c +7 -0
index 5b48bae0f6..2fe5c1473f 100644
@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
     uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;
     uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
+    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL);
 
     /* Check if hot-plug is disabled on the slot */
     if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) {
@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         return;
     }
 
+    if ((sltctl & PCI_EXP_SLTCTL_PIC) == PCI_EXP_SLTCTL_PWR_IND_BLINK) {
+        error_setg(errp, "Hot-plug failed: %s is in Power Transition",
+                   DEVICE(hotplug_pdev)->id);
+        return;
+    }
+
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
 }
 
Signature · 1 line
2.17.2
#30 lore raw

Hi Michael,

Quoted 13 lines · Michael S. Tsirkin
Quoted 3 lines · Marcel Apfelbaum

From: Marcel Apfelbaum <[email protected]>

During PCIe Root Port's transition from Power-Off to Power-ON (or

vice-versa)

Quoted 8 lines

the "Slot Control Register" has the "Power Indicator Control" set to "Blinking" expressing a "power transition" mode.

Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.

Detect and refuse hotplug operations in such case.

Quoted 8 lines
Quoted 1 line

Going back to this I have another question: could we get a bit more detail on when do we get into this situation? When does guest start blinking the indicator without us first starting a hotplug operation?

While David has more insight on the kernel behavior during the mentioned issue,
it seems the PCI subsystem sets the Power Indicator
to "blinking" as part of the init sequence of the PCIe Root Port. The kernel will turn it off as soon as it finishes the init sequence.

A hotplug operation during the "init" sequence will surprise the Guest OS. I'll let David to give us more info.

Thank you,
Marcel

Quoted 35 lines

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler

Quoted 9 lines
---
 hw/pci/pcie.c | 7 +++++++
 1 file changed, 7 insertions(+)
hw/pci/pcie.c +0 -0
index 5b48bae0f6..2fe5c1473f 100644
@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler

*hotplug_dev, DeviceState *dev,

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler

Quoted 8 lines
hw/pci/pcie.c +1 -0
     PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
     uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;
     uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
+    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL);
 
     /* Check if hot-plug is disabled on the slot */
     if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) {
@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler

*hotplug_dev, DeviceState *dev,

Quoted 14 lines
hw/pci/pcie.c +6 -1
         return;
     }
 
+    if ((sltctl & PCI_EXP_SLTCTL_PIC) == PCI_EXP_SLTCTL_PWR_IND_BLINK) {
+        error_setg(errp, "Hot-plug failed: %s is in Power Transition",
+                   DEVICE(hotplug_pdev)->id);
+        return;
+    }
+
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
 }
 
--

2.17.2

#31 lore raw
In reply to Marcel Apfelbaum
Quoted 30 lines · Marcel Apfelbaum

From: Marcel Apfelbaum <[email protected]>

During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) the "Slot Control Register" has the "Power Indicator Control"
set to "Blinking" expressing a "power transition" mode.

Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.

Detect and refuse hotplug operations in such case.

---
 hw/pci/pcie.c | 7 +++++++
 1 file changed, 7 insertions(+)
hw/pci/pcie.c +1 -0
index 5b48bae0f6..2fe5c1473f 100644
@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
     PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
     uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;
     uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
+    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL);
 
     /* Check if hot-plug is disabled on the slot */
     if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) {
@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
         return;

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 429

Quoted 10 lines
hw/pci/pcie.c +6 -0
     }
 
+    if ((sltctl & PCI_EXP_SLTCTL_PIC) == PCI_EXP_SLTCTL_PWR_IND_BLINK) {
+        error_setg(errp, "Hot-plug failed: %s is in Power Transition",
+                   DEVICE(hotplug_pdev)->id);
+        return;
+    }
+
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
 }

I wonder if hw/pci/shpc.c is free from this issue?

Roman.

#32 lore raw

Hi Roman,

Quoted 45 lines · Roman Kagan
Quoted 3 lines · Marcel Apfelbaum

From: Marcel Apfelbaum <[email protected]>

During PCIe Root Port's transition from Power-Off to Power-ON (or

vice-versa)

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler

Quoted 18 lines

the "Slot Control Register" has the "Power Indicator Control" set to "Blinking" expressing a "power transition" mode.

Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.

Detect and refuse hotplug operations in such case.

---
 hw/pci/pcie.c | 7 +++++++
 1 file changed, 7 insertions(+)
hw/pci/pcie.c +0 -0
index 5b48bae0f6..2fe5c1473f 100644
@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler

*hotplug_dev, DeviceState *dev,

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler

Quoted 8 lines
hw/pci/pcie.c +1 -0
     PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev);
     uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;
     uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP);
+    uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL);
 
     /* Check if hot-plug is disabled on the slot */
     if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) {
@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler

*hotplug_dev, DeviceState *dev,

hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 429

Quoted 11 lines
hw/pci/pcie.c +6 -0
         return;
     }
 
+    if ((sltctl & PCI_EXP_SLTCTL_PIC) == PCI_EXP_SLTCTL_PWR_IND_BLINK) {
+        error_setg(errp, "Hot-plug failed: %s is in Power Transition",
+                   DEVICE(hotplug_pdev)->id);
+        return;
+    }
+
     pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
 }
Quoted 3 lines

I wonder if hw/pci/shpc.c is free from this issue?

I did not encounter the issue with shpc.

However, the SHPC spec describes the Power Indicator behavior for hotplug only leaving the booting sequence out of it:
"Although the Standard Usage Model presented in Section 2.2.1.2
requires a specific
relationship between the Power Indicator and the state of the slot,
this relationship is
enforced by software, not by the SHPC."

It looks like it depends on the software implementation.

Thanks,
Marcel

Quoted 2 lines

Roman.