pci: Refuse to hotplug PCI Devices when the Guest OS is not ready
PATCH 1/1 pci: Refuse to hotplug PCI Devices when the Guest OS is not ready
From: Marcel Apfelbaum <[email protected]>
During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) the "Slot Control Register" has the "Power Indicator Control"
set to "Blinking" expressing a "power transition" mode.
Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.
Detect and refuse hotplug operations in such case.
--- hw/pci/pcie.c | 7 +++++++ 1 file changed, 7 insertions(+)
hw/pci/pcie.c +7 -0
@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,PCIDevice *hotplug_pdev =PCI_DEVICE (hotplug_dev);uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;uint32_t sltcap =pci_get_word (exp_cap +PCI_EXP_SLTCAP ); +uint32_t sltctl =pci_get_word (exp_cap +PCI_EXP_SLTCTL );/* Check if hot-plug is disabled on the slot */ if (dev->hotplugged && (sltcap &PCI_EXP_SLTCAP_HPC ) ==0 ) {
@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,return ; } +if ((sltctl &PCI_EXP_SLTCTL_PIC ) ==PCI_EXP_SLTCTL_PWR_IND_BLINK ) { +error_setg (errp,"Hot-plug failed: %s is in Power Transition" , +DEVICE (hotplug_pdev)->id); +return ; + } +pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); }
Signature · 1 line
2.17.2
Quoted 30 lines · Marcel Apfelbaum
From: Marcel Apfelbaum <[email protected]>
During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) the "Slot Control Register" has the "Power Indicator Control"
set to "Blinking" expressing a "power transition" mode.Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.
Detect and refuse hotplug operations in such case.
Signed-off-by Marcel Apfelbaum <[email protected]>--- hw/pci/pcie.c | 7 +++++++ 1 file changed, 7 insertions(+)hw/pci/pcie.c +1
-0@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,PCIDevice *hotplug_pdev =PCI_DEVICE (hotplug_dev);uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;uint32_t sltcap =pci_get_word (exp_cap +PCI_EXP_SLTCAP ); +uint32_t sltctl =pci_get_word (exp_cap +PCI_EXP_SLTCTL );/* Check if hot-plug is disabled on the slot */ if (dev->hotplugged && (sltcap &PCI_EXP_SLTCAP_HPC ) ==0 ) {@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,return ;
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 429
Quoted 10 lines
hw/pci/pcie.c +6
-0} +if ((sltctl &PCI_EXP_SLTCTL_PIC ) ==PCI_EXP_SLTCTL_PWR_IND_BLINK ) { +error_setg (errp,"Hot-plug failed: %s is in Power Transition" , +DEVICE (hotplug_pdev)->id); +return ; + } +pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); }
Probably the only way to handle for existing machine types. For new ones, can't we queue it in host memory somewhere?
Quoted 2 lines
Signature · 1 line
2.17.2
Quoted 33 lines · On Thu, 22 Oct 2020 08:06:55 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 32 lines · Marcel Apfelbaum
From: Marcel Apfelbaum <[email protected]>
During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) the "Slot Control Register" has the "Power Indicator Control"
set to "Blinking" expressing a "power transition" mode.Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.
Detect and refuse hotplug operations in such case.
Signed-off-by Marcel Apfelbaum <[email protected]>--- hw/pci/pcie.c | 7 +++++++ 1 file changed, 7 insertions(+)hw/pci/pcie.c +1
-0@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,PCIDevice *hotplug_pdev =PCI_DEVICE (hotplug_dev);uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;uint32_t sltcap =pci_get_word (exp_cap +PCI_EXP_SLTCAP ); +uint32_t sltctl =pci_get_word (exp_cap +PCI_EXP_SLTCTL );/* Check if hot-plug is disabled on the slot */ if (dev->hotplugged && (sltcap &PCI_EXP_SLTCAP_HPC ) ==0 ) {@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,return ; }
Quoted 11 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 429
Quoted 8 lines
hw/pci/pcie.c +6
-0+if ((sltctl &PCI_EXP_SLTCTL_PIC ) ==PCI_EXP_SLTCTL_PWR_IND_BLINK ) { +error_setg (errp,"Hot-plug failed: %s is in Power Transition" , +DEVICE (hotplug_pdev)->id); +return ; + } +pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); }Probably the only way to handle for existing machine types. For new ones, can't we queue it in host memory somewhere?
I'm not actually convinced we can't do that even for existing machine types. So I'm a bit hesitant to suggest going ahead with this without looking a bit closer at whether we can implement a wait-for-ready in qemu, rather than forcing every user of qemu (human or machine) to do so.
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Quoted 48 lines · David Gibson
Quoted 33 lines · On Thu, 22 Oct 2020 08:06:55 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 32 lines · Marcel Apfelbaum
From: Marcel Apfelbaum <[email protected]>
During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) the "Slot Control Register" has the "Power Indicator Control"
set to "Blinking" expressing a "power transition" mode.Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.
Detect and refuse hotplug operations in such case.
Signed-off-by Marcel Apfelbaum <[email protected]>--- hw/pci/pcie.c | 7 +++++++ 1 file changed, 7 insertions(+)hw/pci/pcie.c +1
-0@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,PCIDevice *hotplug_pdev =PCI_DEVICE (hotplug_dev);uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;uint32_t sltcap =pci_get_word (exp_cap +PCI_EXP_SLTCAP ); +uint32_t sltctl =pci_get_word (exp_cap +PCI_EXP_SLTCTL );/* Check if hot-plug is disabled on the slot */ if (dev->hotplugged && (sltcap &PCI_EXP_SLTCAP_HPC ) ==0 ) {@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,return ; }Quoted 11 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 429
Quoted 8 lines
hw/pci/pcie.c +6
-0+if ((sltctl &PCI_EXP_SLTCTL_PIC ) ==PCI_EXP_SLTCTL_PWR_IND_BLINK ) { +error_setg (errp,"Hot-plug failed: %s is in Power Transition" , +DEVICE (hotplug_pdev)->id); +return ; + } +pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); }Probably the only way to handle for existing machine types. For new ones, can't we queue it in host memory somewhere?
Quoted 2 lines
I'm not actually convinced we can't do that even for existing machine types.
The difficulty would be in migrating the extra "reuested but defferred" state.
Quoted 9 lines
So I'm a bit hesitant to suggest going ahead with this without
looking a bit closer at whether we can implement a wait-for-ready in qemu, rather than forcing every user of qemu (human or machine) to do so.Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Quoted 5 lines · On Thu, 22 Oct 2020 09:15:28 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
On Thu, Oct 22, 2020 at 11:56:32PM +1100, David Gibson wrote:
[...]
[...]
[...]Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Probably the only way to handle for existing machine types. For new ones, can't we queue it in host memory somewhere?
I'm not actually convinced we can't do that even for existing machine types.
The difficulty would be in migrating the extra "reuested but defferred" state.
Ah, true. Although we could block migration for the duration instead.
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Hi David, Michael,
Quoted 38 lines · David Gibson
Quoted 4 lines · On Thu, 22 Oct 2020 08:06:55 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 3 lines · Marcel Apfelbaum
From: Marcel Apfelbaum <[email protected]>
During PCIe Root Port's transition from Power-Off to Power-ON (or
vice-versa)
Quoted 4 lines
Quoted 4 lines
the "Slot Control Register" has the "Power Indicator Control" set to "Blinking" expressing a "power transition" mode.
Any hotplug operation during the "power transition" mode is not
permitted
Quoted 9 lines
Quoted 9 lines
or at least not expected by the Guest OS leading to strange failures.
Detect and refuse hotplug operations in such case.
Signed-off-by Marcel Apfelbaum <[email protected]>--- hw/pci/pcie.c | 7 +++++++ 1 file changed, 7 insertions(+)Quoted 5 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler
Quoted 5 lines
hw/pci/pcie.c +0
-0@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler
*hotplug_dev, DeviceState *dev,Quoted 2 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler · line 411
Quoted 2 lines
hw/pci/pcie.c +0
-0PCIDevice *hotplug_pdev =PCI_DEVICE (hotplug_dev); uint8_t *exp_cap = hotplug_pdev->config +
hotplug_pdev->exp.exp_cap;Quoted 6 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler
Quoted 6 lines
hw/pci/pcie.c +1
-0uint32_t sltcap =pci_get_word (exp_cap +PCI_EXP_SLTCAP ); +uint32_t sltctl =pci_get_word (exp_cap +PCI_EXP_SLTCTL );/* Check if hot-plug is disabled on the slot */ if (dev->hotplugged && (sltcap &PCI_EXP_SLTCAP_HPC ) ==0 ) {@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler
*hotplug_dev, DeviceState *dev,
Quoted 15 lines
Quoted 4 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler · line 422
Quoted 4 lines
hw/pci/pcie.c +1
-0return ; } + if ((sltctl & PCI_EXP_SLTCTL_PIC) ==
PCI_EXP_SLTCTL_PWR_IND_BLINK) {Quoted 9 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 429
Quoted 7 lines
hw/pci/pcie.c +5
-0+error_setg (errp,"Hot-plug failed: %s is in Power Transition" , +DEVICE (hotplug_pdev)->id); +return ; + } +pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); }Probably the only way to handle for existing machine types.
I agree
Quoted 3 lines
Quoted 1 line
For new ones, can't we queue it in host memory somewhere?
I am not sure I understand what will be the flow.
- The user asks for a hotplug operation.
- QEMU deferred operation.
After that the operation may still fail, how would the user know if the operation
succeeded or not?
Quoted 2 lines
I'm not actually convinced we can't do that even for existing machine types.
Is a Guest visible change, I don't think we can do it.
Quoted 5 lines
So I'm a bit hesitant to suggest going ahead with this without looking a bit closer at whether we can implement a wait-for-ready in qemu, rather than forcing every user of qemu (human or machine) to do so.
While I agree it is a pain from the usability point of view, hotplug operations
are allowed to fail. This is not more than a corner case, ensuring the right response (gracefully erroring out) may be enough.
Thanks,
Marcel
Quoted 6 lines
--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
Quoted 65 lines · Marcel Apfelbaum
Hi David, Michael,
On Thu, Oct 22, 2020 at 3:56 PM David Gibson <[email protected]> wrote:
On Thu, 22 Oct 2020 08:06:55 -0400 "Michael S. Tsirkin" <[email protected]> wrote: > On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel Apfelbaum wrote: > > From: Marcel Apfelbaum <[email protected]> > > > > During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) > > the "Slot Control Register" has the "Power Indicator Control" > > set to "Blinking" expressing a "power transition" mode. > > > > Any hotplug operation during the "power transition" mode is not permitted > > or at least not expected by the Guest OS leading to strange failures. > > > > Detect and refuse hotplug operations in such case. > > > > Signed-off-by: Marcel Apfelbaum <[email protected]> > > --- > >Â hw/pci/pcie.c | 7 +++++++ > >Â 1 file changed, 7 insertions(+) > > > > diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c > > index 5b48bae0f6..2fe5c1473f 100644 > > --- a/hw/pci/pcie.c > > +++ b/hw/pci/pcie.c > > @@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, > >Â Â Â PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev); > >Â Â Â uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev-> exp.exp_cap; > >Â Â Â uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP); > > +Â Â uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL); > >Â > >Â Â Â /* Check if hot-plug is disabled on the slot */ > >Â Â Â if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) { > > @@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, > >Â Â Â Â Â return; > >Â Â Â } > >Â > > +Â Â if ((sltctl & PCI_EXP_SLTCTL_PIC) == PCI_EXP_SLTCTL_PWR_IND_BLINK) { > > +Â Â Â Â error_setg(errp, "Hot-plug failed: %s is in Power Transition", > > +Â Â Â Â Â Â Â Â Â Â DEVICE(hotplug_pdev)->id); > > +Â Â Â Â return; > > +Â Â } > > + > >Â Â Â pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp); > >Â }Â > > Probably the only way to handle for existing machine types.I agree
ÂQuoted 1 line
For new ones, can't we queue it in host memory somewhere?
Quoted 6 lines
I am not sure I understand what will be the flow.
 - The user asks for a hotplug operation.
 - QEMU deferred operation.
After that the operation may still fail, how would the user know if the operation
succeeded or not?
How can it fail? It's just a button press ...
Quoted 15 lines
Â
I'm not actually convinced we can't do that even for existing machine types.Â
Is a Guest visible change, I don't think we can do it. Â
So I'm a bit hesitant to suggest going ahead with this without looking a bit closer at whether we can implement a wait-for-ready in qemu, rather than forcing every user of qemu (human or machine) to do so.
Quoted 7 lines
While I agree it is a pain from the usability point of view, hotplug operations are allowed to fail. This is not more than a corner case, ensuring the right response (gracefully erroring out) may be enough.
Thanks,
Marcel
I don't think they ever failed in the past so management is unlikely to handle the failure by retrying ...
Quoted 7 lines
--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
Quoted 79 lines · Michael S. Tsirkin
Quoted 10 lines · Marcel Apfelbaum
Hi David, Michael,
On Thu, Oct 22, 2020 at 3:56 PM David Gibson <[email protected]> wrote:
On Thu, 22 Oct 2020 08:06:55 -0400
"Michael S. Tsirkin" <[email protected]> wrote:> On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel Apfelbaum wrote:
> > From: Marcel Apfelbaum <[email protected]>
> >Quoted 8 lines
> > During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa)
> > the "Slot Control Register" has the "Power Indicator Control" > > set to "Blinking" expressing a "power transition" mode. > >
> > Any hotplug operation during the "power transition" mode is not permitted
> > or at least not expected by the Guest OS leading to strangefailures.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Detect and refuse hotplug operations in such case.
Signed-off-by Marcel Apfelbaum <[email protected]>--- hw/pci/pcie.c | 7 +++++++ 1 file changed, 7 insertions(+)Quoted 15 lines
Quoted 5 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler
Quoted 5 lines
hw/pci/pcie.c +0
-0@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler*hotplug_dev, DeviceState *dev, > > PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev); > > uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev-> exp.exp_cap; > > uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP); > > + uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL); > > > > /* Check if hot-plug is disabled on the slot */ > > if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC) == 0) { > > @@ -418,6 +419,12 @@ voidpcie_cap_slot_pre_plug_cb(HotplugHandler
Quoted 5 lines
*hotplug_dev, DeviceState *dev, > > return; > > } > > > > + if ((sltctl & PCI_EXP_SLTCTL_PIC) ==
PCI_EXP_SLTCTL_PWR_IND_BLINK)Quoted 2 lines
{
> > + error_setg(errp, "Hot-plug failed: %s is in PowerTransition",
Quoted 5 lines
Quoted 5 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,
Quoted 5 lines
hw/pci/pcie.c +4
-0+ DEVICE(hotplug_pdev)->id); +return ; + } + pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev,errp);
Quoted 12 lines
Quoted 3 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler · line 415
Quoted 1 line
hw/pci/pcie.c +0
-0}Probably the only way to handle for existing machine types.
I agree
Quoted 1 line
For new ones, can't we queue it in host memory somewhere?
Quoted 6 lines
I am not sure I understand what will be the flow.
- The user asks for a hotplug operation.
- QEMU deferred operation.
After that the operation may still fail, how would the user know if the operation
succeeded or not?
Quoted 3 lines
How can it fail? It's just a button press ...
Currently we have "Hotplug unsupported."
With this change we have "Guest/System not ready"
Quoted 21 lines
Quoted 9 lines
I'm not actually convinced we can't do that even for existing machine types.
Is a Guest visible change, I don't think we can do it.
Quoted 7 lines
So I'm a bit hesitant to suggest going ahead with this without looking a bit closer at whether we can implement a wait-for-ready in qemu, rather than forcing every user of qemu (human or machine) to do so.
While I agree it is a pain from the usability point of view, hotplug
operations
Quoted 1 line
are allowed to fail. This is not more than a corner case, ensuring the
right
Quoted 2 lines
response (gracefully erroring out) may be enough.
Quoted 8 lines
Quoted 3 lines
Thanks,
MarcelI don't think they ever failed in the past so management is unlikely to handle the failure by retrying ...
That would require some management handling, yes.
But even without a "retry", failing is better than strange OS behavior.
Trying a better alternative like deferring the operation for new machines would make sense, however is out of the scope of this patch that simply detects the error leaving us in a slightly better state than today.
Thanks,
Marcel
Quoted 10 lines
Quoted 8 lines
--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
Quoted 83 lines · Marcel Apfelbaum
On Thu, Oct 22, 2020 at 5:01 PM Michael S. Tsirkin <[email protected]> wrote:
On Thu, Oct 22, 2020 at 04:55:10PM +0300, Marcel Apfelbaum wrote:
> Hi David, Michael,
>
> On Thu, Oct 22, 2020 at 3:56 PM David Gibson <[email protected]> wrote:
>
>Â Â Â On Thu, 22 Oct 2020 08:06:55 -0400
>Â Â Â "Michael S. Tsirkin" <[email protected]> wrote:
>
>Â Â Â > On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel Apfelbaum wrote:
>Â Â Â > > From: Marcel Apfelbaum <[email protected]>
>Â Â Â > >
>Â Â Â > > During PCIe Root Port's transition from Power-Off to Power-ON (or >Â Â Â vice-versa)
>Â Â Â > > the "Slot Control Register" has the "Power Indicator Control" >Â Â Â > > set to "Blinking" expressing a "power transition" mode. >Â Â Â > >
>Â Â Â > > Any hotplug operation during the "power transition" mode is not >Â Â Â permitted
>Â Â Â > > or at least not expected by the Guest OS leading to strange failures.
>Â Â Â > >
>Â Â Â > > Detect and refuse hotplug operations in such case.
>Â Â Â > >
>Â Â Â > > Signed-off-by: Marcel Apfelbaum <[email protected]> >Â Â Â > > ---
>Â Â Â > >Âhw/pci/pcie.c| 7 +++++++
>Â Â Â > >Â 1 file changed, 7 insertions(+)
>Â Â Â > >
>Â Â Â > > diff --gita/hw/pci/pcie.cb/hw/pci/pcie.c
>Â Â Â > > index5b48bae0f6..2fe5c1473f100644
>Â Â Â > > ---a/hw/pci/pcie.c
>Â Â Â > > +++b/hw/pci/pcie.c
>Â Â Â > > @@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler >Â Â Â *hotplug_dev, DeviceState *dev,
>Â Â Â > >Â Â Â PCIDevice*hotplug_pdev=PCI_DEVICE(hotplug_dev); >Â Â Â > >Â Â Âuint8_t*exp_cap=hotplug_pdev->config+ hotplug_pdev-> >Â Â Â exp.exp_cap;
>Â Â Â > >Â Â Âuint32_tsltcap =pci_get_word(exp_cap + PCI_EXP_SLTCAP); >Â Â Â > > +Â Âuint32_tsltctl =pci_get_word(exp_cap + PCI_EXP_SLTCTL); >Â Â Â > >Â
>Â Â Â > >Â Â Â /* Check if hot-plug is disabled on the slot */ >Â Â Â > >Â Â Â if (dev->hotplugged&& (sltcap &PCI_EXP_SLTCAP_HPC) == 0) { >Â Â Â > > @@ -418,6 +419,12 @@ voidpcie_cap_slot_pre_plug_cb(HotplugHandler
>Â Â Â *hotplug_dev, DeviceState *dev,
>Â Â Â > >Â Â Â Â Â return;
>Â Â Â > >Â Â Â }
>Â Â Â > >Â
>Â Â Â > > +Â Â if ((sltctl &PCI_EXP_SLTCTL_PIC) ==PCI_EXP_SLTCTL_PWR_IND_BLINK)
>Â Â Â {
>Â Â Â > > +Â Â Â Â error_setg(errp, "Hot-plug failed: %s is in Power Transition",
>Â Â Â > > +Â Â Â Â Â Â Â Â Â Â DEVICE(hotplug_pdev)->id); >Â Â Â > > +Â Â Â Â return;
>Â Â Â > > +Â Â }
>Â Â Â > > +
>Â Â Â > >Â Â Âpcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev, errp);
>Â Â Â > >Â }Â
>Â Â Â >
>Â Â Â > Probably the only way to handle for existing machine types. >
>
> I agree
> Â
>
>Â Â Â > For new ones, can't we queue it in host memory somewhere? >
>
>
> I am not sure I understand what will be the flow.
> Â - The user asks for a hotplug operation.
> Â -Â QEMU deferred operation.
> After that the operation may still fail, how would the user know if the > operation
> succeeded or not?
Quoted 6 lines
How can it fail? It's just a button press ...
Currently we have "Hotplug unsupported."
With this change we have "Guest/System not ready"
Hotplug unsupported is not an error that can trigger with a well behaved management such as libvirt.
Quoted 32 lines
Â
Quoted 9 lines
Â
   I'm not actually convinced we can't do that even for existing machine    types.Â
Is a Guest visible change, I don't think we can do it. Â
Quoted 7 lines
   So I'm a bit hesitant to suggest going ahead with this without    looking a bit closer at whether we can implement a wait-for-ready in    qemu, rather than forcing every user of qemu (human or machine) to do    so.
While I agree it is a pain from the usability point of view, hotplug
operations
> are allowed to fail. This is not more than a corner case, ensuring the right
> response (gracefully erroring out) may be enough.
>
> Thanks,
> Marcel
>I don't think they ever failed in the past so management is unlikely to handle the failure by retrying ...
Quoted 5 lines
That would require some management handling, yes.
But even without a "retry", failing is better than strange OS behavior.Trying a better alternative like deferring the operation for new machines would make sense, however is out of the scope of this patch
Expand the scope please. The scope should be "solve a problem xx" not "solve a problem xx by doing abc".
Quoted 5 lines
that simply
detects the error leaving us in a slightly better state than today.Thanks,
Marcel
Not applying a patch is the only tool we maintainers have to influence people to solve the problem fully. That's why I'm not inclined to apply "slightly better" patches generally.
Quoted 11 lines
Quoted 9 lines
   --
   David Gibson <[email protected]>
   Principal Software Engineer, Virtualization, Red Hat
Quoted 147 lines · Michael S. Tsirkin
Quoted 3 lines · Marcel Apfelbaum
On Thu, Oct 22, 2020 at 5:01 PM Michael S. Tsirkin <[email protected]>
Quoted 5 lines · wrote:
On Thu, Oct 22, 2020 at 04:55:10PM +0300, Marcel Apfelbaum wrote: > Hi David, Michael, > > On Thu, Oct 22, 2020 at 3:56 PM David Gibson <[email protected]>Quoted 5 lines · wrote:
Quoted 5 lines
On Thu, 22 Oct 2020 08:06:55 -0400 "Michael S. Tsirkin" <[email protected]> wrote: > On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel ApfelbaumQuoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
From: Marcel Apfelbaum <[email protected]>
During PCIe Root Port's transition from Power-Off to
Power-ON (or
Quoted 2 lines
Quoted 2 lines
vice-versa)
> > the "Slot Control Register" has the "Power IndicatorControl"
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
set to "Blinking" expressing a "power transition" mode.
Any hotplug operation during the "power transition" mode
is not
Quoted 7 lines
Quoted 2 lines
permitted
> > or at least not expected by the Guest OS leading to strangefailures.
> > >
> > > Detect and refuse hotplug operations in such case.
> > >
> > > Signed-off-by: Marcel Apfelbaum <Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
hw/pci/pcie.c void
Quoted 9 lines
--- hw/pci/pcie.c | 7 +++++++ 1 file changed, 7 insertions(+)hw/pci/pcie.c +0
-0@@ -410,6 +410,7 @@ voidpcie_cap_slot_pre_plug_cb(HotplugHandler
Quoted 3 lines
Quoted 3 lines
*hotplug_dev, DeviceState *dev, > > PCIDevice *hotplug_pdev = PCI_DEVICE(hotplug_dev); > > uint8_t *exp_cap = hotplug_pdev->config +hotplug_pdev->
Quoted 2 lines
Quoted 2 lines
exp.exp_cap;
> >uint32_tsltcap = pci_get_word(exp_cap +
PCI_EXP_SLTCAP);Quoted 1 line
Quoted 1 line
Quoted 1 line
hw/pci/pcie.c void · line 410
Quoted 1 line
hw/pci/pcie.c +1
-0+ uint32_t sltctl = pci_get_word(exp_cap +
PCI_EXP_SLTCTL);Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hw/pci/pcie.c void · line 412
Quoted 3 lines
hw/pci/pcie.c +0
-0/* Check if hot-plug is disabled on the slot */ if (dev->hotplugged && (sltcap & PCI_EXP_SLTCAP_HPC)== 0) {
Quoted 16 lines
Quoted 1 line
Quoted 1 line
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb
Quoted 1 line
hw/pci/pcie.c +0
-0@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler > *hotplug_dev, DeviceState *dev, > > > return; > > > } > > > > > > + if ((sltctl & PCI_EXP_SLTCTL_PIC) == PCI_EXP_SLTCTL_PWR_IND_BLINK) > { > > > + error_setg(errp, "Hot-plug failed: %s is in Power Transition", > > > + DEVICE(hotplug_pdev)->id); > > > + return; > > > + } > > > + > > > pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev),dev,
Quoted 9 lines
errp); > > > } > > > > Probably the only way to handle for existing machine types. > > > I agree > >Quoted 8 lines
> > For new ones, can't we queue it in host memory somewhere? > > > > I am not sure I understand what will be the flow. > - The user asks for a hotplug operation. > - QEMU deferred operation. > After that the operation may still fail, how would the user knowif the
Quoted 10 lines
Quoted 2 lines
operation
succeeded or not?How can it fail? It's just a button press ...
Currently we have "Hotplug unsupported."
With this change we have "Guest/System not ready"Hotplug unsupported is not an error that can trigger with a well behaved management such as libvirt.
Quoted 5 lines
Quoted 5 lines
I'm not actually convinced we can't do that even for existing
machine
Quoted 8 lines
Quoted 8 lines
types.
Is a Guest visible change, I don't think we can do it.
So I'm a bit hesitant to suggest going ahead with this without looking a bit closer at whether we can implement a
wait-for-ready in
Quoted 1 line
Quoted 1 line
qemu, rather than forcing every user of qemu (human or
machine) to do
Quoted 4 lines
Quoted 4 lines
so.While I agree it is a pain from the usability point of view,
hotplug
Quoted 2 lines
operations
> are allowed to fail. This is not more than a corner case, ensuringthe
Quoted 12 lines
right
> response (gracefully erroring out) may be enough. >
> Thanks,
> Marcel
>I don't think they ever failed in the past so management is unlikely to handle the failure by retrying ...
Quoted 10 lines
Quoted 5 lines
That would require some management handling, yes.
But even without a "retry", failing is better than strange OS behavior.Trying a better alternative like deferring the operation for new machines would make sense, however is out of the scope of this patch
Expand the scope please. The scope should be "solve a problem xx" not "solve a problem xx by doing abc".
The scope is detecting a hotplug error early instead passing to the Guest OS a hotplug operation that we know it will fail.
Quoted 8 lines
Quoted 5 lines
that simply
detects the error leaving us in a slightly better state than today.Thanks,
MarcelNot applying a patch is the only tool we maintainers have to influence people to solve the problem fully.
That's why I'm not inclined to apply
Quoted 3 lines
"slightly better" patches generally.
The patch is a proposal following some offline discussions on this matter. I personally see the value of it versus what we have today.
Thanks,
Marcel
Quoted 14 lines
Quoted 12 lines
Quoted 10 lines
--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
Quoted 157 lines · Marcel Apfelbaum
On Thu, Oct 22, 2020 at 5:33 PM Michael S. Tsirkin <[email protected]> wrote:
On Thu, Oct 22, 2020 at 05:10:43PM +0300, Marcel Apfelbaum wrote:
>
>
> On Thu, Oct 22, 2020 at 5:01 PM Michael S. Tsirkin <[email protected]> wrote:
>
>Â Â Â On Thu, Oct 22, 2020 at 04:55:10PM +0300, Marcel Apfelbaum wrote:
>Â Â Â > Hi David, Michael,
>Â Â Â >
>Â Â Â > On Thu, Oct 22, 2020 at 3:56 PM David Gibson <[email protected]> wrote:
>Â Â Â >
>Â Â Â >Â Â Â On Thu, 22 Oct 2020 08:06:55 -0400
>Â Â Â >Â Â Â "Michael S. Tsirkin" <[email protected]> wrote:
>Â Â Â >
>Â Â Â >Â Â Â > On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel Apfelbaum wrote:
>Â Â Â >Â Â Â > > From: Marcel Apfelbaum <[email protected]>
>Â Â Â >Â Â Â > >
>Â Â Â >Â Â Â > > During PCIe Root Port's transition from Power-Off to Power-ON (or
>Â Â Â >Â Â Â vice-versa)
>Â Â Â >Â Â Â > > the "Slot Control Register" has the "Power Indicator Control"
>Â Â Â >Â Â Â > > set to "Blinking" expressing a "power transition" mode. >Â Â Â >Â Â Â > >
>Â Â Â >Â Â Â > > Any hotplug operation during the "power transition" mode is not
>Â Â Â >Â Â Â permitted
>Â Â Â >Â Â Â > > or at least not expected by the Guest OS leading to strange >Â Â Â failures.
>Â Â Â >Â Â Â > >
>Â Â Â >Â Â Â > > Detect and refuse hotplug operations in such case. >Â Â Â >Â Â Â > >
>Â Â Â >Â Â Â > > Signed-off-by: Marcel Apfelbaum <[email protected] >
>Â Â Â >Â Â Â > > ---
>Â Â Â >Â Â Â > >Âhw/pci/pcie.c| 7 +++++++
>Â Â Â >Â Â Â > >Â 1 file changed, 7 insertions(+)
>Â Â Â >Â Â Â > >
>Â Â Â >Â Â Â > > diff --gita/hw/pci/pcie.cb/hw/pci/pcie.c
>Â Â Â >Â Â Â > > index5b48bae0f6..2fe5c1473f100644
>Â Â Â >Â Â Â > > ---a/hw/pci/pcie.c
>Â Â Â >Â Â Â > > +++b/hw/pci/pcie.c
>Â Â Â >Â Â Â > > @@ -410,6 +410,7 @@ voidpcie_cap_slot_pre_plug_cb(HotplugHandler
>Â Â Â >Â Â Â *hotplug_dev, DeviceState *dev,
>Â Â Â >Â Â Â > >Â Â Â PCIDevice*hotplug_pdev=PCI_DEVICE(hotplug_dev); >Â Â Â >Â Â Â > >Â Â Âuint8_t*exp_cap=hotplug_pdev->config+ hotplug_pdev->
>Â Â Â >Â Â Â exp.exp_cap;
>Â Â Â >Â Â Â > >Â Â Âuint32_tsltcap =pci_get_word(exp_cap + PCI_EXP_SLTCAP);
>Â Â Â >Â Â Â > > +Â Âuint32_tsltctl =pci_get_word(exp_cap + PCI_EXP_SLTCTL);
>Â Â Â >Â Â Â > >Â
>Â Â Â >Â Â Â > >Â Â Â /* Check if hot-plug is disabled on the slot */ >Â Â Â >Â Â Â > >Â Â Â if (dev->hotplugged&& (sltcap &PCI_EXP_SLTCAP_HPC) = = 0) {
>Â Â Â >Â Â Â > > @@ -418,6 +419,12 @@ voidpcie_cap_slot_pre_plug_cb>Â Â Â (HotplugHandler
>Â Â Â >Â Â Â *hotplug_dev, DeviceState *dev,
>Â Â Â >Â Â Â > >Â Â Â Â Â return;
>Â Â Â >Â Â Â > >Â Â Â }
>Â Â Â >Â Â Â > >Â
>Â Â Â >Â Â Â > > +Â Â if ((sltctl &PCI_EXP_SLTCTL_PIC) ==
>Â Â Â PCI_EXP_SLTCTL_PWR_IND_BLINK)
>Â Â Â >Â Â Â {
>Â Â Â >Â Â Â > > +Â Â Â Â error_setg(errp, "Hot-plug failed: %s is in Power >Â Â Â Transition",
>Â Â Â >Â Â Â > > +Â Â Â Â Â Â Â Â Â Â DEVICE(hotplug_pdev)->id); >Â Â Â >Â Â Â > > +Â Â Â Â return;
>Â Â Â >Â Â Â > > +Â Â }
>Â Â Â >Â Â Â > > +
>Â Â Â >Â Â Â > >Â Â Â pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev), dev,
>Â Â Â errp);
>Â Â Â >Â Â Â > >Â }Â
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â > Probably the only way to handle for existing machine types. >Â Â Â >
>Â Â Â >
>Â Â Â > I agree
>Â Â Â > Â
>Â Â Â >
>Â Â Â >Â Â Â > For new ones, can't we queue it in host memory somewhere? >Â Â Â >
>Â Â Â >
>Â Â Â >
>Â Â Â > I am not sure I understand what will be the flow.
>Â Â Â > Â - The user asks for a hotplug operation.
>Â Â Â > Â -Â QEMU deferred operation.
>Â Â Â > After that the operation may still fail, how would the user know if the
>Â Â Â > operation
>Â Â Â > succeeded or not?
>
>
>Â Â Â How can it fail? It's just a button press ...
>
>
>
> Currently we have "Hotplug unsupported."
> With this change we have "Guest/System not ready"Hotplug unsupported is not an error that can trigger with a well behaved management such as libvirt.
> Â
>
>Â Â Â > Â
>Â Â Â >
>Â Â Â >Â Â Â I'm not actually convinced we can't do that even for existing machine
>Â Â Â >Â Â Â types.Â
>Â Â Â >
>Â Â Â >
>Â Â Â > Is a Guest visible change, I don't think we can do it. >Â Â Â > Â
>Â Â Â >
>Â Â Â >Â Â Â So I'm a bit hesitant to suggest going ahead with this without >Â Â Â >Â Â Â looking a bit closer at whether we can implement a wait-for-ready in
>Â Â Â >Â Â Â qemu, rather than forcing every user of qemu (human or machine) to do
>Â Â Â >Â Â Â so.
>Â Â Â >
>Â Â Â >
>Â Â Â > While I agree it is a pain from the usability point of view, hotplug
>Â Â Â operations
>Â Â Â > are allowed to fail. This is not more than a corner case, ensuring the
>Â Â Â right
>Â Â Â > response (gracefully erroring out) may be enough.
>Â Â Â >
>Â Â Â > Thanks,
>Â Â Â > Marcel
>Â Â Â >
>
>
>Â Â Â I don't think they ever failed in the past so management is unlikely >Â Â Â to handle the failure by retrying ...
>
>
> That would require some management handling, yes.
> But even without a "retry", failing is better than strange OS behavior. >
> Trying a better alternative like deferring the operation for new machines > would make sense, however is out of the scope of this patch
Quoted 8 lines
Expand the scope please. The scope should be "solve a problem xx" not "solve a problem xx by doing abc".
The scope is detecting a hotplug error early instead passing to the Guest OS a hotplug operation that we know it will fail.
Right. After detecting just failing unconditionally it a bit too simplistic IMHO.
Quoted 35 lines
Quoted 6 lines
that simply
detects the error leaving us in a slightly better state than today.Thanks,
MarcelNot applying a patch is the only tool we maintainers have to influence people to solve the problem fully.Â
That's why I'm not inclined to apply
"slightly better" patches generally.The patch is a proposal following some offline discussions on this matter. I personally see the value of it versus what we have today.
Thanks,
MarcelQuoted 11 lines
   >
   >
   >
   >   --
   >   David Gibson <[email protected]>
   >   Principal Software Engineer, Virtualization, Red Hat    >
Quoted 5 lines · On Thu, 22 Oct 2020 11:01:04 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
[...]Right. After detecting just failing unconditionally it a bit too simplistic IMHO.
There's also another factor here, which I thought I'd mentioned already, but looks like I didn't: I think we're still missing some details in what's going on.
The premise for this patch is that plugging while the indicator is in transition state is allowed to fail in any way on the guest side. I don't think that's a reasonable interpretation, because it's unworkable for physical hotplug. If the indicator starts blinking while you're in the middle of shoving a card in, you'd be in trouble.
So, what I'm assuming here is that while "don't plug while blinking" is the instruction for the operator to obey as best they can, on the guest side the rule has to be "start blinking, wait a while and by the time you leave blinking state again, you can be confident any plugs or unplugs have completed". Obviously still racy in the strict computer science sense, but about the best you can do with slow humans in the mix.
So, qemu should of course endeavour to follow that rule as though it was a human operator on a physical machine and not plug when the indicator is blinking. *But* the qemu plug will in practice be fast enough that if we're hitting real problems here, it suggests the guest is still doing something wrong.
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Hi David,
Quoted 27 lines · David Gibson
Quoted 5 lines · On Thu, 22 Oct 2020 11:01:04 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
[...]Right. After detecting just failing unconditionally it a bit too simplistic IMHO.
There's also another factor here, which I thought I'd mentioned already, but looks like I didn't: I think we're still missing some details in what's going on.
The premise for this patch is that plugging while the indicator is in transition state is allowed to fail in any way on the guest side. I don't think that's a reasonable interpretation, because it's unworkable for physical hotplug. If the indicator starts blinking while you're in the middle of shoving a card in, you'd be in trouble.
So, what I'm assuming here is that while "don't plug while blinking" is the instruction for the operator to obey as best they can, on the guest side the rule has to be "start blinking, wait a while and by the time you leave blinking state again, you can be confident any plugs or unplugs have completed". Obviously still racy in the strict computer science sense, but about the best you can do with slow humans in the mix.
Quoted 6 lines
So, qemu should of course endeavour to follow that rule as though it was a human operator on a physical machine and not plug when the indicator is blinking. *But* the qemu plug will in practice be fast enough that if we're hitting real problems here, it suggests the guest is still doing something wrong.
I personally think there is a little bit of over-engineering here. Let's start with the spec:
Power Indicator Blinking
A blinking Power Indicator indicates that the slot is powering up or
powering down and that
insertion or removal of the adapter is not permitted.
What exactly is an interpretation here?
As you stated, the races are theoretical, the whole point of the indicator is to let the operator know he can't plug the device just yet.
I understand it would be more user friendly if the QEMU would wait internally for the
blinking to end, but the whole point of the indicator is to let the operator (human or machine)
know they can't plug the device at a specific time.
Should QEMU take the responsibility of the operator? Is it even correct?
Even if we would want such a feature, how is it related to this patch? The patch simply refuses to start a hotplug operation when it knows it will not succeed.
Another way that would make sense to me would be is a new QEMU interface other than
"add_device", let's say "adding_device_allowed", that would return true if the hotplug is allowed
at this point of time. (I am aware of the theoretical races)
The above will at least mimic the mechanics of the pyhs world. The operator looks at the indicator,
the management software checks if adding the device is allowed. Since it is a corner case I would prefer the device_add to fail rather than introducing a new interface,
but that's just me.
Thanks,
Marcel
Quoted 4 lines
--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
Quoted 57 lines · Marcel Apfelbaum
Hi David,
On Fri, Oct 23, 2020 at 6:49 AM David Gibson <[email protected]> wrote:
On Thu, 22 Oct 2020 11:01:04 -0400
"Michael S. Tsirkin" <[email protected]> wrote:> On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
>Â [...]Â
>
> Right. After detecting just failing unconditionally it a bit too > simplistic IMHO.There's also another factor here, which I thought I'd mentioned already, but looks like I didn't: I think we're still missing some details in what's going on.
The premise for this patch is that plugging while the indicator is in transition state is allowed to fail in any way on the guest side. I don't think that's a reasonable interpretation, because it's unworkable for physical hotplug. If the indicator starts blinking while you're in the middle of shoving a card in, you'd be in trouble.
So, what I'm assuming here is that while "don't plug while blinking" is the instruction for the operator to obey as best they can, on the guest side the rule has to be "start blinking, wait a while and by the time you leave blinking state again, you can be confident any plugs or unplugs have completed". Obviously still racy in the strict computer science sense, but about the best you can do with slow humans in the mix.
So, qemu should of course endeavour to follow that rule as though it was a human operator on a physical machine and not plug when the indicator is blinking. *But* the qemu plug will in practice be fast enough that if we're hitting real problems here, it suggests the guest is still doing something wrong.
I personally think there is a little bit of over-engineering here. Let's start with the spec:
  Power Indicator Blinking
  A blinking Power Indicator indicates that the slot is powering up or powering down and that
  insertion or removal of the adapter is not permitted.What exactly is an interpretation here?
As you stated, the races are theoretical, the whole point of the indicator is to let the operator know he can't plug the device just yet.I understand it would be more user friendly if the QEMU would wait internally for the
blinking to end, but the whole point of the indicator is to let the operator (human or machine)
know they can't plug the device at a specific time.
Should QEMU take the responsibility of the operator? Is it even correct?
Quoted 9 lines
Even if we would want such a feature, how is it related to this patch? The patch simply refuses to start a hotplug operation when it knows it will not succeed.Â
Â
Another way that would make sense to me would be is a new QEMU interface other than
"add_device", let's say "adding_device_allowed", that would return true if the hotplug is allowed
at this point of time. (I am aware of the theoretical races)Â
Rather than adding_device_allowed, something like "query slot" might be helpful for debugging. That would help user figure out e.g. why isn't device visible without any races.
Quoted 10 lines
The above will at least mimic the mechanics of the pyhs world. The operator looks at the indicator,
the management software checks if adding the device is allowed. Since it is a corner case I would prefer thedevice_addto fail rather than introducing a new interface,
but that's just me.Thanks,
Marcel
I think we want QEMU management interface to be reasonably abstract and agnostic if possible. Pushing knowledge of hardware detail to management will just lead to pain IMHO.
We supported device_add which practically never fails for years, at this point it's easier to keep supporting it than change all users ...
Quoted 5 lines
--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
Quoted 68 lines · On Fri, 23 Oct 2020 11:54:40 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 57 lines · Marcel Apfelbaum
Hi David,
On Fri, Oct 23, 2020 at 6:49 AM David Gibson <[email protected]> wrote:
On Thu, 22 Oct 2020 11:01:04 -0400
"Michael S. Tsirkin" <[email protected]> wrote:> On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
> [...]ÂÂ
>
> Right. After detecting just failing unconditionally it a bit too > simplistic IMHO.There's also another factor here, which I thought I'd mentioned already, but looks like I didn't: I think we're still missing some details in what's going on.
The premise for this patch is that plugging while the indicator is in transition state is allowed to fail in any way on the guest side. I don't think that's a reasonable interpretation, because it's unworkable for physical hotplug. If the indicator starts blinking while you're in the middle of shoving a card in, you'd be in trouble.
So, what I'm assuming here is that while "don't plug while blinking" is the instruction for the operator to obey as best they can, on the guest side the rule has to be "start blinking, wait a while and by the time you leave blinking state again, you can be confident any plugs or unplugs have completed". Obviously still racy in the strict computer science sense, but about the best you can do with slow humans in the mix.
So, qemu should of course endeavour to follow that rule as though it was a human operator on a physical machine and not plug when the indicator is blinking. *But* the qemu plug will in practice be fast enough that if we're hitting real problems here, it suggests the guest is still doing something wrong.
I personally think there is a little bit of over-engineering here. Let's start with the spec:
  Power Indicator Blinking
  A blinking Power Indicator indicates that the slot is powering up or powering down and that
  insertion or removal of the adapter is not permitted.What exactly is an interpretation here?
As you stated, the races are theoretical, the whole point of the indicator is to let the operator know he can't plug the device just yet.I understand it would be more user friendly if the QEMU would wait internally for the
blinking to end, but the whole point of the indicator is to let the operator (human or machine)
know they can't plug the device at a specific time.
Should QEMU take the responsibility of the operator? Is it even correct?Quoted 9 lines
Even if we would want such a feature, how is it related to this patch? The patch simply refuses to start a hotplug operation when it knows it will not succeed.ÂÂ
ÂÂ
Another way that would make sense to me would be is a new QEMU interface other than
"add_device", let's say "adding_device_allowed", that would return true if the hotplug is allowed
at this point of time. (I am aware of the theoretical races)ÂÂ
Quoted 3 lines
Rather than
adding_device_allowed, something like "query slot" might be helpful for debugging. That would help user figure out e.g. why isn't device visible without any races.
Would be new command useful tough? What we end up is broken guest (if I read commit message right) and a user who has no idea if device_add was successful or not.
So what user should do in this case
- wait till it explodes?
- can user remove it or it would be stuck there forever?
- poll slot before hotplug, manually?
(if this is the case then failing device_add cleanly doesn't sound bad,
it looks similar to another error we have "/* Check if hot-plug is disabled on the slot */" in pcie_cap_slot_pre_plug_cb)
CCing libvirt, as it concerns not only QEMU.
Quoted 8 lines
Quoted 8 lines
The above will at least mimic the mechanics of the pyhs world. The operator looks at the indicator,
the management software checks if adding the device is allowed.
Since it is a corner case I would prefer thedevice_addto fail rather than introducing a new interface,
but that's just me.
Quoted 8 lines
Quoted 3 lines
Thanks,
MarcelI think we want QEMU management interface to be reasonably abstract and agnostic if possible. Pushing knowledge of hardware detail to management will just lead to pain IMHO.
We supporteddevice_addwhich practically never fails for years,
For CPUs and RAM, device_add can fail so maybe management is also prepared to handle errors on PCI hotplug path.
Quoted 11 lines
at this point it's easier to keep supporting it than change all users ...
Quoted 5 lines
--
David Gibson <[email protected]>
Principal Software Engineer, Virtualization, Red Hat
Quoted 29 lines · On Fri, 23 Oct 2020 19:27:55 +0200 Igor Mammedov <[email protected]> wrote:
On Fri, 23 Oct 2020 11:54:40 -0400
"Michael S. Tsirkin" <[email protected]> wrote:[...]
[...]
[...]
[...]Quoted 4 lines
Rather than
adding_device_allowed, something like "query slot" might be helpful for debugging. That would help user figure out e.g. why isn't device visible without any races.Would be new command useful tough? What we end up is broken guest (if I read commit message right) and a user who has no idea if
device_addwas successful or not.
So what user should do in this case
- wait till it explodes?
- can user remove it or it would be stuck there forever?
- poll slot before hotplug, manually?(if this is the case then failing
device_addcleanly doesn't sound bad,
it looks similar to another error we have "/* Check if hot-plug is disabled on the slot */" inpcie_cap_slot_pre_plug_cb)CCing libvirt, as it concerns not only QEMU.
[...]
[...]Quoted 1 line
Quoted 7 lines
Quoted 4 lines
I think we want QEMU management interface to be reasonably abstract and agnostic if possible. Pushing knowledge of hardware detail to management will just lead to pain IMHO.
We supporteddevice_addwhich practically never fails for years,For CPUs and RAM,
device_addcan fail so maybe management is also prepared to handle errors on PCI hotplug path.
There can be unarguable reasons for PCI hotplug to fail as well (attempting to plug to a bus that can't support it for one). The difference here is that it's a failure that we expect to be transitory.
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Quoted 83 lines · Igor Mammedov
Quoted 68 lines · On Fri, 23 Oct 2020 11:54:40 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 57 lines · Marcel Apfelbaum
Hi David,
On Fri, Oct 23, 2020 at 6:49 AM David Gibson <[email protected]> wrote:
On Thu, 22 Oct 2020 11:01:04 -0400
"Michael S. Tsirkin" <[email protected]> wrote:> On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
> [...]ÂÂ
>
> Right. After detecting just failing unconditionally it a bit too > simplistic IMHO.There's also another factor here, which I thought I'd mentioned already, but looks like I didn't: I think we're still missing some details in what's going on.
The premise for this patch is that plugging while the indicator is in transition state is allowed to fail in any way on the guest side. I don't think that's a reasonable interpretation, because it's unworkable for physical hotplug. If the indicator starts blinking while you're in the middle of shoving a card in, you'd be in trouble.
So, what I'm assuming here is that while "don't plug while blinking" is the instruction for the operator to obey as best they can, on the guest side the rule has to be "start blinking, wait a while and by the time you leave blinking state again, you can be confident any plugs or unplugs have completed". Obviously still racy in the strict computer science sense, but about the best you can do with slow humans in the mix.
So, qemu should of course endeavour to follow that rule as though it was a human operator on a physical machine and not plug when the indicator is blinking. *But* the qemu plug will in practice be fast enough that if we're hitting real problems here, it suggests the guest is still doing something wrong.
I personally think there is a little bit of over-engineering here. Let's start with the spec:
  Power Indicator Blinking
  A blinking Power Indicator indicates that the slot is powering up or powering down and that
  insertion or removal of the adapter is not permitted.What exactly is an interpretation here?
As you stated, the races are theoretical, the whole point of the indicator is to let the operator know he can't plug the device just yet.I understand it would be more user friendly if the QEMU would wait internally for the
blinking to end, but the whole point of the indicator is to let the operator (human or machine)
know they can't plug the device at a specific time.
Should QEMU take the responsibility of the operator? Is it even correct?Quoted 9 lines
Even if we would want such a feature, how is it related to this patch? The patch simply refuses to start a hotplug operation when it knows it will not succeed.ÂÂ
ÂÂ
Another way that would make sense to me would be is a new QEMU interface other than
"add_device", let's say "adding_device_allowed", that would return true if the hotplug is allowed
at this point of time. (I am aware of the theoretical races)ÂÂQuoted 3 lines
Rather than
adding_device_allowed, something like "query slot" might be helpful for debugging. That would help user figure out e.g. why isn't device visible without any races.Would be new command useful tough? What we end up is broken guest (if I read commit message right) and a user who has no idea if
device_addwas successful or not.
So what user should do in this case
- wait till it explodes?
- can user remove it or it would be stuck there forever?
- poll slot before hotplug, manually?
Quoted 5 lines
(if this is the case then failing
device_addcleanly doesn't sound bad,
it looks similar to another error we have "/* Check if hot-plug is disabled on the slot */" inpcie_cap_slot_pre_plug_cb)CCing libvirt, as it concerns not only QEMU.
The only reason a separate command might make sense is if libvirt would want to provide a specific error to the user/upper management layer that the operation failed due to a transient failure (so that it can be retried later).
We don't really want to go to a policy decision of how long to wait in such case, so unless qemu waits libvirt will plainly want to report an error.
That said IMO 'device_add' should still fail if it's certain that the device won't be plugged in. This will fix any client which is currently in use. Adding a separate command is worth only for pre-checking for saner error handling.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
The above will at least mimic the mechanics of the pyhs world. The operator looks at the indicator,
the management software checks if adding the device is allowed.
Since it is a corner case I would prefer thedevice_addto fail rather than introducing a new interface,
but that's just me.Thanks,
Marcel
Quoted 7 lines
Quoted 4 lines
I think we want QEMU management interface to be reasonably abstract and agnostic if possible. Pushing knowledge of hardware detail to management will just lead to pain IMHO.
We supporteddevice_addwhich practically never fails for years,For CPUs and RAM,
device_addcan fail so maybe management is also prepared to handle errors on PCI hotplug path.
While it was me who implemented device_add for cpu/memory I don't remmeber any more whether we are really prepared for it.
We certainly want to do it if there's a possibility to do it.
Quoted 38 lines · On Fri, 23 Oct 2020 09:47:14 +0300 Marcel Apfelbaum…
Hi David,
Quoted 27 lines · David Gibson
Quoted 5 lines · On Thu, 22 Oct 2020 11:01:04 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
[...]Right. After detecting just failing unconditionally it a bit too simplistic IMHO.
There's also another factor here, which I thought I'd mentioned already, but looks like I didn't: I think we're still missing some details in what's going on.
The premise for this patch is that plugging while the indicator is in transition state is allowed to fail in any way on the guest side. I don't think that's a reasonable interpretation, because it's unworkable for physical hotplug. If the indicator starts blinking while you're in the middle of shoving a card in, you'd be in trouble.
So, what I'm assuming here is that while "don't plug while blinking" is the instruction for the operator to obey as best they can, on the guest side the rule has to be "start blinking, wait a while and by the time you leave blinking state again, you can be confident any plugs or unplugs have completed". Obviously still racy in the strict computer science sense, but about the best you can do with slow humans in the mix.
Quoted 6 lines
So, qemu should of course endeavour to follow that rule as though it was a human operator on a physical machine and not plug when the indicator is blinking. *But* the qemu plug will in practice be fast enough that if we're hitting real problems here, it suggests the guest is still doing something wrong.
Quoted 7 lines
I personally think there is a little bit of over-engineering here. Let's start with the spec:
Power Indicator Blinking
A blinking Power Indicator indicates that the slot is powering up or
powering down and that
insertion or removal of the adapter is not permitted.
Right, but the weird bit here is that IIUC, the kernel during general probe is switching the indicator from off -> blinking -> off without it ever going to "on" state. And it seems to do the power on and presence check with the indicator still in blinking state. This is different from the normal sequence on a hotplug:
press button
indicator goes to blinking
...wait...
indicator goes to full on
slot powers on
presence detect
Or am I missing something?
Quoted 10 lines
What exactly is an interpretation here?
As you stated, the races are theoretical, the whole point of the indicator is to let the operator know he can't plug the device just yet.I understand it would be more user friendly if the QEMU would wait internally for the
blinking to end, but the whole point of the indicator is to let the operator (human or machine)
know they can't plug the device at a specific time.
Should QEMU take the responsibility of the operator? Is it even correct?
I don't think there's an unambiguous correct answer here. But I think it's reasonable to interpret a general "device_add" as "instruct the virtual operator to plug in the card ASAP" as easily as "shove the virtual card in right now". device_add already covers a bunch of different pluggable interfaces, so I don't think precisely aligning to pci-e semantics is really a priority.
Quoted 3 lines
Even if we would want such a feature, how is it related to this patch? The patch simply refuses to start a hotplug operation when it knows it will not succeed.
I don't think I was clear. There are two separate and unrelated things I'm bringing up here. The first is that having the device_add fail for transitory reasons that the management layer doesn't really care about is really bad UX.
But the second point I'm raising here is that "don't plug when blinking" is not, strictly speaking an implementable strategy for a human operator. That makes me conclude that the idea here is plugs started at basically the same time as the blinking starts (which could be a little before or a little after, humans being fuzzy) should actually be acceptable, even if they finish after the indicator is blinking. It's not clear to me that the kernel's current behaviour actually permits that.
Quoted 6 lines
Another way that would make sense to me would be is a new QEMU interface other than
"add_device", let's say "adding_device_allowed", that would return true if the hotplug is allowed
at this point of time. (I am aware of the theoretical races)
Quoted 6 lines
The above will at least mimic the mechanics of the pyhs world. The operator looks at the indicator,
the management software checks if adding the device is allowed. Since it is a corner case I would prefer thedevice_addto fail rather than introducing a new interface,
but that's just me.
Oh, no, that's not what I'm suggesting. Adding an "is allowed" command is even worse. It makes the management's task *more* complex, which making any possible race here even wider.
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Hi Michael,
Quoted 186 lines · Michael S. Tsirkin
Quoted 3 lines · Marcel Apfelbaum
On Thu, Oct 22, 2020 at 5:33 PM Michael S. Tsirkin <[email protected]>
Quoted 9 lines · wrote:
On Thu, Oct 22, 2020 at 05:10:43PM +0300, Marcel Apfelbaum wrote: > > > On Thu, Oct 22, 2020 at 5:01 PM Michael S. Tsirkin <[email protected] wrote: > > On Thu, Oct 22, 2020 at 04:55:10PM +0300, Marcel ApfelbaumQuoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Hi David, Michael,
On Thu, Oct 22, 2020 at 3:56 PM David Gibson <
Quoted 6 lines
wrote: > > > > On Thu, 22 Oct 2020 08:06:55 -0400 > > "Michael S. Tsirkin" <[email protected]> wrote: > > > > > On Thu, Oct 22, 2020 at 02:40:26PM +0300, MarcelApfelbaum
Quoted 9 lines
wrote:
> > > > From: Marcel Apfelbaum <[email protected]>
> > > >
> > > > During PCIe Root Port's transition from Power-Off to
Power-ON (or
> > vice-versa)
> > > > the "Slot Control Register" has the "Power Indicator
Control"
> > > > set to "Blinking" expressing a "power transition"mode.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Any hotplug operation during the "power transition"
mode is
Quoted 3 lines
not
> > permitted
> > > > or at least not expected by the Guest OS leading tostrange
Quoted 5 lines
Quoted 5 lines
failures.
> > >
> > > Detect and refuse hotplug operations in such case.
> > >
> > > Signed-off-by: Marcel Apfelbaum <Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
--- hw/pci/pcie.c | 7 +++++++ 1 file changed, 7 insertions(+)Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb
Quoted 5 lines
hw/pci/pcie.c +0
-0@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler
> >*hotplug_dev, DeviceState *dev,
> > > > PCIDevice*hotplug_pdev=
PCI_DEVICE(hotplug_dev);Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb · line 410
Quoted 1 line
hw/pci/pcie.c +0
-0uint8_t *exp_cap = hotplug_pdev->config +hotplug_pdev-> > > exp.exp_cap; > > > > uint32_t sltcap = pci_get_word(exp_cap + PCI_EXP_SLTCAP); > > > > + uint32_t sltctl = pci_get_word(exp_cap + PCI_EXP_SLTCTL); > > > > > > > > /* Check if hot-plug is disabled on the slot */ > > > > if (dev->hotplugged && (sltcap &
PCI_EXP_SLTCAP_HPC) =Quoted 11 lines
= 0) { > > > > @@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb > (HotplugHandler > > *hotplug_dev, DeviceState *dev, > > > > return; > > > > } > > > > > > > > + if ((sltctl & PCI_EXP_SLTCTL_PIC) == > PCI_EXP_SLTCTL_PWR_IND_BLINK) > > { > > > > + error_setg(errp, "Hot-plug failed: %s is inPower
Quoted 6 lines
Quoted 6 lines
Transition", > > > + DEVICE(hotplug_pdev)->id); > > > + return; > > > + } > > > + > > >pcie_cap_slot_plug_common(PCI_DEVICE(hotplug_dev),
Quoted 5 lines
dev,
> errp);
> > > > }
> > >
> > > Probably the only way to handle for existing machinetypes.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
I agree
Quoted 1 line
For new ones, can't we queue it in host memory
somewhere?
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
I am not sure I understand what will be the flow.
- The user asks for a hotplug operation.
- QEMU deferred operation.
After that the operation may still fail, how would the userknow if
Quoted 14 lines
the > > operation > > succeeded or not? > > > How can it fail? It's just a button press ... > > > > Currently we have "Hotplug unsupported." > With this change we have "Guest/System not ready" Hotplug unsupported is not an error that can trigger withQuoted 8 lines
a well behaved management such as libvirt. > > > > > > > > I'm not actually convinced we can't do that even forexisting
Quoted 8 lines
machine > > types. > > > > > > Is a Guest visible change, I don't think we can do it. > > > > > > So I'm a bit hesitant to suggest going ahead with thiswithout
Quoted 3 lines
Quoted 1 line
Quoted 1 line
looking a bit closer at whether we can implement a
wait-for-ready in
> > qemu, rather than forcing every user of qemu (human ormachine)
Quoted 8 lines
to do
> > so.
> >
> >
> > While I agree it is a pain from the usability point of view,
hotplug
> operations
> > are allowed to fail. This is not more than a corner case,ensuring
Quoted 10 lines
the
> right
> > response (gracefully erroring out) may be enough.
> >
> > Thanks,
> > Marcel
> >
>
>
> I don't think they ever failed in the past so management isunlikely
Quoted 5 lines
Quoted 5 lines
to handle the failure by retrying ...
That would require some management handling, yes. But even without a "retry", failing is better than strange OS
behavior.
Quoted 2 lines
Quoted 2 lines
Trying a better alternative like deferring the operation for new
machines
Quoted 7 lines
Quoted 1 line
would make sense, however is out of the scope of this patch
Expand the scope please. The scope should be "solve a problem xx" not "solve a problem xx by doing abc".
Quoted 8 lines
Quoted 3 lines
The scope is detecting a hotplug error early instead passing to the Guest OS a hotplug operation that we know it will fail.
Right. After detecting just failing unconditionally it a bit too simplistic IMHO.
Simplistic does not mean wrong or incorrect.
I fail to see why it is not enough.
What QEMU can do better? Wait an unbounded time for the blinking to finish? What if we have a buggy guest with a kernel stuck in blinking? Is QEMU's responsibility to emulate the operator itself? Because the operator
is the one who is supposed to wait.
Thanks,
Marcel
[...]
Quoted 106 lines · On Fri, 23 Oct 2020 09:26:48 +0300 Marcel Apfelbaum…
Hi Michael,
On Thu, Oct 22, 2020 at 6:01 PM Michael S. Tsirkin <[email protected]> wrote:
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
Simplistic does not mean wrong or incorrect.
I fail to see why it is not enough.
Quoted 1 line
What QEMU can do better? Wait an unbounded time for the blinking to finish?
It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.
Quoted 10 lines
What if we have a buggy guest with a kernel stuck in blinking? Is QEMU's responsibility to emulate the operator itself? Because the operator
is the one who is supposed to wait.Thanks,
Marcel[...]
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Quoted 109 lines · David Gibson
Quoted 106 lines · On Fri, 23 Oct 2020 09:26:48 +0300 Marcel Apfelbaum…
Hi Michael,
On Thu, Oct 22, 2020 at 6:01 PM Michael S. Tsirkin <[email protected]> wrote:
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
Simplistic does not mean wrong or incorrect.
I fail to see why it is not enough.
Quoted 4 lines
Quoted 1 line
What QEMU can do better? Wait an unbounded time for the blinking to finish?
It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.
If it's helpful, I'd add a query to check state
so management can figure out why doesn't guest see device yet. But otherwise just buffer the request until such time as we can deliver it to guest ...
Quoted 15 lines
Quoted 10 lines
What if we have a buggy guest with a kernel stuck in blinking? Is QEMU's responsibility to emulate the operator itself? Because the operator
is the one who is supposed to wait.Thanks,
Marcel[...]
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Quoted 9 lines · On Tue, 27 Oct 2020 07:26:44 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 8 lines · David Gibson
Quoted 5 lines · On Fri, 23 Oct 2020 09:26:48 +0300 Marcel Apfelbaum…
Hi Michael,
On Thu, Oct 22, 2020 at 6:01 PM Michael S. Tsirkin <[email protected]> wrote:
[...]
[...]
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Simplistic does not mean wrong or incorrect.
I fail to see why it is not enough.What QEMU can do better? Wait an unbounded time for the blinking to finish?
It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.
racy, timeout is bound to break once it's in overcommited env.
Quoted 2 lines
If it's helpful, I'd add a query to check state
so management can figure out why doesn't guest see device yet.
that means mgmt would have to poll it and forward it to user somehow.
Quoted 2 lines
But otherwise just buffer the request until such time as we can deliver it to guest ...
I have more questions wrt the suggestion/workflow:
* at what place would you suggest buffering it?
* what would be the request in this case, i.e. create PCI device anyways and try to signal hotplug event later?
* what would baremethal do in such case?
* what to do in case guest is never ready, what user should do in such case?
* can be such device be removed?
not sure that all of this is worth of the effort and added complexity.
alternatively:
maybe ports can send QMP events about it's state changes, which end user would be able to see + error like in this patch.
On top of it, mgmt could build a better UIx, like retry/notify logic if that's what user really wishes for and configures (it would be up to user to define behaviour).
Quoted 18 lines
Quoted 15 lines
Quoted 10 lines
What if we have a buggy guest with a kernel stuck in blinking? Is QEMU's responsibility to emulate the operator itself? Because the operator
is the one who is supposed to wait.Thanks,
Marcel[...]
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Quoted 24 lines · Igor Mammedov
Quoted 9 lines · On Tue, 27 Oct 2020 07:26:44 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 8 lines · David Gibson
Quoted 5 lines · On Fri, 23 Oct 2020 09:26:48 +0300 Marcel Apfelbaum…
Hi Michael,
On Thu, Oct 22, 2020 at 6:01 PM Michael S. Tsirkin <[email protected]> wrote:
[...]
[...]
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Simplistic does not mean wrong or incorrect.
I fail to see why it is not enough.What QEMU can do better? Wait an unbounded time for the blinking to finish?
It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.
racy, timeout is bound to break once it's in overcommited env.
Quoted 2 lines
If it's helpful, I'd add a query to check state
so management can figure out why doesn't guest see device yet.
Quoted 7 lines
that means mgmt would have to poll it and forward it to user somehow.
Quoted 2 lines
But otherwise just buffer the request until such time as we can deliver it to guest ...
I have more questions wrt the suggestion/workflow:
* at what place would you suggest buffering it?
PCIESlot maybe?
Quoted 2 lines
* what would be the request in this case, i.e. create PCI device anyways and try to signal hotplug event later?
that was my idea, yes.
Quoted 1 line
* what would baremethal do in such case?
exactly the same, human would wait until blinking stops.
Quoted 1 line
* what to do in case guest is never ready, what user should do in such case?
As in guest is stuck? Do we care? It can't use the device.
Quoted 1 line
* can be such device be removed?
why not? device_del could complete immediately ...
Quoted 9 lines
not sure that all of this is worth of the effort and added complexity.
alternatively:
maybe ports can send QMP events about it's state changes, which end user would be able to see + error like in this patch.On top of it, mgmt could build a better UIx, like retry/notify logic if that's what user really wishes for and configures (it would be up to user to define behaviour).
I'd say let's get expected behaviour for existing commands first. We can add events and stuff on top.
Quoted 18 lines
Quoted 18 lines
Quoted 15 lines
Quoted 10 lines
What if we have a buggy guest with a kernel stuck in blinking? Is QEMU's responsibility to emulate the operator itself? Because the operator
is the one who is supposed to wait.Thanks,
Marcel[...]
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Quoted 15 lines · On Tue, 27 Oct 2020 09:02:06 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
On Tue, Oct 27, 2020 at 01:54:26PM +0100, Igor Mammedov wrote:
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]Quoted 2 lines
I have more questions wrt the suggestion/workflow:
* at what place would you suggest buffering it?
Quoted 7 lines
PCIESlot maybe?
Quoted 2 lines
* what would be the request in this case, i.e. create PCI device anyways and try to signal hotplug event later?
that was my idea, yes.
Actually, I don't think that will quite work. A whole chunk of the problem here is that because the device is realized, the guest sees it as part of its general scan *before* the hotplug event (ABP and PDC interrupts) appears. That makes the guest misinterpret the ABP as an *unplug* request.
So delaying the interrupt without delaying the realize (or at least filtering config space access to it based on.. something) will just trigger the same problem AFAICT.
Quoted 21 lines
Quoted 1 line
* what would baremethal do in such case?
exactly the same, human would wait until blinking stops.
Quoted 1 line
* what to do in case guest is never ready, what user should do in such case?
As in guest is stuck? Do we care? It can't use the device.
Quoted 1 line
* can be such device be removed?
why not?
device_delcould complete immediately ...[...]
I'd say let's get expected behaviour for existing commands first. We can add events and stuff on top.
[...]
[...]
[...]
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Quoted 9 lines · On Tue, 27 Oct 2020 13:54:26 +0100 Igor Mammedov <[email protected]> wrote:
On Tue, 27 Oct 2020 07:26:44 -0400
"Michael S. Tsirkin" <[email protected]> wrote:[...]
[...]
[...]
[...]
[...]Quoted 1 line
Quoted 1 line
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.
racy, timeout is bound to break once it's in overcommited env.
Hm. That's no less true at the management layer than it is at the qemu layer.
Quoted 4 lines
Quoted 2 lines
If it's helpful, I'd add a query to check state
so management can figure out why doesn't guest see device yet.that means mgmt would have to poll it and forward it to user somehow.
If that even makes sense. In the case of Kata, it's supposed to be autonomously creating the VM, so there's nothing meaningful it can forward to the user other than "failed to create the container because of some hotplug problem that means nothing to you".
Quoted 11 lines
[...]
I have more questions wrt the suggestion/workflow:
* at what place would you suggest buffering it?
* what would be the request in this case, i.e. create PCI device anyways and try to signal hotplug event later?
* what would baremethal do in such case?
* what to do in case guest is never ready, what user should do in such case?
* can be such device be removed?not sure that all of this is worth of the effort and added complexity.
Quoted 7 lines
alternatively:
maybe ports can send QMP events about it's state changes, which end user would be able to see + error like in this patch.On top of it, mgmt could build a better UIx, like retry/notify logic if that's what user really wishes for and configures (it would be up to user to define behaviour).
That kind of makes sense if the user is explicitly requesting hotplugs, but that's not necessarily the case.
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Quoted 12 lines · On Wed, 28 Oct 2020 14:31:35 +1100 David Gibson <[email protected]> wrote:
Quoted 9 lines · On Tue, 27 Oct 2020 13:54:26 +0100 Igor Mammedov <[email protected]> wrote:
On Tue, 27 Oct 2020 07:26:44 -0400
"Michael S. Tsirkin" <[email protected]> wrote:[...]
[...]
[...]
[...]
[...]Quoted 1 line
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.
racy, timeout is bound to break once it's in overcommited env.
Hm. That's no less true at the management layer than it is at the qemu layer.
true, but it's user policy which is defined by user not by QEMU.
Quoted 26 lines
Quoted 5 lines
Quoted 3 lines
If it's helpful, I'd add a query to check state
so management can figure out why doesn't guest see device yet.that means mgmt would have to poll it and forward it to user somehow.
If that even makes sense. In the case of Kata, it's supposed to be autonomously creating the VM, so there's nothing meaningful it can forward to the user other than "failed to create the container because of some hotplug problem that means nothing to you".
Quoted 15 lines
[...]
I have more questions wrt the suggestion/workflow:
* at what place would you suggest buffering it?
* what would be the request in this case, i.e. create PCI device anyways and try to signal hotplug event later?
* what would baremethal do in such case?
* what to do in case guest is never ready, what user should do in such case?
* can be such device be removed?not sure that all of this is worth of the effort and added complexity.
alternatively:
maybe ports can send QMP events about it's state changes, which end user would be able to see + error like in this patch.
Quoted 6 lines
Quoted 3 lines
On top of it, mgmt could build a better UIx, like retry/notify logic if that's what user really wishes for and configures (it would be up to user to define behaviour).
That kind of makes sense if the user is explicitly requesting hotplugs, but that's not necessarily the case.
user doesn't have to be a human, it could be some mgmt layer that would automate retry logic, depending on what actually user needs for particular task (i.e. fail immediately, retry N time then fail, retry with time out - then fail, don't care - succeed, ...). The point is for QEMU to provide means for mgmt to implement whatever policy user would need.
PS:
but then, I know close to nothing about PCI, so all of above might be nonsense.
Quoted 53 lines · Igor Mammedov
Quoted 12 lines · On Wed, 28 Oct 2020 14:31:35 +1100 David Gibson <[email protected]> wrote:
Quoted 9 lines · On Tue, 27 Oct 2020 13:54:26 +0100 Igor Mammedov <[email protected]> wrote:
On Tue, 27 Oct 2020 07:26:44 -0400
"Michael S. Tsirkin" <[email protected]> wrote:[...]
[...]
[...]
[...]
[...]Quoted 1 line
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
It certainly shouldn't wait an unbounded time. But a wait with timeout seems worth investigating to me.
racy, timeout is bound to break once it's in overcommited env.
Hm. That's no less true at the management layer than it is at the qemu layer.
true, but it's user policy which is defined by user not by QEMU.
Quoted 30 lines
Quoted 5 lines
Quoted 3 lines
If it's helpful, I'd add a query to check state
so management can figure out why doesn't guest see device yet.that means mgmt would have to poll it and forward it to user somehow.
If that even makes sense. In the case of Kata, it's supposed to be autonomously creating the VM, so there's nothing meaningful it can forward to the user other than "failed to create the container because of some hotplug problem that means nothing to you".
Quoted 18 lines
[...]
I have more questions wrt the suggestion/workflow:
* at what place would you suggest buffering it?
* what would be the request in this case, i.e. create PCI device anyways and try to signal hotplug event later?
* what would baremethal do in such case?
* what to do in case guest is never ready, what user should do in such case?
* can be such device be removed?not sure that all of this is worth of the effort and added complexity.
alternatively:
maybe ports can send QMP events about it's state changes, which end user would be able to see + error like in this patch.On top of it, mgmt could build a better UIx, like retry/notify logic if that's what user really wishes for and configures (it would be up to user to define behaviour).
Quoted 7 lines
Quoted 2 lines
That kind of makes sense if the user is explicitly requesting hotplugs, but that's not necessarily the case.
user doesn't have to be a human, it could be some mgmt layer that would automate retry logic, depending on what actually user needs for particular task (i.e. fail immediately, retry N time then fail, retry with time out - then fail, don't care - succeed, ...). The point is for QEMU to provide means for mgmt to implement whatever policy user would need.
We are not coming up with new APIs here. Let's make existing ones work reliably first. We can talk about a flag where it fails instead of deferring hotplug, separately.
Quoted 2 lines
PS:
but then, I know close to nothing about PCI, so all of above might be nonsense.
Quoted 193 lines · Marcel Apfelbaum
Hi Michael,
On Thu, Oct 22, 2020 at 6:01 PM Michael S. Tsirkin <[email protected]> wrote:
On Thu, Oct 22, 2020 at 05:50:51PM +0300, Marcel Apfelbaum wrote:
>
>
> On Thu, Oct 22, 2020 at 5:33 PM Michael S. Tsirkin <[email protected]> wrote:
>
>Â Â Â On Thu, Oct 22, 2020 at 05:10:43PM +0300, Marcel Apfelbaum wrote:
>Â Â Â >
>Â Â Â >
>Â Â Â > On Thu, Oct 22, 2020 at 5:01 PM Michael S. Tsirkin <[email protected]> >Â Â Â wrote:
>Â Â Â >
>Â Â Â >Â Â Â On Thu, Oct 22, 2020 at 04:55:10PM +0300, Marcel Apfelbaum wrote:
>Â Â Â >Â Â Â > Hi David, Michael,
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â > On Thu, Oct 22, 2020 at 3:56 PM David Gibson < [email protected]>
>Â Â Â wrote:
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â >Â Â Â On Thu, 22 Oct 2020 08:06:55 -0400
>Â Â Â >Â Â Â >Â Â Â "Michael S. Tsirkin" <[email protected]> wrote:
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â >Â Â Â > On Thu, Oct 22, 2020 at 02:40:26PM +0300, Marcel Apfelbaum
>Â Â Â wrote:
>Â Â Â >Â Â Â >Â Â Â > > From: Marcel Apfelbaum <[email protected]> >Â Â Â >Â Â Â >Â Â Â > >
>Â Â Â >Â Â Â >Â Â Â > > During PCIe Root Port's transition from Power-Off to >Â Â Â Power-ON (or
>Â Â Â >Â Â Â >Â Â Â vice-versa)
>Â Â Â >Â Â Â >Â Â Â > > the "Slot Control Register" has the "Power Indicator >Â Â Â Control"
>Â Â Â >Â Â Â >Â Â Â > > set to "Blinking" expressing a "power transition" mode.
>Â Â Â >Â Â Â >Â Â Â > >
>Â Â Â >Â Â Â >Â Â Â > > Any hotplug operation during the "power transition" mode is
>Â Â Â not
>Â Â Â >Â Â Â >Â Â Â permitted
>Â Â Â >Â Â Â >Â Â Â > > or at least not expected by the Guest OS leading to strange
>Â Â Â >Â Â Â failures.
>Â Â Â >Â Â Â >Â Â Â > >
>Â Â Â >Â Â Â >Â Â Â > > Detect and refuse hotplug operations in such case. >Â Â Â >Â Â Â >Â Â Â > >
>Â Â Â >Â Â Â >Â Â Â > > Signed-off-by: Marcel Apfelbaum < [email protected]
>Â Â Â >
>Â Â Â >Â Â Â >Â Â Â > > ---
>Â Â Â >Â Â Â >Â Â Â > >Âhw/pci/pcie.c| 7 +++++++
>Â Â Â >Â Â Â >Â Â Â > >Â 1 file changed, 7 insertions(+)
>Â Â Â >Â Â Â >Â Â Â > >
>Â Â Â >Â Â Â >Â Â Â > > diff --gita/hw/pci/pcie.cb/hw/pci/pcie.c>Â Â Â >Â Â Â >Â Â Â > > index5b48bae0f6..2fe5c1473f100644
>Â Â Â >Â Â Â >Â Â Â > > ---a/hw/pci/pcie.c
>Â Â Â >Â Â Â >Â Â Â > > +++b/hw/pci/pcie.c
>Â Â Â >Â Â Â >Â Â Â > > @@ -410,6 +410,7 @@ voidpcie_cap_slot_pre_plug_cb>Â Â Â (HotplugHandler
>Â Â Â >Â Â Â >Â Â Â *hotplug_dev, DeviceState *dev,
>Â Â Â >Â Â Â >Â Â Â > >Â Â Â PCIDevice*hotplug_pdev=PCI_DEVICE(hotplug_dev);
>Â Â Â >Â Â Â >Â Â Â > >Â Â Âuint8_t*exp_cap=hotplug_pdev->config+ >Â Â Â hotplug_pdev->
>Â Â Â >Â Â Â >Â Â Â exp.exp_cap;
>Â Â Â >Â Â Â >Â Â Â > >Â Â Âuint32_tsltcap =pci_get_word(exp_cap + >Â Â Â PCI_EXP_SLTCAP);
>Â Â Â >Â Â Â >Â Â Â > > +Â Âuint32_tsltctl =pci_get_word(exp_cap + >Â Â Â PCI_EXP_SLTCTL);
>Â Â Â >Â Â Â >Â Â Â > >Â
>Â Â Â >Â Â Â >Â Â Â > >Â Â Â /* Check if hot-plug is disabled on the slot */ >Â Â Â >Â Â Â >Â Â Â > >Â Â Â if (dev->hotplugged&& (sltcap &PCI_EXP_SLTCAP_HPC) =
>Â Â Â = 0) {
>Â Â Â >Â Â Â >Â Â Â > > @@ -418,6 +419,12 @@ voidpcie_cap_slot_pre_plug_cb>Â Â Â >Â Â Â (HotplugHandler
>Â Â Â >Â Â Â >Â Â Â *hotplug_dev, DeviceState *dev,
>Â Â Â >Â Â Â >Â Â Â > >Â Â Â Â Â return;
>Â Â Â >Â Â Â >Â Â Â > >Â Â Â }
>Â Â Â >Â Â Â >Â Â Â > >Â
>Â Â Â >Â Â Â >Â Â Â > > +Â Â if ((sltctl &PCI_EXP_SLTCTL_PIC) == >Â Â Â >Â Â Â PCI_EXP_SLTCTL_PWR_IND_BLINK)
>Â Â Â >Â Â Â >Â Â Â {
>Â Â Â >Â Â Â >Â Â Â > > +Â Â Â Â error_setg(errp, "Hot-plug failed: %s is in Power
>Â Â Â >Â Â Â Transition",
>Â Â Â >Â Â Â >Â Â Â > > +Â Â Â Â Â Â Â Â Â Â DEVICE(hotplug_pdev)->id); >Â Â Â >Â Â Â >Â Â Â > > +Â Â Â Â return;
>Â Â Â >Â Â Â >Â Â Â > > +Â Â }
>Â Â Â >Â Â Â >Â Â Â > > +
>Â Â Â >Â Â Â >Â Â Â > >Â Â Â pcie_cap_slot_plug_common(PCI_DEVICE (hotplug_dev),
>Â Â Â dev,
>Â Â Â >Â Â Â errp);
>Â Â Â >Â Â Â >Â Â Â > >Â }Â
>Â Â Â >Â Â Â >Â Â Â >
>Â Â Â >Â Â Â >Â Â Â > Probably the only way to handle for existing machine types.
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â > I agree
>Â Â Â >Â Â Â > Â
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â >Â Â Â > For new ones, can't we queue it in host memory somewhere?
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â > I am not sure I understand what will be the flow.
>Â Â Â >Â Â Â > Â - The user asks for a hotplug operation.
>Â Â Â >Â Â Â > Â -Â QEMU deferred operation.
>Â Â Â >Â Â Â > After that the operation may still fail, how would the user know if
>Â Â Â the
>Â Â Â >Â Â Â > operation
>Â Â Â >Â Â Â > succeeded or not?
>Â Â Â >
>Â Â Â >
>Â Â Â >Â Â Â How can it fail? It's just a button press ...
>Â Â Â >
>Â Â Â >
>Â Â Â >
>Â Â Â > Currently we have "Hotplug unsupported."
>Â Â Â > With this change we have "Guest/System not ready"
>
>
>Â Â Â Hotplug unsupported is not an error that can trigger with
>Â Â Â a well behaved management such as libvirt.
>
>
>Â Â Â > Â
>Â Â Â >
>Â Â Â >Â Â Â > Â
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â >Â Â Â I'm not actually convinced we can't do that even for existing
>Â Â Â machine
>Â Â Â >Â Â Â >Â Â Â types.Â
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â > Is a Guest visible change, I don't think we can do it. >Â Â Â >Â Â Â > Â
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â >Â Â Â So I'm a bit hesitant to suggest going ahead with this without
>Â Â Â >Â Â Â >Â Â Â looking a bit closer at whether we can implement a >Â Â Â wait-for-ready in
>Â Â Â >Â Â Â >Â Â Â qemu, rather than forcing every user of qemu (human or machine)
>Â Â Â to do
>Â Â Â >Â Â Â >Â Â Â so.
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â > While I agree it is a pain from the usability point of view, >Â Â Â hotplug
>Â Â Â >Â Â Â operations
>Â Â Â >Â Â Â > are allowed to fail. This is not more than a corner case, ensuring
>Â Â Â the
>Â Â Â >Â Â Â right
>Â Â Â >Â Â Â > response (gracefully erroring out) may be enough.
>Â Â Â >Â Â Â >
>Â Â Â >Â Â Â > Thanks,
>Â Â Â >Â Â Â > Marcel
>Â Â Â >Â Â Â >
>Â Â Â >
>Â Â Â >
>Â Â Â >Â Â Â I don't think they ever failed in the past so management is unlikely
>Â Â Â >Â Â Â to handle the failure by retrying ...
>Â Â Â >
>Â Â Â >
>Â Â Â > That would require some management handling, yes.
>   > But even without a "retry", failing is better than strange OS behavior.
>Â Â Â >
>Â Â Â > Trying a better alternative like deferring the operation for new machines
>Â Â Â > would make sense, however is out of the scope of this patch
>
>Â Â Â Expand the scope please. The scope should be "solve a problem xx" not >Â Â Â "solve a problem xx by doing abc".
>
>
>
> The scope is detecting a hotplug error early instead
> passing to the Guest OS a hotplug operation that we know it will fail. >
Quoted 7 lines
Right. After detecting just failing unconditionally it a bit too simplistic IMHO.
Simplistic does not mean wrong or incorrect.
I fail to see why it is not enough.
The failure patch requires management to retry later.
A more elaborate scheme will fix the bug without need for management changes.
Quoted 2 lines
What QEMU can do better? Wait an unbounded time for the blinking to finish? What if we have a buggy guest with a kernel stuck in blinking?
Then it won't see the new device ever but does it even matter? It's stuck ... I'd ack adding a query command to see what is going on with the device. Can be generic, implementable on top of ACPI too.
Quoted 2 lines
Is QEMU's responsibility to emulate the operator itself? Because the operator is the one who is supposed to wait.
I think these details are immaterial for users. They don't read pci spec.
Quoted 5 lines
Thanks,
Marcel[...]Â
Quoted 38 lines · On Thu, 22 Oct 2020 16:55:10 +0300 Marcel Apfelbaum…
Hi David, Michael,
On Thu, Oct 22, 2020 at 3:56 PM David Gibson <[email protected]> wrote:
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]
[...]Quoted 3 lines
Quoted 2 lines
Probably the only way to handle for existing machine types.
I agree
Quoted 3 lines
Quoted 1 line
For new ones, can't we queue it in host memory somewhere?
I am not sure I understand what will be the flow.
- The user asks for a hotplug operation.
- QEMU deferred operation.
After that the operation may still fail, how would the user know if the operation
succeeded or not?
Quoted 5 lines
Quoted 2 lines
I'm not actually convinced we can't do that even for existing machine types.
Is a Guest visible change, I don't think we can do it.
How is it a guest visible change?
Quoted 16 lines
Quoted 4 lines
So I'm a bit hesitant to suggest going ahead with this without looking a bit closer at whether we can implement a wait-for-ready in qemu, rather than forcing every user of qemu (human or machine) to do so.
While I agree it is a pain from the usability point of view, hotplug operations
are allowed to fail. This is not more than a corner case, ensuring the right response (gracefully erroring out) may be enough.Thanks,
Marcel[...]
Signature · 2 lines
David Gibson <[email protected]> Principal Software Engineer, Virtualization, Red Hat
Quoted 6 lines · Marcel Apfelbaum
From: Marcel Apfelbaum <[email protected]>
During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) the "Slot Control Register" has the "Power Indicator Control"
set to "Blinking" expressing a "power transition" mode.
Quoted 6 lines
Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.
Detect and refuse hotplug operations in such case.
Signed-off-by Marcel Apfelbaum <[email protected]>
Going back to this I have another question: could we get a bit more detail on when do we get into this situation? When does guest start blinking the indicator without us first starting a hotplug operation?
Quoted 31 lines
--- hw/pci/pcie.c | 7 +++++++ 1 file changed, 7 insertions(+)hw/pci/pcie.c +7
-0@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,PCIDevice *hotplug_pdev =PCI_DEVICE (hotplug_dev);uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;uint32_t sltcap =pci_get_word (exp_cap +PCI_EXP_SLTCAP ); +uint32_t sltctl =pci_get_word (exp_cap +PCI_EXP_SLTCTL );/* Check if hot-plug is disabled on the slot */ if (dev->hotplugged && (sltcap &PCI_EXP_SLTCAP_HPC ) ==0 ) {@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,return ; } +if ((sltctl &PCI_EXP_SLTCTL_PIC ) ==PCI_EXP_SLTCTL_PWR_IND_BLINK ) { +error_setg (errp,"Hot-plug failed: %s is in Power Transition" , +DEVICE (hotplug_pdev)->id); +return ; + } +pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); }Signature · 1 line
2.17.2
Hi Michael,
Quoted 13 lines · Michael S. Tsirkin
Quoted 3 lines · Marcel Apfelbaum
From: Marcel Apfelbaum <[email protected]>
During PCIe Root Port's transition from Power-Off to Power-ON (or
vice-versa)
Quoted 8 lines
the "Slot Control Register" has the "Power Indicator Control" set to "Blinking" expressing a "power transition" mode.
Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.
Detect and refuse hotplug operations in such case.
Quoted 8 lines
Quoted 1 line
Signed-off-by Marcel Apfelbaum <[email protected]>Going back to this I have another question: could we get a bit more detail on when do we get into this situation? When does guest start blinking the indicator without us first starting a hotplug operation?
While David has more insight on the kernel behavior during the mentioned issue,
it seems the PCI subsystem sets the Power Indicator
to "blinking" as part of the init sequence of the PCIe Root Port. The kernel will turn it off as soon as it finishes the init sequence.
A hotplug operation during the "init" sequence will surprise the Guest OS. I'll let David to give us more info.
Thank you,
Marcel
Quoted 35 lines
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler
Quoted 9 lines
--- hw/pci/pcie.c | 7 +++++++ 1 file changed, 7 insertions(+)hw/pci/pcie.c +0
-0@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler
*hotplug_dev, DeviceState *dev,hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler
Quoted 8 lines
hw/pci/pcie.c +1
-0PCIDevice *hotplug_pdev =PCI_DEVICE (hotplug_dev);uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;uint32_t sltcap =pci_get_word (exp_cap +PCI_EXP_SLTCAP ); +uint32_t sltctl =pci_get_word (exp_cap +PCI_EXP_SLTCTL );/* Check if hot-plug is disabled on the slot */ if (dev->hotplugged && (sltcap &PCI_EXP_SLTCAP_HPC ) ==0 ) {@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler
*hotplug_dev, DeviceState *dev,Quoted 14 lines
hw/pci/pcie.c +6
-1return ; } +if ((sltctl &PCI_EXP_SLTCTL_PIC ) ==PCI_EXP_SLTCTL_PWR_IND_BLINK ) { +error_setg (errp,"Hot-plug failed: %s is in Power Transition" , +DEVICE (hotplug_pdev)->id); +return ; + } +pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); } --2.17.2
Quoted 30 lines · Marcel Apfelbaum
From: Marcel Apfelbaum <[email protected]>
During PCIe Root Port's transition from Power-Off to Power-ON (or vice-versa) the "Slot Control Register" has the "Power Indicator Control"
set to "Blinking" expressing a "power transition" mode.Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.
Detect and refuse hotplug operations in such case.
Signed-off-by Marcel Apfelbaum <[email protected]>--- hw/pci/pcie.c | 7 +++++++ 1 file changed, 7 insertions(+)hw/pci/pcie.c +1
-0@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,PCIDevice *hotplug_pdev =PCI_DEVICE (hotplug_dev);uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;uint32_t sltcap =pci_get_word (exp_cap +PCI_EXP_SLTCAP ); +uint32_t sltctl =pci_get_word (exp_cap +PCI_EXP_SLTCTL );/* Check if hot-plug is disabled on the slot */ if (dev->hotplugged && (sltcap &PCI_EXP_SLTCAP_HPC ) ==0 ) {@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,return ;
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 429
Quoted 10 lines
hw/pci/pcie.c +6
-0} +if ((sltctl &PCI_EXP_SLTCTL_PIC ) ==PCI_EXP_SLTCTL_PWR_IND_BLINK ) { +error_setg (errp,"Hot-plug failed: %s is in Power Transition" , +DEVICE (hotplug_pdev)->id); +return ; + } +pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); }
I wonder if hw/pci/shpc.c is free from this issue?
Roman.
Hi Roman,
Quoted 45 lines · Roman Kagan
Quoted 3 lines · Marcel Apfelbaum
From: Marcel Apfelbaum <[email protected]>
During PCIe Root Port's transition from Power-Off to Power-ON (or
vice-versa)
hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler
Quoted 18 lines
the "Slot Control Register" has the "Power Indicator Control" set to "Blinking" expressing a "power transition" mode.
Any hotplug operation during the "power transition" mode is not permitted or at least not expected by the Guest OS leading to strange failures.
Detect and refuse hotplug operations in such case.
Signed-off-by Marcel Apfelbaum <[email protected]>--- hw/pci/pcie.c | 7 +++++++ 1 file changed, 7 insertions(+)hw/pci/pcie.c +0
-0@@ -410,6 +410,7 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler
*hotplug_dev, DeviceState *dev,hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler
Quoted 8 lines
hw/pci/pcie.c +1
-0PCIDevice *hotplug_pdev =PCI_DEVICE (hotplug_dev);uint8_t *exp_cap = hotplug_pdev->config + hotplug_pdev->exp.exp_cap;uint32_t sltcap =pci_get_word (exp_cap +PCI_EXP_SLTCAP ); +uint32_t sltctl =pci_get_word (exp_cap +PCI_EXP_SLTCTL );/* Check if hot-plug is disabled on the slot */ if (dev->hotplugged && (sltcap &PCI_EXP_SLTCAP_HPC ) ==0 ) {@@ -418,6 +419,12 @@ void pcie_cap_slot_pre_plug_cb(HotplugHandler
*hotplug_dev, DeviceState *dev,hw/pci/pcie.c void pcie_cap_slot_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, · line 429
Quoted 11 lines
hw/pci/pcie.c +6
-0return ; } +if ((sltctl &PCI_EXP_SLTCTL_PIC ) ==PCI_EXP_SLTCTL_PWR_IND_BLINK ) { +error_setg (errp,"Hot-plug failed: %s is in Power Transition" , +DEVICE (hotplug_pdev)->id); +return ; + } +pcie_cap_slot_plug_common (PCI_DEVICE (hotplug_dev), dev, errp); }
Quoted 3 lines
I wonder if
hw/pci/shpc.cis free from this issue?
I did not encounter the issue with shpc.
However, the SHPC spec describes the Power Indicator behavior for hotplug only leaving the booting sequence out of it:
"Although the Standard Usage Model presented in Section 2.2.1.2
requires a specific
relationship between the Power Indicator and the state of the slot,
this relationship is
enforced by software, not by the SHPC."
It looks like it depends on the software implementation.
Thanks,
Marcel
Quoted 2 lines
Roman.