Skip to contentSkip to conversation

Xen/pciback: Implement PCI slot or bus reset with 'do_flr' SysFS attribute

PATCH In review Govinda Tatti 2 participants 20 messages Nov 8, 2017 – Dec 4, 2017 Xen hypervisor interface

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 Xen/pciback: Implement PCI slot or bus reset with 'do_flr' SysFS attribute

The life-cycle of a PCI device in Xen pciback is complex and is constrained by the generic PCI locking mechanism.

- It starts with the device being bound to us, for which we do a function reset (done via SysFS so the PCI lock is held).
- If the device is unbound from us, we also do a function reset (done via SysFS so the PCI lock is held).
- If the device is un-assigned from a guest - we do a function reset (no PCI lock is held).

All reset operations are done on the individual PCI function level (so bus:device:function).

The reset for an individual PCI function means device must support FLR (PCIe or AF), PM reset on D3hot->D0 device specific reset, or a secondary bus reset for a singleton device on a bus but FLR does not have widespread support or it is not reliable in some cases. So, we need to provide an alternate mechanism to users to perform a slot or bus level reset.

Currently, a slot or bus reset is not exposed in SysFS as there is no good way of exposing a bus topology there. This is due to the complexity - we MUST know that the different functions of a PCIe device are not in use by other drivers, or if they are in use (say one of them is assigned to a guest and the other is idle) - it is still OK to reset the slot (assuming both of them are owned by Xen pciback).

This patch does that by doing a slot or bus reset (if slot not supported) if all of the functions of a PCIe device belong to Xen PCIback.

Due to the complexity with the PCI lock we cannot do the reset when a device is bound ('echo $BDF > bind') or when unbound ('echo $BDF > unbind') as the pci_[slot|bus]_reset also takes the same lock resulting in a dead-lock.

Putting the reset function in a work-queue or thread won't work either - as we have to do the reset function outside the 'unbind' context (it holds the PCI lock). But once you 'unbind' a device the device is no longer under the ownership of Xen pciback and the pci_set_drvdata has been reset, so we cannot use a thread for this.

Instead of doing all this complex dance, we depend on the tool-stack doing the right thing. As such, we implement the 'do_flr' SysFS attribute which 'xl' uses when a device is detached or attached from/to a guest. It bypasses the need to worry about the PCI lock.

To not inadvertently do a bus reset that would affect devices that are in use by other drivers (other than Xen pciback) prior to the reset, we check that all of the devices under the bridge are owned by Xen pciback. If they are not, we refrain from executing the bus (or slot) reset.

---
 Documentation/ABI/testing/sysfs-driver-pciback |  12 +++
 drivers/xen/xen-pciback/pci_stub.c             | 119 +++++++++++++++++++++++++
 2 files changed, 131 insertions(+)
Documentation/ABI/testing/sysfs-driver-pciback +12 -0
index 6a733bf..ccf7dc0 100644
@@ -11,3 +11,15 @@ Description:
                 #echo 00:19.0-E0:2:FF > /sys/bus/pci/drivers/pciback/quirks
                 will allow the guest to read and write to the configuration
                 register 0x0E.
+
+What:           /sys/bus/pci/drivers/pciback/do_flr
+Date:           Nov 2017
+KernelVersion:  4.15
+Contact:        [email protected]
+Description:
+                An option to perform a slot or bus reset when a PCI device
+		is owned by Xen PCI backend. Writing a string of DDDD:BB:DD.F
+		will cause the pciback driver to perform a slot or bus reset
+		if the device supports it. It also checks to make sure that
+		all of the devices under the bridge are owned by Xen PCI
+		backend.
drivers/xen/xen-pciback/pci_stub.c +119 -0

143 lines not on this page · raw message

Signature · 7 lines
2.9.5


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
#1 lore raw

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 248

Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 09.11.17 at 00:06, <[email protected]> wrote:
drivers/xen/xen-pciback/pci_stub.c +2 -0
@@ -244,6 +244,91 @@ struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev,
 	return found_dev;
 }
 
+struct pcistub_args {
+	struct pci_dev *dev;

Please don't ignore prior review comments: Either carry out what was requested, or explain why the request can't be fulfilled. You saying "This field will point to first device that is not owned by pcistub" to Roger's request to make this a pointer to const is not a valid reason to not do the adjustment; in fact your reply is entirely unrelated to the request.

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 256

Quoted 5 lines
drivers/xen/xen-pciback/pci_stub.c +5 -0
+static int pcistub_search_dev(struct pci_dev *dev, void *data)
+{
+	struct pcistub_device *psdev;
+	struct pcistub_args *arg = data;
+	bool found_dev = false;

Purely cosmetical, but anyway: Why not just "found"? What else could be (not) found here other than the device in question?

Quoted 9 lines
drivers/xen/xen-pciback/pci_stub.c +9 -0
+	unsigned long flags;
+
+	spin_lock_irqsave(&pcistub_devices_lock, flags);
+
+	list_for_each_entry(psdev, &pcistub_devices, dev_list) {
+		if (psdev->dev == dev) {
+			found_dev = true;
+			arg->dcount++;
+			break;

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 275

Quoted 10 lines
drivers/xen/xen-pciback/pci_stub.c +10 -0
+		}
+	}
+
+	spin_unlock_irqrestore(&pcistub_devices_lock, flags);
+
+	/* Device not owned by pcistub, someone owns it. Abort the walk */
+	if (!found_dev)
+		arg->dev = dev;
+
+	return found_dev ? 0 : 1;

Despite the function needing to return int, this can be simplified to "return !found_dev". I'd also like to note that the part of the earlier comment related to this is sort of disconnected. How about

	/* Device not owned by pcistub, someone owns it. Abort the walk */
	if (!found_dev) {
		arg->dev = dev;
		return 1;
	}

	return 0;

And finally - I don't think the comment is entirely correct - the device not being owned by pciback doesn't necessarily mean it's owned by another driver. It could as well be unowned.

Quoted 14 lines
drivers/xen/xen-pciback/pci_stub.c +14 -0
+static int pcistub_reset_dev(struct pci_dev *dev)
+{
+	struct xen_pcibk_dev_data *dev_data;
+	bool slot = false, bus = false;
+	struct pcistub_args arg = {};
+
+	if (!dev)
+		return -EINVAL;
+
+	dev_dbg(&dev->dev, "[%s]\n", __func__);
+
+	if (!pci_probe_reset_slot(dev->slot))
+		slot = true;
+	else if ((!pci_probe_reset_bus(dev->bus)) &&

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 301

Quoted 10 lines
drivers/xen/xen-pciback/pci_stub.c +10 -0
+		 (!pci_is_root_bus(dev->bus)))
+		bus = true;
+
+	if (!bus && !slot)
+		return -EOPNOTSUPP;
+
+	/*
+	 * Make sure all devices on this bus are owned by the
+	 * PCI backend so that we can safely reset the whole bus.
+	 */

Is that really the case when you mean to do a slot reset? It was for a reason that I had asked about a missing "else" in v1 review, rather than questioning the conditional around the logic.

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 306

Quoted 5 lines
drivers/xen/xen-pciback/pci_stub.c +5 -0
+	pci_walk_bus(dev->bus, pcistub_search_dev, &arg);
+
+	/* All devices under the bus should be part of pcistub! */
+	if (arg.dev) {
+		dev_err(&dev->dev, "%s device on bus 0x%x is not owned by pcistub\n",

%#x

Yet then, thinking about what would be useful information should the situation really arise, I'm not convinced printing a bare bus number here is useful either. Especially for the case of multiple parallel requests you want to make it possible to match each message to the original request (guest start or whatever). Hence I think you want something like

"%s on the same bus as %s is not owned by " DRV_NAME "\n"

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 313

Quoted 7 lines
drivers/xen/xen-pciback/pci_stub.c +7 -0
+			pci_name(arg.dev), dev->bus->number);
+
+		return -EBUSY;
+	}
+
+	dev_dbg(&dev->dev, "pcistub owns %d devices on bus 0x%x\n",
+		arg.dcount, dev->bus->number);

While here the original device is perhaps not necessary to print, the bare bus number doesn't carry enough information: You'll want to prefix it by the segment number. Plus you'll want to use canonical formatting (ssss:bb), so one can get matches when suitably grep-ing the log. Perhaps bus->name is what you're after.

Jan

#2 lore raw

Jan,

Please see below for my comments.

Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 09.11.17 at 00:06, <[email protected]> wrote:
drivers/xen/xen-pciback/pci_stub.c +0 -0
@@ -244,6 +244,91 @@ struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev,
  	return found_dev;
  }
 
Quoted 8 lines

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 248

Quoted 2 lines
drivers/xen/xen-pciback/pci_stub.c +2 -0
+struct pcistub_args {
+	struct pci_dev *dev;

Please don't ignore prior review comments: Either carry out what was requested, or explain why the request can't be fulfilled. You saying "This field will point to first device that is not owned by pcistub" to Roger's request to make this a pointer to const is not a valid reason to not do the adjustment; in fact your reply is entirely unrelated to the request.

We can use "const" with above field since we will set this field only once. I will change it.

Quoted 8 lines

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 256

Quoted 6 lines
drivers/xen/xen-pciback/pci_stub.c +5 -0
+static int pcistub_search_dev(struct pci_dev *dev, void *data)
+{
+	struct pcistub_device *psdev;
+	struct pcistub_args *arg = data;
+	bool found_dev = false;

Purely cosmetical, but anyway: Why not just "found"? What else could be (not) found here other than the device in question?

Sure, I will change it to "found".

Quoted 24 lines
Quoted 10 lines
drivers/xen/xen-pciback/pci_stub.c +9 -0
+	unsigned long flags;
+
+	spin_lock_irqsave(&pcistub_devices_lock, flags);
+
+	list_for_each_entry(psdev, &pcistub_devices, dev_list) {
+		if (psdev->dev == dev) {
+			found_dev = true;
+			arg->dcount++;
+			break;

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 275

Quoted 10 lines
drivers/xen/xen-pciback/pci_stub.c +10 -0
+		}
+	}
+
+	spin_unlock_irqrestore(&pcistub_devices_lock, flags);
+
+	/* Device not owned by pcistub, someone owns it. Abort the walk */
+	if (!found_dev)
+		arg->dev = dev;
+
+	return found_dev ? 0 : 1;

Despite the function needing to return int, this can be simplified to "return !found_dev". I'd also like to note that the part of the earlier comment related to this is sort of disconnected. How about

Quoted 7 lines
	/* Device not owned by pcistub, someone owns it. Abort the walk */
	if (!found_dev) {
		arg->dev = dev;
		return 1;
	}

	return 0;

Fine with me.

Quoted 4 lines

And finally - I don't think the comment is entirely correct - the device not being owned by pciback doesn't necessarily mean it's owned by another driver. It could as well be unowned.

OK. I will modify this comment as " Device not owned by pcistub. Abort the walk".

Quoted 20 lines
Quoted 20 lines
drivers/xen/xen-pciback/pci_stub.c +19 -0
+static int pcistub_reset_dev(struct pci_dev *dev)
+{
+	struct xen_pcibk_dev_data *dev_data;
+	bool slot = false, bus = false;
+	struct pcistub_args arg = {};
+
+	if (!dev)
+		return -EINVAL;
+
+	dev_dbg(&dev->dev, "[%s]\n", __func__);
+
+	if (!pci_probe_reset_slot(dev->slot))
+		slot = true;
+	else if ((!pci_probe_reset_bus(dev->bus)) &&
+		 (!pci_is_root_bus(dev->bus)))
+		bus = true;
+
+	if (!bus && !slot)
+		return -EOPNOTSUPP;
Quoted 8 lines

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 301

Quoted 5 lines
drivers/xen/xen-pciback/pci_stub.c +5 -0
+
+	/*
+	 * Make sure all devices on this bus are owned by the
+	 * PCI backend so that we can safely reset the whole bus.
+	 */

Is that really the case when you mean to do a slot reset? It was for a reason that I had asked about a missing "else" in v1 review, rather than questioning the conditional around the logic.

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint. The connected card/endpoint can be multi-function device. So, same walk-through and checking is needed irrespective of type of reset being used.

Quoted 8 lines

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 306

Quoted 6 lines
drivers/xen/xen-pciback/pci_stub.c +5 -0
+	pci_walk_bus(dev->bus, pcistub_search_dev, &arg);
+
+	/* All devices under the bus should be part of pcistub! */
+	if (arg.dev) {
+		dev_err(&dev->dev, "%s device on bus 0x%x is not owned by pcistub\n",

%#x

Quoted 8 lines

Yet then, thinking about what would be useful information should the situation really arise, I'm not convinced printing a bare bus number here is useful either. Especially for the case of multiple parallel requests you want to make it possible to match each message to the original request (guest start or whatever). Hence I think you want something like

"%s on the same bus as %s is not owned by " DRV_NAME "\n"

Sure, I will make this change.

Quoted 14 lines

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 313

Quoted 8 lines
drivers/xen/xen-pciback/pci_stub.c +7 -0
+			pci_name(arg.dev), dev->bus->number);
+
+		return -EBUSY;
+	}
+
+	dev_dbg(&dev->dev, "pcistub owns %d devices on bus 0x%x\n",
+		arg.dcount, dev->bus->number);

While here the original device is perhaps not necessary to print, the bare bus number doesn't carry enough information: You'll want to prefix it by the segment number. Plus you'll want to use canonical formatting (ssss:bb), so one can get matches when suitably grep-ing the log. Perhaps bus->name is what you're after.

Sure, I can change it to

dev_dbg(&dev->dev, "pcistub owns %d devices on PCI Bus %04x:%02x", pci_domain_nr(bus), bus->number);

Cheers
GOVINDA

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw
In reply to Jan Beulich

Jan,

Please see below for my comments.

Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 09.11.17 at 00:06, <[email protected]> wrote:
drivers/xen/xen-pciback/pci_stub.c +0 -0
@@ -244,6 +244,91 @@ struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev,
  	return found_dev;
  }
 
Quoted 8 lines

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 248

Quoted 2 lines
drivers/xen/xen-pciback/pci_stub.c +2 -0
+struct pcistub_args {
+	struct pci_dev *dev;

Please don't ignore prior review comments: Either carry out what was requested, or explain why the request can't be fulfilled. You saying "This field will point to first device that is not owned by pcistub" to Roger's request to make this a pointer to const is not a valid reason to not do the adjustment; in fact your reply is entirely unrelated to the request.

We can use "const" with above field since we will set this field only once. I will change it.

Quoted 8 lines

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 256

Quoted 6 lines
drivers/xen/xen-pciback/pci_stub.c +5 -0
+static int pcistub_search_dev(struct pci_dev *dev, void *data)
+{
+	struct pcistub_device *psdev;
+	struct pcistub_args *arg = data;
+	bool found_dev = false;

Purely cosmetical, but anyway: Why not just "found"? What else could be (not) found here other than the device in question?

Sure, I will change it to "found".

Quoted 24 lines
Quoted 10 lines
drivers/xen/xen-pciback/pci_stub.c +9 -0
+	unsigned long flags;
+
+	spin_lock_irqsave(&pcistub_devices_lock, flags);
+
+	list_for_each_entry(psdev, &pcistub_devices, dev_list) {
+		if (psdev->dev == dev) {
+			found_dev = true;
+			arg->dcount++;
+			break;

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 275

Quoted 10 lines
drivers/xen/xen-pciback/pci_stub.c +10 -0
+		}
+	}
+
+	spin_unlock_irqrestore(&pcistub_devices_lock, flags);
+
+	/* Device not owned by pcistub, someone owns it. Abort the walk */
+	if (!found_dev)
+		arg->dev = dev;
+
+	return found_dev ? 0 : 1;

Despite the function needing to return int, this can be simplified to "return !found_dev". I'd also like to note that the part of the earlier comment related to this is sort of disconnected. How about

Quoted 7 lines
	/* Device not owned by pcistub, someone owns it. Abort the walk */
	if (!found_dev) {
		arg->dev = dev;
		return 1;
	}

	return 0;

Fine with me.

Quoted 4 lines

And finally - I don't think the comment is entirely correct - the device not being owned by pciback doesn't necessarily mean it's owned by another driver. It could as well be unowned.

OK. I will modify this comment as " Device not owned by pcistub. Abort the walk".

Quoted 20 lines
Quoted 20 lines
drivers/xen/xen-pciback/pci_stub.c +19 -0
+static int pcistub_reset_dev(struct pci_dev *dev)
+{
+	struct xen_pcibk_dev_data *dev_data;
+	bool slot = false, bus = false;
+	struct pcistub_args arg = {};
+
+	if (!dev)
+		return -EINVAL;
+
+	dev_dbg(&dev->dev, "[%s]\n", __func__);
+
+	if (!pci_probe_reset_slot(dev->slot))
+		slot = true;
+	else if ((!pci_probe_reset_bus(dev->bus)) &&
+		 (!pci_is_root_bus(dev->bus)))
+		bus = true;
+
+	if (!bus && !slot)
+		return -EOPNOTSUPP;
Quoted 8 lines

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 301

Quoted 5 lines
drivers/xen/xen-pciback/pci_stub.c +5 -0
+
+	/*
+	 * Make sure all devices on this bus are owned by the
+	 * PCI backend so that we can safely reset the whole bus.
+	 */

Is that really the case when you mean to do a slot reset? It was for a reason that I had asked about a missing "else" in v1 review, rather than questioning the conditional around the logic.

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint. The connected card/endpoint can be multi-function device. So, same walk-through and checking is needed irrespective of type of reset being used.

Quoted 8 lines

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 306

Quoted 6 lines
drivers/xen/xen-pciback/pci_stub.c +5 -0
+	pci_walk_bus(dev->bus, pcistub_search_dev, &arg);
+
+	/* All devices under the bus should be part of pcistub! */
+	if (arg.dev) {
+		dev_err(&dev->dev, "%s device on bus 0x%x is not owned by pcistub\n",

%#x

Quoted 8 lines

Yet then, thinking about what would be useful information should the situation really arise, I'm not convinced printing a bare bus number here is useful either. Especially for the case of multiple parallel requests you want to make it possible to match each message to the original request (guest start or whatever). Hence I think you want something like

"%s on the same bus as %s is not owned by " DRV_NAME "\n"

Sure, I will make this change.

Quoted 14 lines

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 313

Quoted 8 lines
drivers/xen/xen-pciback/pci_stub.c +7 -0
+			pci_name(arg.dev), dev->bus->number);
+
+		return -EBUSY;
+	}
+
+	dev_dbg(&dev->dev, "pcistub owns %d devices on bus 0x%x\n",
+		arg.dcount, dev->bus->number);

While here the original device is perhaps not necessary to print, the bare bus number doesn't carry enough information: You'll want to prefix it by the segment number. Plus you'll want to use canonical formatting (ssss:bb), so one can get matches when suitably grep-ing the log. Perhaps bus->name is what you're after.

Sure, I can change it to

dev_dbg(&dev->dev, "pcistub owns %d devices on PCI Bus %04x:%02x", pci_domain_nr(bus), bus->number);

Cheers
GOVINDA

#4 lore raw
Quoted 31 lines
Quoted 1 line
Quoted 1 line · On 29.11.17 at 16:37, <[email protected]> wrote:
Quoted 20 lines · Jan Beulich
Quoted 20 lines
Quoted 1 line
Quoted 1 line · On 09.11.17 at 00:06, <[email protected]> wrote:
drivers/xen/xen-pciback/pci_stub.c +19 -0
+static int pcistub_reset_dev(struct pci_dev *dev)
+{
+	struct xen_pcibk_dev_data *dev_data;
+	bool slot = false, bus = false;
+	struct pcistub_args arg = {};
+
+	if (!dev)
+		return -EINVAL;
+
+	dev_dbg(&dev->dev, "[%s]\n", __func__);
+
+	if (!pci_probe_reset_slot(dev->slot))
+		slot = true;
+	else if ((!pci_probe_reset_bus(dev->bus)) &&
+		 (!pci_is_root_bus(dev->bus)))
+		bus = true;
+
+	if (!bus && !slot)
+		return -EOPNOTSUPP;
Quoted 8 lines

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 301

Quoted 5 lines
drivers/xen/xen-pciback/pci_stub.c +5 -0
+
+	/*
+	 * Make sure all devices on this bus are owned by the
+	 * PCI backend so that we can safely reset the whole bus.
+	 */

Is that really the case when you mean to do a slot reset? It was for a reason that I had asked about a missing "else" in v1 review, rather than questioning the conditional around the logic.

Quoted 5 lines

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

Jan

#5 lore raw
Quoted 8 lines
Quoted 6 lines

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

For PCIe platforms, both slot and bus reset endup resetting all connected device/functions on thesecondary bus (behind the root-port or downstream-port).

Cheers
GOVINDA

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
In reply to Jan Beulich
Quoted 8 lines
Quoted 6 lines

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

For PCIe platforms, both slot and bus reset endup resetting all connected device/functions on thesecondary bus (behind the root-port or downstream-port).

Cheers
GOVINDA

#7 lore raw
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines · On 29.11.17 at 18:38, <[email protected]> wrote:

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

For PCIe platforms, both slot and bus reset endup resetting all connected device/functions on thesecondary bus (behind the root-port or downstream-port).

According to my understanding this contradicts the comment ahead of pci_reset_slot(), which talks of multiple slots per bus. In such a setup, I can't see why resetting on slot would affect other slots on the same bus. At the same time the comment says that the slot reset may resolve to a bus one when there's just a single slot on the bus.

Jan

#8 lore raw
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines · On 29.11.17 at 18:38, <[email protected]> wrote:

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

For PCIe platforms, both slot and bus reset endup resetting all connected device/functions on thesecondary bus (behind the root-port or downstream-port).

Quoted 6 lines

According to my understanding this contradicts the comment ahead of pci_reset_slot(), which talks of multiple slots per bus. In such a setup, I can't see why resetting on slot would affect other slots on the same bus. At the same time the comment says that the slot reset may resolve to a bus one when there's just a single slot on the bus.

For legacy PCI/PCI-X, we can have multiple slots per bus but not with PCI-Express (each link will be on a separate bus). In anycase, we need to walk-through other device/functions on the same bus/slot and check their status before using slot/bus reset.

Cheers
GOVINDA

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#9 lore raw
In reply to Jan Beulich
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines · On 29.11.17 at 18:38, <[email protected]> wrote:

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

For PCIe platforms, both slot and bus reset endup resetting all connected device/functions on thesecondary bus (behind the root-port or downstream-port).

Quoted 6 lines

According to my understanding this contradicts the comment ahead of pci_reset_slot(), which talks of multiple slots per bus. In such a setup, I can't see why resetting on slot would affect other slots on the same bus. At the same time the comment says that the slot reset may resolve to a bus one when there's just a single slot on the bus.

For legacy PCI/PCI-X, we can have multiple slots per bus but not with PCI-Express (each link will be on a separate bus). In anycase, we need to walk-through other device/functions on the same bus/slot and check their status before using slot/bus reset.

Cheers
GOVINDA

#10 lore raw
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 30.11.17 at 15:15, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines · On 29.11.17 at 18:38, <[email protected]> wrote:

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

For PCIe platforms, both slot and bus reset endup resetting all connected device/functions on thesecondary bus (behind the root-port or downstream-port).

Quoted 6 lines

According to my understanding this contradicts the comment ahead of pci_reset_slot(), which talks of multiple slots per bus. In such a setup, I can't see why resetting on slot would affect other slots on the same bus. At the same time the comment says that the slot reset may resolve to a bus one when there's just a single slot on the bus.

Quoted 3 lines

For legacy PCI/PCI-X, we can have multiple slots per bus but not with PCI-Express
(each link will be on a separate bus).

Is that true even for root complex integrated end points? A random system's lspci output doesn't seem to agree with what you say. A typical example would be USB controllers all sitting on bus 0, but having different slot numbers. You clearly won't be able to ever bus-reset these, and if you checked all devices on bus 0 you would then also not be able to slot-reset them.

Jan

#11 lore raw
Quoted 22 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 30.11.17 at 15:15, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines · On 29.11.17 at 18:38, <[email protected]> wrote:

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

For PCIe platforms, both slot and bus reset endup resetting all connected device/functions on thesecondary bus (behind the root-port or downstream-port).

Quoted 6 lines

According to my understanding this contradicts the comment ahead of pci_reset_slot(), which talks of multiple slots per bus. In such a setup, I can't see why resetting on slot would affect other slots on the same bus. At the same time the comment says that the slot reset may resolve to a bus one when there's just a single slot on the bus.

Quoted 3 lines

For legacy PCI/PCI-X, we can have multiple slots per bus but not with PCI-Express
(each link will be on a separate bus).

Quoted 6 lines

Is that true even for root complex integrated end points? A random system's lspci output doesn't seem to agree with what you say. A typical example would be USB controllers all sitting on bus 0, but having different slot numbers. You clearly won't be able to ever bus-reset these, and if you checked all devices on bus 0 you would then also not be able to slot-reset them.

Here, slot reset refers to any PCIe slot that implements or supports hotplug feature. The slot reset ultimately invokes "pciehp_reset_slot()". W.r.t integrated endpoints, these can be reset either through FLR or secondary bus reset methods only.

Cheers
GOVINDA

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw
In reply to Jan Beulich
Quoted 22 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 30.11.17 at 15:15, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines · On 29.11.17 at 18:38, <[email protected]> wrote:

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

For PCIe platforms, both slot and bus reset endup resetting all connected device/functions on thesecondary bus (behind the root-port or downstream-port).

Quoted 6 lines

According to my understanding this contradicts the comment ahead of pci_reset_slot(), which talks of multiple slots per bus. In such a setup, I can't see why resetting on slot would affect other slots on the same bus. At the same time the comment says that the slot reset may resolve to a bus one when there's just a single slot on the bus.

Quoted 3 lines

For legacy PCI/PCI-X, we can have multiple slots per bus but not with PCI-Express
(each link will be on a separate bus).

Quoted 6 lines

Is that true even for root complex integrated end points? A random system's lspci output doesn't seem to agree with what you say. A typical example would be USB controllers all sitting on bus 0, but having different slot numbers. You clearly won't be able to ever bus-reset these, and if you checked all devices on bus 0 you would then also not be able to slot-reset them.

Here, slot reset refers to any PCIe slot that implements or supports hotplug feature. The slot reset ultimately invokes "pciehp_reset_slot()". W.r.t integrated endpoints, these can be reset either through FLR or secondary bus reset methods only.

Cheers
GOVINDA

#13 lore raw

Jan,

Do you have any further comments on the current version of this patch?. Otherwise, I will work on the review comments and publish next version of this patch later this week. Please let me know. Thanks.

Cheers
GOVINDA

Quoted 35 lines · Govinda Tatti
Quoted 22 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 30.11.17 at 15:15, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines · On 29.11.17 at 18:38, <[email protected]> wrote:

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

For PCIe platforms, both slot and bus reset endup resetting all connected device/functions on thesecondary bus (behind the root-port or downstream-port).

Quoted 6 lines

According to my understanding this contradicts the comment ahead of pci_reset_slot(), which talks of multiple slots per bus. In such a setup, I can't see why resetting on slot would affect other slots on the same bus. At the same time the comment says that the slot reset may resolve to a bus one when there's just a single slot on the bus.

Quoted 3 lines

For legacy PCI/PCI-X, we can have multiple slots per bus but not with PCI-Express
(each link will be on a separate bus).

Quoted 6 lines

Is that true even for root complex integrated end points? A random system's lspci output doesn't seem to agree with what you say. A typical example would be USB controllers all sitting on bus 0, but having different slot numbers. You clearly won't be able to ever bus-reset these, and if you checked all devices on bus 0 you would then also not be able to slot-reset them.

Here, slot reset refers to any PCIe slot that implements or supports hotplug feature. The slot reset ultimately invokes "pciehp_reset_slot()". W.r.t integrated endpoints, these can be reset either through FLR or secondary bus reset methods only.

Quoted 7 lines

Cheers
GOVINDA

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#14 lore raw
In reply to Govinda Tatti

Jan,

Do you have any further comments on the current version of this patch?. Otherwise, I will work on the review comments and publish next version of this patch later this week. Please let me know. Thanks.

Cheers
GOVINDA

Quoted 42 lines · Govinda Tatti
Quoted 22 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 30.11.17 at 15:15, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines · On 29.11.17 at 18:38, <[email protected]> wrote:

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

For PCIe platforms, both slot and bus reset endup resetting all connected device/functions on thesecondary bus (behind the root-port or downstream-port).

Quoted 6 lines

According to my understanding this contradicts the comment ahead of pci_reset_slot(), which talks of multiple slots per bus. In such a setup, I can't see why resetting on slot would affect other slots on the same bus. At the same time the comment says that the slot reset may resolve to a bus one when there's just a single slot on the bus.

Quoted 3 lines

For legacy PCI/PCI-X, we can have multiple slots per bus but not with PCI-Express
(each link will be on a separate bus).

Quoted 6 lines

Is that true even for root complex integrated end points? A random system's lspci output doesn't seem to agree with what you say. A typical example would be USB controllers all sitting on bus 0, but having different slot numbers. You clearly won't be able to ever bus-reset these, and if you checked all devices on bus 0 you would then also not be able to slot-reset them.

Here, slot reset refers to any PCIe slot that implements or supports hotplug feature. The slot reset ultimately invokes "pciehp_reset_slot()". W.r.t integrated endpoints, these can be reset either through FLR or secondary bus reset methods only.

Cheers
GOVINDA

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#16 lore raw
In reply to Govinda Tatti
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 04.12.17 at 17:16, <[email protected]> wrote:

Do you have any further comments on the current version of this patch?.

No. I'm not fully understanding your most recent slot related comments, but I'll trust you and Konrad to get this into suitable shape.

Jan

#17 lore raw
In reply to Govinda Tatti
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines · On 29.11.17 at 18:38, <[email protected]> wrote:

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

For PCIe platforms, both slot and bus reset endup resetting all connected device/functions on thesecondary bus (behind the root-port or downstream-port).

According to my understanding this contradicts the comment ahead of pci_reset_slot(), which talks of multiple slots per bus. In such a setup, I can't see why resetting on slot would affect other slots on the same bus. At the same time the comment says that the slot reset may resolve to a bus one when there's just a single slot on the bus.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#18 lore raw
In reply to Govinda Tatti
Quoted 31 lines
Quoted 1 line
Quoted 1 line · On 29.11.17 at 16:37, <[email protected]> wrote:
Quoted 20 lines · Jan Beulich
Quoted 20 lines
Quoted 1 line
Quoted 1 line · On 09.11.17 at 00:06, <[email protected]> wrote:
drivers/xen/xen-pciback/pci_stub.c +19 -0
+static int pcistub_reset_dev(struct pci_dev *dev)
+{
+	struct xen_pcibk_dev_data *dev_data;
+	bool slot = false, bus = false;
+	struct pcistub_args arg = {};
+
+	if (!dev)
+		return -EINVAL;
+
+	dev_dbg(&dev->dev, "[%s]\n", __func__);
+
+	if (!pci_probe_reset_slot(dev->slot))
+		slot = true;
+	else if ((!pci_probe_reset_bus(dev->bus)) &&
+		 (!pci_is_root_bus(dev->bus)))
+		bus = true;
+
+	if (!bus && !slot)
+		return -EOPNOTSUPP;
Quoted 8 lines

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 301

Quoted 5 lines
drivers/xen/xen-pciback/pci_stub.c +5 -0
+
+	/*
+	 * Make sure all devices on this bus are owned by the
+	 * PCI backend so that we can safely reset the whole bus.
+	 */

Is that really the case when you mean to do a slot reset? It was for a reason that I had asked about a missing "else" in v1 review, rather than questioning the conditional around the logic.

Quoted 5 lines

In the case of bus or slot reset, our goal is to reset connected PCIe fabric/card/endpoint.
The connected card/endpoint can be multi-function device. So, same walk-through and checking
is needed irrespective of type of reset being used.

I don't follow: The scope of other devices/functions possibly affected by a reset depends on the type of reset, doesn't it?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#19 lore raw
In reply to Govinda Tatti

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 248

Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 09.11.17 at 00:06, <[email protected]> wrote:
drivers/xen/xen-pciback/pci_stub.c +2 -0
@@ -244,6 +244,91 @@ struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev,
 	return found_dev;
 }
 
+struct pcistub_args {
+	struct pci_dev *dev;

Please don't ignore prior review comments: Either carry out what was requested, or explain why the request can't be fulfilled. You saying "This field will point to first device that is not owned by pcistub" to Roger's request to make this a pointer to const is not a valid reason to not do the adjustment; in fact your reply is entirely unrelated to the request.

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 256

Quoted 5 lines
drivers/xen/xen-pciback/pci_stub.c +5 -0
+static int pcistub_search_dev(struct pci_dev *dev, void *data)
+{
+	struct pcistub_device *psdev;
+	struct pcistub_args *arg = data;
+	bool found_dev = false;

Purely cosmetical, but anyway: Why not just "found"? What else could be (not) found here other than the device in question?

Quoted 9 lines
drivers/xen/xen-pciback/pci_stub.c +9 -0
+	unsigned long flags;
+
+	spin_lock_irqsave(&pcistub_devices_lock, flags);
+
+	list_for_each_entry(psdev, &pcistub_devices, dev_list) {
+		if (psdev->dev == dev) {
+			found_dev = true;
+			arg->dcount++;
+			break;

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 275

Quoted 10 lines
drivers/xen/xen-pciback/pci_stub.c +10 -0
+		}
+	}
+
+	spin_unlock_irqrestore(&pcistub_devices_lock, flags);
+
+	/* Device not owned by pcistub, someone owns it. Abort the walk */
+	if (!found_dev)
+		arg->dev = dev;
+
+	return found_dev ? 0 : 1;

Despite the function needing to return int, this can be simplified to "return !found_dev". I'd also like to note that the part of the earlier comment related to this is sort of disconnected. How about

	/* Device not owned by pcistub, someone owns it. Abort the walk */
	if (!found_dev) {
		arg->dev = dev;
		return 1;
	}

	return 0;

And finally - I don't think the comment is entirely correct - the device not being owned by pciback doesn't necessarily mean it's owned by another driver. It could as well be unowned.

Quoted 14 lines
drivers/xen/xen-pciback/pci_stub.c +14 -0
+static int pcistub_reset_dev(struct pci_dev *dev)
+{
+	struct xen_pcibk_dev_data *dev_data;
+	bool slot = false, bus = false;
+	struct pcistub_args arg = {};
+
+	if (!dev)
+		return -EINVAL;
+
+	dev_dbg(&dev->dev, "[%s]\n", __func__);
+
+	if (!pci_probe_reset_slot(dev->slot))
+		slot = true;
+	else if ((!pci_probe_reset_bus(dev->bus)) &&

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 301

Quoted 10 lines
drivers/xen/xen-pciback/pci_stub.c +10 -0
+		 (!pci_is_root_bus(dev->bus)))
+		bus = true;
+
+	if (!bus && !slot)
+		return -EOPNOTSUPP;
+
+	/*
+	 * Make sure all devices on this bus are owned by the
+	 * PCI backend so that we can safely reset the whole bus.
+	 */

Is that really the case when you mean to do a slot reset? It was for a reason that I had asked about a missing "else" in v1 review, rather than questioning the conditional around the logic.

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 306

Quoted 5 lines
drivers/xen/xen-pciback/pci_stub.c +5 -0
+	pci_walk_bus(dev->bus, pcistub_search_dev, &arg);
+
+	/* All devices under the bus should be part of pcistub! */
+	if (arg.dev) {
+		dev_err(&dev->dev, "%s device on bus 0x%x is not owned by pcistub\n",

%#x

Yet then, thinking about what would be useful information should the situation really arise, I'm not convinced printing a bare bus number here is useful either. Especially for the case of multiple parallel requests you want to make it possible to match each message to the original request (guest start or whatever). Hence I think you want something like

"%s on the same bus as %s is not owned by " DRV_NAME "\n"

drivers/xen/xen-pciback/pci_stub.c struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev, · line 313

Quoted 7 lines
drivers/xen/xen-pciback/pci_stub.c +7 -0
+			pci_name(arg.dev), dev->bus->number);
+
+		return -EBUSY;
+	}
+
+	dev_dbg(&dev->dev, "pcistub owns %d devices on bus 0x%x\n",
+		arg.dcount, dev->bus->number);

While here the original device is perhaps not necessary to print, the bare bus number doesn't carry enough information: You'll want to prefix it by the segment number. Plus you'll want to use canonical formatting (ssss:bb), so one can get matches when suitably grep-ing the log. Perhaps bus->name is what you're after.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel