Skip to contentSkip to conversation

PCI: pciehp: Add quirk to handle spurious DLLSC on a x4x4 SSD

PATCH In review Jon Derrick 4 participants 11 messages Aug 30, 2021 – Sep 26, 2022 PCI

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 PCI: pciehp: Add quirk to handle spurious DLLSC on a x4x4 SSD

From: James Puthukattukaran <[email protected]>

When an Intel P5608 SSD is bifurcated into x4x4 mode, and the upstream ports both support hotplugging on each respective x4 device, a slot management system for the SSD requires both x4 slots to have power removed via sysfs (echo 0 > slot/N/power), from the OS before it can safely turn-off physical power for the whole x8 device. The implications are that slot status will display powered off and link inactive statuses for the x4 devices where the devices are actually powered until both ports have powered off.

The issue with the SSD manifests when power is removed from the first-half and then the second-half. During the first-half removal, slot status shows the slot as powered-down and link-inactive, while internal power and link remain active while waiting for the second-half to have power removed. When power is then removed from the second-half, the first-half starts shutdown sequence and will trigger a DLLSC event. This is misinterpreted as an enabling event and causes the first-half to be re-enabled.

The spurious enable can be resolved by ignoring link status change events when no link is active when in the off state. This patch adds a quirk for specific P5608 SSDs which have been tested for compatibility.

---
v2->v3: Clarified commit message and comment block
Added second supported subdevice ID Added hotplug ifdef blocks

 drivers/pci/hotplug/pciehp_ctrl.c |  7 +++++++
 drivers/pci/quirks.c              | 34 +++++++++++++++++++++++++++++++
 include/linux/pci.h               |  1 +
 3 files changed, 42 insertions(+)
drivers/pci/hotplug/pciehp_ctrl.c +7 -0
index 529c34808440..db41f78bfac8 100644
@@ -225,6 +225,7 @@ void pciehp_handle_disable_request(struct controller *ctrl)
 void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events)
 {
 	int present, link_active;
+	struct pci_dev *pdev = ctrl->pcie->port;
 
 	/*
 	 * If the slot is on and presence or link has changed, turn it off.
@@ -265,6 +266,12 @@ void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events)
 		cancel_delayed_work(&ctrl->button_work);
 		fallthrough;
 	case OFF_STATE:
+		if (pdev->shared_pcc_and_link_slot &&
+		    (events & PCI_EXP_SLTSTA_DLLSC) && !link_active) {
+			mutex_unlock(&ctrl->state_lock);
+			break;
+		}
+
 		ctrl->state = POWERON_STATE;
 		mutex_unlock(&ctrl->state_lock);
 		if (present)
drivers/pci/quirks.c +34 -0
index 10122e3318cf..c6b48ddc5c3d 100644
@@ -5750,3 +5750,37 @@ static void apex_pci_fixup_class(struct pci_dev *pdev)
 }
 DECLARE_PCI_FIXUP_CLASS_HEADER(0x1ac1, 0x089a,
 			       PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class);
+
+#ifdef CONFIG_HOTPLUG_PCI_PCIE
+/*
+ * This is an Intel NVMe SSD which sits in a x8 pciehp slot but is bifurcated
+ * as a x4x4 and manifests as two slots with respect to PCIe hot plug register
+ * states. However, the hotplug controller treats these slots as a single x8
+ * slot for link and power. Either one of the two slots can be powered down
+ * separately and the slot status will show negative power and link states, but
+ * internal power and link will be active until the last of the two slots is
+ * powered down. When the last of the two x4 slots is turned off, power and
+ * link will be turned off for the x8 slot by the HP controller. This
+ * configuration causes some interesting behavior in bringup sequence
+ *
+ * When the second slot is powered off to remove the card, this will cause the
+ * link to go down for both x4 slots. So, the x4 that is already powered down
+ * earlier will see a DLLSC event and attempt to bring itself up (card present,
+ * link change event, link state is down). Special handling is required in
+ * pciehp_handle_presence_or_link_change to prevent this unintended bring up
+ */
+static void quirk_shared_pcc_and_link_slot(struct pci_dev *pdev)
+{
+	struct pci_dev *parent = pci_upstream_bridge(pdev);
+
+	if (parent && pdev->subsystem_vendor == 0x108e) {
+		switch (pdev->subsystem_device) {
+		/* P5608 */
+		case 0x487d:
+		case 0x488d:
+			parent->shared_pcc_and_link_slot = 1;
+		}
+	}
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0b60, quirk_shared_pcc_and_link_slot);
+#endif /* CONFIG_HOTPLUG_PCI_PCIE */
include/linux/pci.h +1 -0
index 7ed95f11c6bd..bcef73209487 100644
@@ -463,6 +463,7 @@ struct pci_dev {
 
 #ifdef CONFIG_HOTPLUG_PCI_PCIE
 	unsigned int	broken_cmd_compl:1;	/* No compl for some cmds */
+	unsigned int	shared_pcc_and_link_slot:1;
 #endif
 #ifdef CONFIG_PCIE_PTM
 	unsigned int	ptm_root:1;
Signature · 1 line
2.27.0
#1 lore raw

Hi Jonathan

Looks mostly good. Thanks for the update. Reads better, but I'm still a bit confused .. sorry

Quoted 10 lines · Jon Derrick

From: James Puthukattukaran <[email protected]>

When an Intel P5608 SSD is bifurcated into x4x4 mode, and the upstream ports both support hotplugging on each respective x4 device, a slot management system for the SSD requires both x4 slots to have power removed via sysfs (echo 0 > slot/N/power), from the OS before it can safely turn-off physical power for the whole x8 device. The implications are that slot status will display powered off and link inactive statuses for the x4 devices where the devices are actually powered until both ports have powered off.

This part is a bit muddy based on the description in the following writeup.

slot1->power off->link1 DOWN (Should fire a DLLSC event?) or does the link stay active until slot2 is powered off?

slot2->power off, looks like only now link1 is getting DLLSC which you are ignoring skipping to make sure you don't turn this into an ON event.

slot2 will also receive DLLSC, and you would ignore it also for the same reasons.

Maybe this was discussed earlier and I didn't catch it.
When in OFF-state, and DLLSC is received but link is down while in OFF state, shoudn't that be the default handling? It almost looks like its a spurious DLLSC event that can be ignored for all cases without a specific quirk?

Or is there a case we care about for normal devices?

Quoted 9 lines

The issue with the SSD manifests when power is removed from the first-half and then the second-half. During the first-half removal, slot status shows the slot as powered-down and link-inactive, while internal power and link remain active while waiting for the second-half to have power removed. When power is then removed from the second-half, the first-half starts shutdown sequence and will trigger a DLLSC event. This is misinterpreted as an enabling event and causes the first-half to be re-enabled.

Question is:

when slot1 is powered off, do you get a DLLSC for that link that reports link_down? or does external link still report UP until both ports are powered off?

Quoted 99 lines

The spurious enable can be resolved by ignoring link status change events when no link is active when in the off state. This patch adds a quirk for specific P5608 SSDs which have been tested for compatibility.

---
v2->v3: Clarified commit message and comment block
Added second supported subdevice ID Added hotplug ifdef blocks

 drivers/pci/hotplug/pciehp_ctrl.c |  7 +++++++
 drivers/pci/quirks.c              | 34 +++++++++++++++++++++++++++++++
 include/linux/pci.h               |  1 +
 3 files changed, 42 insertions(+)
drivers/pci/hotplug/pciehp_ctrl.c +7 -0
index 529c34808440..db41f78bfac8 100644
@@ -225,6 +225,7 @@ void pciehp_handle_disable_request(struct controller *ctrl)
 void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events)
 {
 	int present, link_active;
+	struct pci_dev *pdev = ctrl->pcie->port;
 
 	/*
 	 * If the slot is on and presence or link has changed, turn it off.
@@ -265,6 +266,12 @@ void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events)
 		cancel_delayed_work(&ctrl->button_work);
 		fallthrough;
 	case OFF_STATE:
+		if (pdev->shared_pcc_and_link_slot &&
+		    (events & PCI_EXP_SLTSTA_DLLSC) && !link_active) {
+			mutex_unlock(&ctrl->state_lock);
+			break;
+		}
+
 		ctrl->state = POWERON_STATE;
 		mutex_unlock(&ctrl->state_lock);
 		if (present)
drivers/pci/quirks.c +34 -0
index 10122e3318cf..c6b48ddc5c3d 100644
@@ -5750,3 +5750,37 @@ static void apex_pci_fixup_class(struct pci_dev *pdev)
 }
 DECLARE_PCI_FIXUP_CLASS_HEADER(0x1ac1, 0x089a,
 			       PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class);
+
+#ifdef CONFIG_HOTPLUG_PCI_PCIE
+/*
+ * This is an Intel NVMe SSD which sits in a x8 pciehp slot but is bifurcated
+ * as a x4x4 and manifests as two slots with respect to PCIe hot plug register
+ * states. However, the hotplug controller treats these slots as a single x8
+ * slot for link and power. Either one of the two slots can be powered down
+ * separately and the slot status will show negative power and link states, but
+ * internal power and link will be active until the last of the two slots is
+ * powered down. When the last of the two x4 slots is turned off, power and
+ * link will be turned off for the x8 slot by the HP controller. This
+ * configuration causes some interesting behavior in bringup sequence
+ *
+ * When the second slot is powered off to remove the card, this will cause the
+ * link to go down for both x4 slots. So, the x4 that is already powered down
+ * earlier will see a DLLSC event and attempt to bring itself up (card present,
+ * link change event, link state is down). Special handling is required in
+ * pciehp_handle_presence_or_link_change to prevent this unintended bring up
+ */
+static void quirk_shared_pcc_and_link_slot(struct pci_dev *pdev)
+{
+	struct pci_dev *parent = pci_upstream_bridge(pdev);
+
+	if (parent && pdev->subsystem_vendor == 0x108e) {
+		switch (pdev->subsystem_device) {
+		/* P5608 */
+		case 0x487d:
+		case 0x488d:
+			parent->shared_pcc_and_link_slot = 1;
+		}
+	}
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0b60, quirk_shared_pcc_and_link_slot);
+#endif /* CONFIG_HOTPLUG_PCI_PCIE */
include/linux/pci.h +1 -0
index 7ed95f11c6bd..bcef73209487 100644
@@ -463,6 +463,7 @@ struct pci_dev {
 
 #ifdef CONFIG_HOTPLUG_PCI_PCIE
 	unsigned int	broken_cmd_compl:1;	/* No compl for some cmds */
+	unsigned int	shared_pcc_and_link_slot:1;
 #endif
 #ifdef CONFIG_PCIE_PTM
 	unsigned int	ptm_root:1;
Signature · 1 line
2.27.0
Signature · 2 lines
Cheers,
Ashok
#2 lore raw

Hi Ashok

Quoted 18 lines · Ashok

Hi Jonathan

Looks mostly good. Thanks for the update. Reads better, but I'm still a bit confused .. sorry

Quoted 10 lines · Jon Derrick

From: James Puthukattukaran <[email protected]>

When an Intel P5608 SSD is bifurcated into x4x4 mode, and the upstream ports both support hotplugging on each respective x4 device, a slot management system for the SSD requires both x4 slots to have power removed via sysfs (echo 0 > slot/N/power), from the OS before it can safely turn-off physical power for the whole x8 device. The implications are that slot status will display powered off and link inactive statuses for the x4 devices where the devices are actually powered until both ports have powered off.

Quoted 4 lines

This part is a bit muddy based on the description in the following writeup.

slot1->power off->link1 DOWN (Should fire a DLLSC event?) or does the link stay active until slot2 is powered off?

Internal link is active but link is presented as down in the slot status register

Quoted 3 lines

slot2->power off, looks like only now link1 is getting DLLSC which you are ignoring skipping to make sure you don't turn this into an ON event.

I think the dllsc on slot 1 occurs when the whole complex powers down after both slots are powered-off

Quoted 10 lines

slot2 will also receive DLLSC, and you would ignore it also for the same reasons.

Maybe this was discussed earlier and I didn't catch it.
When in OFF-state, and DLLSC is received but link is down while in OFF state, shoudn't that be the default handling? It almost looks like its a spurious DLLSC event that can be ignored for all cases without a specific quirk?

Or is there a case we care about for normal devices?

Lukas presented a possible race [1] with a spurious surprise hotplug where you could be in off state and link is still inactive, expecting the state machine to power on the slot.

This is avoided in this SSD because the individual slots in the complex won’t be surprise hotplugged, avoiding spurious events.

[1] https://lore.kernel.org/linux-pci/[email protected]/

Quoted 10 lines
Quoted 9 lines

The issue with the SSD manifests when power is removed from the first-half and then the second-half. During the first-half removal, slot status shows the slot as powered-down and link-inactive, while internal power and link remain active while waiting for the second-half to have power removed. When power is then removed from the second-half, the first-half starts shutdown sequence and will trigger a DLLSC event. This is misinterpreted as an enabling event and causes the first-half to be re-enabled.

Quoted 5 lines

Question is:

when slot1 is powered off, do you get a DLLSC for that link that reports link_down? or does external link still report UP until both ports are powered off?

As far as the kernel knows by checking the slot status register, the link is inactive. Internally to the complex, link is still active until both slots are powered off.

It’s the powering off of the whole complex that causes slot 1 to throw dllsc, where internally slot 1 is going from active to inactive, and externally the kernel sees it staying inactive.

Quoted 103 lines
Quoted 92 lines

The spurious enable can be resolved by ignoring link status change events when no link is active when in the off state. This patch adds a quirk for specific P5608 SSDs which have been tested for compatibility.

---
v2->v3: Clarified commit message and comment block Added second supported subdevice ID
Added hotplug ifdef blocks

drivers/pci/hotplug/pciehp_ctrl.c | 7 +++++++ drivers/pci/quirks.c | 34 +++++++++++++++++++++++++++++++ include/linux/pci.h | 1 +

3 files changed, 42 insertions(+)
drivers/pci/hotplug/pciehp_ctrl.c +1 -0
index 529c34808440..db41f78bfac8 100644
@@ -225,6 +225,7 @@ void pciehp_handle_disable_request(struct controller *ctrl)
 void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events)
 {
 int present, link_active;
+ struct pci_dev *pdev = ctrl->pcie->port;
 
 /*
 * If the slot is on and presence or link has changed, turn it off.
@@ -265,6 +266,12 @@ void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32
 events)
 cancel_delayed_work(&ctrl->button_work);
 fallthrough;

case OFF_STATE:
+ if (pdev->shared_pcc_and_link_slot &&
+ (events & PCI_EXP_SLTSTA_DLLSC) && !link_active) {
+ mutex_unlock(&ctrl->state_lock);
+ break;
+ }
+
ctrl->state = POWERON_STATE; mutex_unlock(&ctrl->state_lock);
if (present)

drivers/pci/quirks.c +34 -0
index 10122e3318cf..c6b48ddc5c3d 100644
@@ -5750,3 +5750,37 @@ static void apex_pci_fixup_class(struct pci_dev *pdev)
 }
 DECLARE_PCI_FIXUP_CLASS_HEADER(0x1ac1, 0x089a,
 PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class);
+
+#ifdef CONFIG_HOTPLUG_PCI_PCIE
+/*
+ * This is an Intel NVMe SSD which sits in a x8 pciehp slot but is bifurcated
+ * as a x4x4 and manifests as two slots with respect to PCIe hot plug register
+ * states. However, the hotplug controller treats these slots as a single x8
+ * slot for link and power. Either one of the two slots can be powered down
+ * separately and the slot status will show negative power and link states, but
+ * internal power and link will be active until the last of the two slots is
+ * powered down. When the last of the two x4 slots is turned off, power and
+ * link will be turned off for the x8 slot by the HP controller. This
+ * configuration causes some interesting behavior in bringup sequence
+ *
+ * When the second slot is powered off to remove the card, this will cause the
+ * link to go down for both x4 slots. So, the x4 that is already powered down
+ * earlier will see a DLLSC event and attempt to bring itself up (card present,
+ * link change event, link state is down). Special handling is required in
+ * pciehp_handle_presence_or_link_change to prevent this unintended bring up
+ */
+static void quirk_shared_pcc_and_link_slot(struct pci_dev *pdev)
+{
+ struct pci_dev *parent = pci_upstream_bridge(pdev);
+
+ if (parent && pdev->subsystem_vendor == 0x108e) {
+ switch (pdev->subsystem_device) {
+ /* P5608 */
+ case 0x487d:
+ case 0x488d:
+ parent->shared_pcc_and_link_slot = 1;
+ }
+ }
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0b60, quirk_shared_pcc_and_link_slot);
+#endif /* CONFIG_HOTPLUG_PCI_PCIE */
include/linux/pci.h +0 -0
index 7ed95f11c6bd..bcef73209487 100644
@@ -463,6 +463,7 @@ struct pci_dev {
 
 #ifdef CONFIG_HOTPLUG_PCI_PCIE
Quoted 7 lines

unsigned int broken_cmd_compl:1; /* No compl for some cmds */
+ unsigned int shared_pcc_and_link_slot:1;
#endif
#ifdef CONFIG_PCIE_PTM
unsigned int ptm_root:1;
--
2.27.0

--
Cheers,
Ashok

#3 lore raw
In reply to Jon Derrick
Quoted 8 lines · Jon Derrick

When an Intel P5608 SSD is bifurcated into x4x4 mode, and the upstream ports both support hotplugging on each respective x4 device, a slot management system for the SSD requires both x4 slots to have power removed via sysfs (echo 0 > slot/N/power), from the OS before it can safely turn-off physical power for the whole x8 device. The implications are that slot status will display powered off and link inactive statuses for the x4 devices where the devices are actually powered until both ports have powered off.

Just to get a better understanding, does the P5608 have an internal PCIe switch with hotplug capability on the Downstream Ports or does it plug into two separate PCIe slots? I recall previous patches mentioned a CEM interposer? (An lspci listing might be helpful.)

drivers/pci/hotplug/pciehp_ctrl.c void pciehp_handle_disable_request(struct controller *ctrl) · line 228

Quoted 7 lines
drivers/pci/hotplug/pciehp_ctrl.c +1 -0
@@ -225,6 +225,7 @@ void pciehp_handle_disable_request(struct controller *ctrl)
 void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events)
 {
 	int present, link_active;
+	struct pci_dev *pdev = ctrl->pcie->port;

Nit: Reverse christmas tree.

drivers/pci/hotplug/pciehp_ctrl.c void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events) · line 274

Quoted 10 lines
drivers/pci/hotplug/pciehp_ctrl.c +6 -0
@@ -265,6 +266,12 @@ void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events)
 		cancel_delayed_work(&ctrl->button_work);
 		fallthrough;
 	case OFF_STATE:
+		if (pdev->shared_pcc_and_link_slot &&
+		    (events & PCI_EXP_SLTSTA_DLLSC) && !link_active) {
+			mutex_unlock(&ctrl->state_lock);
+			break;
+		}
+

I think you also need to add...

			pdev->shared_pcc_and_link_slot = false;

... here to reset the shared_pcc_and_link_slot attribute in case the next card plugged into the slot doesn't have the quirk.

(This can't be done in pciehp_unconfigure_device() because the attribute is queried *after* the slot has been brought down.)

drivers/pci/quirks.c static void apex_pci_fixup_class(struct pci_dev *pdev) · line 5754

Quoted 8 lines
drivers/pci/quirks.c +2 -0
@@ -5750,3 +5750,37 @@ static void apex_pci_fixup_class(struct pci_dev *pdev)
 }
 DECLARE_PCI_FIXUP_CLASS_HEADER(0x1ac1, 0x089a,
 			       PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class);
+
+#ifdef CONFIG_HOTPLUG_PCI_PCIE

It's possible to put the quirk at the bottom of pciehp_ctrl.c and thus avoid the need for the #ifdef here. (We've got another pciehp-specific quirk at the bottom of pciehp_hpc.c.)

Otherwise LGTM.

Thanks,

Lukas

#4 lore raw
Quoted 10 lines · Lukas Wunner
Quoted 8 lines · Jon Derrick

When an Intel P5608 SSD is bifurcated into x4x4 mode, and the upstream ports both support hotplugging on each respective x4 device, a slot management system for the SSD requires both x4 slots to have power removed via sysfs (echo 0 > slot/N/power), from the OS before it can safely turn-off physical power for the whole x8 device. The implications are that slot status will display powered off and link inactive statuses for the x4 devices where the devices are actually powered until both ports have powered off.

Quoted 4 lines

Just to get a better understanding, does the P5608 have an internal PCIe switch with hotplug capability on the Downstream Ports or does it plug into two separate PCIe slots? I recall previous patches mentioned a CEM interposer? (An lspci listing might be helpful.)

It looks like 2 NVMe endpoints plugged into two different root ports, ex, 80:00.0 Root port to [81-86]
80:01.0 Root port to [87-8b]
81:00.0 NVMe
87:00.0 NVMe

The x8 is bifurcated to x4x4. Physically they share the same slot power/clock/reset but are logically separate per root port.

Quoted 11 lines

drivers/pci/hotplug/pciehp_ctrl.c void pciehp_handle_disable_request(struct controller *ctrl) · line 228

Quoted 9 lines
drivers/pci/hotplug/pciehp_ctrl.c +1 -0
@@ -225,6 +225,7 @@ void pciehp_handle_disable_request(struct controller *ctrl)
 void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events)
 {
 	int present, link_active;
+	struct pci_dev *pdev = ctrl->pcie->port;

Nit: Reverse christmas tree.

Sure

Quoted 15 lines

drivers/pci/hotplug/pciehp_ctrl.c void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events) · line 274

Quoted 12 lines
drivers/pci/hotplug/pciehp_ctrl.c +6 -0
@@ -265,6 +266,12 @@ void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events)
 		cancel_delayed_work(&ctrl->button_work);
 		fallthrough;
 	case OFF_STATE:
+		if (pdev->shared_pcc_and_link_slot &&
+		    (events & PCI_EXP_SLTSTA_DLLSC) && !link_active) {
+			mutex_unlock(&ctrl->state_lock);
+			break;
+		}
+

I think you also need to add...

Quoted 7 lines
			pdev->shared_pcc_and_link_slot = false;

... here to reset the shared_pcc_and_link_slot attribute in case the next card plugged into the slot doesn't have the quirk.

(This can't be done in pciehp_unconfigure_device() because the attribute is queried *after* the slot has been brought down.)

Agreed. I'll find a good spot for it.

Quoted 14 lines

drivers/pci/quirks.c static void apex_pci_fixup_class(struct pci_dev *pdev) · line 5754

Quoted 10 lines
drivers/pci/quirks.c +2 -0
@@ -5750,3 +5750,37 @@ static void apex_pci_fixup_class(struct pci_dev *pdev)
 }
 DECLARE_PCI_FIXUP_CLASS_HEADER(0x1ac1, 0x089a,
 			       PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class);
+
+#ifdef CONFIG_HOTPLUG_PCI_PCIE

It's possible to put the quirk at the bottom of pciehp_ctrl.c and thus avoid the need for the #ifdef here. (We've got another pciehp-specific quirk at the bottom of pciehp_hpc.c.)

Sure that would look fine

Quoted 7 lines

Otherwise LGTM.

Thanks,

Lukas

#5 lore raw
Quoted 16 lines · Jon Derrick
Quoted 10 lines · Lukas Wunner
Quoted 8 lines · Jon Derrick

When an Intel P5608 SSD is bifurcated into x4x4 mode, and the upstream ports both support hotplugging on each respective x4 device, a slot management system for the SSD requires both x4 slots to have power removed via sysfs (echo 0 > slot/N/power), from the OS before it can safely turn-off physical power for the whole x8 device. The implications are that slot status will display powered off and link inactive statuses for the x4 devices where the devices are actually powered until both ports have powered off.

Quoted 4 lines

Just to get a better understanding, does the P5608 have an internal PCIe switch with hotplug capability on the Downstream Ports or does it plug into two separate PCIe slots? I recall previous patches mentioned a CEM interposer? (An lspci listing might be helpful.)

Quoted 8 lines

It looks like 2 NVMe endpoints plugged into two different root ports, ex, 80:00.0 Root port to [81-86]
80:01.0 Root port to [87-8b]
81:00.0 NVMe
87:00.0 NVMe

The x8 is bifurcated to x4x4. Physically they share the same slot power/clock/reset but are logically separate per root port.

So are these two P5608 drives attached to a single Root Port with an interposer in-between?

I assume the Root Port needs to know that it's bifurcated and has to appear as two slots on the bus. Is this configured with a BIOS setting?

If these assumptions are true, the quirk isn't really specific to the P5608 but should rather apply to the bifurcation-capable Root Port and the quirk should set the flag if the Root Port is indeed configured for bifurcation.

Quoted 15 lines
Quoted 15 lines

drivers/pci/hotplug/pciehp_ctrl.c void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events) · line 274

Quoted 10 lines
drivers/pci/hotplug/pciehp_ctrl.c +6 -0
@@ -265,6 +266,12 @@ void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events)
 		cancel_delayed_work(&ctrl->button_work);
 		fallthrough;
 	case OFF_STATE:
+		if (pdev->shared_pcc_and_link_slot &&
+		    (events & PCI_EXP_SLTSTA_DLLSC) && !link_active) {
+			mutex_unlock(&ctrl->state_lock);
+			break;
+		}
+

I think you also need to add...

			pdev->shared_pcc_and_link_slot = false;
Quoted 7 lines
Quoted 5 lines

... here to reset the shared_pcc_and_link_slot attribute in case the next card plugged into the slot doesn't have the quirk.

(This can't be done in pciehp_unconfigure_device() because the attribute is queried *after* the slot has been brought down.)

Agreed. I'll find a good spot for it.

Adding it in the if-clause above should work. The if-clause is only entered when the sibling card has had its power removed, and this only happens once. When power is reinstated via sysfs, the device in the slot is reenumerated and pdev->shared_pcc_and_link_slot is set to true again if there's a quirked device in the slot.

Thanks,

Lukas

#6 lore raw
Quoted 29 lines · Lukas Wunner
Quoted 16 lines · Jon Derrick
Quoted 10 lines · Lukas Wunner
Quoted 8 lines · Jon Derrick

When an Intel P5608 SSD is bifurcated into x4x4 mode, and the upstream ports both support hotplugging on each respective x4 device, a slot management system for the SSD requires both x4 slots to have power removed via sysfs (echo 0 > slot/N/power), from the OS before it can safely turn-off physical power for the whole x8 device. The implications are that slot status will display powered off and link inactive statuses for the x4 devices where the devices are actually powered until both ports have powered off.

Quoted 4 lines

Just to get a better understanding, does the P5608 have an internal PCIe switch with hotplug capability on the Downstream Ports or does it plug into two separate PCIe slots? I recall previous patches mentioned a CEM interposer? (An lspci listing might be helpful.)

Quoted 8 lines

It looks like 2 NVMe endpoints plugged into two different root ports, ex, 80:00.0 Root port to [81-86]
80:01.0 Root port to [87-8b]
81:00.0 NVMe
87:00.0 NVMe

The x8 is bifurcated to x4x4. Physically they share the same slot power/clock/reset but are logically separate per root port.

So are these two P5608 drives attached to a single Root Port with an interposer in-between?

Quoted 7 lines

I assume the Root Port needs to know that it's bifurcated and has to appear as two slots on the bus. Is this configured with a BIOS setting?

If these assumptions are true, the quirk isn't really specific to the P5608 but should rather apply to the bifurcation-capable Root Port and the quirk should set the flag if the Root Port is indeed configured for bifurcation.

It's a function of the slot + card combination, but we can't distinguish this slot's special power handling behavior from the vanilla behavior. It's modified to handle power on the logically bifurcated, singular physical device.

Quoted 35 lines
Quoted 17 lines
Quoted 17 lines

drivers/pci/hotplug/pciehp_ctrl.c void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events) · line 274

Quoted 12 lines
drivers/pci/hotplug/pciehp_ctrl.c +6 -0
@@ -265,6 +266,12 @@ void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events)
 		cancel_delayed_work(&ctrl->button_work);
 		fallthrough;
 	case OFF_STATE:
+		if (pdev->shared_pcc_and_link_slot &&
+		    (events & PCI_EXP_SLTSTA_DLLSC) && !link_active) {
+			mutex_unlock(&ctrl->state_lock);
+			break;
+		}
+

I think you also need to add...

			pdev->shared_pcc_and_link_slot = false;
Quoted 7 lines
Quoted 5 lines

... here to reset the shared_pcc_and_link_slot attribute in case the next card plugged into the slot doesn't have the quirk.

(This can't be done in pciehp_unconfigure_device() because the attribute is queried *after* the slot has been brought down.)

Agreed. I'll find a good spot for it.

Adding it in the if-clause above should work. The if-clause is only entered when the sibling card has had its power removed, and this only happens once. When power is reinstated via sysfs, the device in the slot is reenumerated and pdev->shared_pcc_and_link_slot is set to true again if there's a quirked device in the slot.

Thanks,

Lukas

#7 lore raw
Quoted 35 lines · Jon Derrick
Quoted 32 lines · Lukas Wunner
Quoted 16 lines · Jon Derrick
Quoted 10 lines · Lukas Wunner
Quoted 8 lines · Jon Derrick

When an Intel P5608 SSD is bifurcated into x4x4 mode, and the upstream ports both support hotplugging on each respective x4 device, a slot management system for the SSD requires both x4 slots to have power removed via sysfs (echo 0 > slot/N/power), from the OS before it can safely turn-off physical power for the whole x8 device. The implications are that slot status will display powered off and link inactive statuses for the x4 devices where the devices are actually powered until both ports have powered off.

Quoted 4 lines

Just to get a better understanding, does the P5608 have an internal PCIe switch with hotplug capability on the Downstream Ports or does it plug into two separate PCIe slots? I recall previous patches mentioned a CEM interposer? (An lspci listing might be helpful.)

Quoted 8 lines

It looks like 2 NVMe endpoints plugged into two different root ports, ex, 80:00.0 Root port to [81-86]
80:01.0 Root port to [87-8b]
81:00.0 NVMe
87:00.0 NVMe

The x8 is bifurcated to x4x4. Physically they share the same slot power/clock/reset but are logically separate per root port.

So are these two P5608 drives attached to a single Root Port with an interposer in-between?

I assume the Root Port needs to know that it's bifurcated and has to appear as two slots on the bus. Is this configured with a BIOS setting?

Quoted 45 lines
Quoted 4 lines

If these assumptions are true, the quirk isn't really specific to the P5608 but should rather apply to the bifurcation-capable Root Port and the quirk should set the flag if the Root Port is indeed configured for bifurcation.

It's a function of the slot + card combination, but we can't distinguish this slot's special power handling behavior from the vanilla behavior. It's modified to handle power on the logically bifurcated, singular physical device.

Quoted 35 lines
Quoted 17 lines
Quoted 17 lines

drivers/pci/hotplug/pciehp_ctrl.c void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events) · line 274

Quoted 12 lines
drivers/pci/hotplug/pciehp_ctrl.c +6 -0
@@ -265,6 +266,12 @@ void pciehp_handle_presence_or_link_change(struct controller *ctrl, u32 events)
 		cancel_delayed_work(&ctrl->button_work);
 		fallthrough;
 	case OFF_STATE:
+		if (pdev->shared_pcc_and_link_slot &&
+		    (events & PCI_EXP_SLTSTA_DLLSC) && !link_active) {
+			mutex_unlock(&ctrl->state_lock);
+			break;
+		}
+

I think you also need to add...

			pdev->shared_pcc_and_link_slot = false;
Quoted 7 lines
Quoted 5 lines

... here to reset the shared_pcc_and_link_slot attribute in case the next card plugged into the slot doesn't have the quirk.

(This can't be done in pciehp_unconfigure_device() because the attribute is queried *after* the slot has been brought down.)

Agreed. I'll find a good spot for it.

Adding it in the if-clause above should work. The if-clause is only entered when the sibling card has had its power removed, and this only happens once. When power is reinstated via sysfs, the device in the slot is reenumerated and pdev->shared_pcc_and_link_slot is set to true again if there's a quirked device in the slot.

Thanks,

Lukas

Hi Bjorn, Lukas,

I need to resubmit this.

Besides the 'pdev->shared_pcc_and_link_slot = false', addition mentioned above, is there anything else that should be changed or any reason this wouldn't be accepted?

#8 lore raw
Quoted 42 lines · Jonathan Derrick
Quoted 33 lines · Jon Derrick
Quoted 32 lines · Lukas Wunner
Quoted 16 lines · Jon Derrick
Quoted 10 lines · Lukas Wunner
Quoted 8 lines · Jon Derrick

When an Intel P5608 SSD is bifurcated into x4x4 mode, and the upstream ports both support hotplugging on each respective x4 device, a slot management system for the SSD requires both x4 slots to have power removed via sysfs (echo 0 > slot/N/power), from the OS before it can safely turn-off physical power for the whole x8 device. The implications are that slot status will display powered off and link inactive statuses for the x4 devices where the devices are actually powered until both ports have powered off.

Quoted 4 lines

Just to get a better understanding, does the P5608 have an internal PCIe switch with hotplug capability on the Downstream Ports or does it plug into two separate PCIe slots? I recall previous patches mentioned a CEM interposer? (An lspci listing might be helpful.)

Quoted 8 lines

It looks like 2 NVMe endpoints plugged into two different root ports, ex, 80:00.0 Root port to [81-86]
80:01.0 Root port to [87-8b]
81:00.0 NVMe
87:00.0 NVMe

The x8 is bifurcated to x4x4. Physically they share the same slot power/clock/reset but are logically separate per root port.

So are these two P5608 drives attached to a single Root Port with an interposer in-between?

I assume the Root Port needs to know that it's bifurcated and has to appear as two slots on the bus. Is this configured with a BIOS setting?

Quoted 7 lines
Quoted 4 lines

If these assumptions are true, the quirk isn't really specific to the P5608 but should rather apply to the bifurcation-capable Root Port and the quirk should set the flag if the Root Port is indeed configured for bifurcation.

It's a function of the slot + card combination, but we can't distinguish this slot's special power handling behavior from the vanilla behavior. It's modified to handle power on the logically bifurcated, singular physical device.

Quoted 7 lines

Hi Bjorn, Lukas,

I need to resubmit this.

Besides the 'pdev->shared_pcc_and_link_slot = false', addition mentioned above, is there anything else that should be changed or any reason this wouldn't be accepted?

Another report has cropped up of spurious DLLSC events:
https://bugzilla.kernel.org/show_bug.cgi?id=216511

That other case differs from yours in that a spurious DLLSC event is seen on plugging *in* a card, whereas in your case the event seems to occur on *removing* a card. In both cases, the spurious event is seen on the hotplug port's sibling.

I'm starting to think that we should probably disable DLLSC events entirely if they're known to be unreliable. The hotplug port solely relies on PDC events then. Otherwise we'd have to clutter the event handling with all sorts of special cases. The code would become fairly difficult to follow.

I've attached an experimental patch to the bug report which disables DLLSC events on a hotplug port if a P5608 SSD is plugged into it:
https://bugzilla.kernel.org/attachment.cgi?id=301845

Would this approach work for you?

One other question: What if the SSD is not bifurcated (i.e. x8 instead of x4x4), don't we need avoid applying the quirk in that case? Your patch doesn't seem to do that. Can we recognize somehow whether the card is bifurcated or not? Is it sufficient to just look at the Maximum Link Width in the Link Capabilities Register? Does the SSD report x4 there if it's bifurcated?

Thanks,

Lukas

#9 lore raw
Quoted 59 lines · Lukas Wunner
Quoted 42 lines · Jonathan Derrick
Quoted 33 lines · Jon Derrick
Quoted 32 lines · Lukas Wunner
Quoted 16 lines · Jon Derrick
Quoted 10 lines · Lukas Wunner
Quoted 8 lines · Jon Derrick

When an Intel P5608 SSD is bifurcated into x4x4 mode, and the upstream ports both support hotplugging on each respective x4 device, a slot management system for the SSD requires both x4 slots to have power removed via sysfs (echo 0 > slot/N/power), from the OS before it can safely turn-off physical power for the whole x8 device. The implications are that slot status will display powered off and link inactive statuses for the x4 devices where the devices are actually powered until both ports have powered off.

Quoted 4 lines

Just to get a better understanding, does the P5608 have an internal PCIe switch with hotplug capability on the Downstream Ports or does it plug into two separate PCIe slots? I recall previous patches mentioned a CEM interposer? (An lspci listing might be helpful.)

Quoted 8 lines

It looks like 2 NVMe endpoints plugged into two different root ports, ex, 80:00.0 Root port to [81-86]
80:01.0 Root port to [87-8b]
81:00.0 NVMe
87:00.0 NVMe

The x8 is bifurcated to x4x4. Physically they share the same slot power/clock/reset but are logically separate per root port.

So are these two P5608 drives attached to a single Root Port with an interposer in-between?

I assume the Root Port needs to know that it's bifurcated and has to appear as two slots on the bus. Is this configured with a BIOS setting?

Quoted 7 lines
Quoted 4 lines

If these assumptions are true, the quirk isn't really specific to the P5608 but should rather apply to the bifurcation-capable Root Port and the quirk should set the flag if the Root Port is indeed configured for bifurcation.

It's a function of the slot + card combination, but we can't distinguish this slot's special power handling behavior from the vanilla behavior. It's modified to handle power on the logically bifurcated, singular physical device.

Quoted 7 lines

Hi Bjorn, Lukas,

I need to resubmit this.

Besides the 'pdev->shared_pcc_and_link_slot = false', addition mentioned above, is there anything else that should be changed or any reason this wouldn't be accepted?

Another report has cropped up of spurious DLLSC events:
https://bugzilla.kernel.org/show_bug.cgi?id=216511

That other case differs from yours in that a spurious DLLSC event is seen on plugging *in* a card, whereas in your case the event seems to occur on *removing* a card. In both cases, the spurious event is seen on the hotplug port's sibling.

Quoted 5 lines

I'm starting to think that we should probably disable DLLSC events entirely if they're known to be unreliable. The hotplug port solely relies on PDC events then. Otherwise we'd have to clutter the event handling with all sorts of special cases. The code would become fairly difficult to follow.

I'm not sure we can do that either. Think of a non-logical interposer. PDC will be static but DLLSC may be the only hotplug status received.

Quoted 6 lines

I've attached an experimental patch to the bug report which disables DLLSC events on a hotplug port if a P5608 SSD is plugged into it:
https://bugzilla.kernel.org/attachment.cgi?id=301845

Would this approach work for you?

It looks correct to me

Quoted 7 lines

One other question: What if the SSD is not bifurcated (i.e. x8 instead of x4x4), don't we need avoid applying the quirk in that case? Your patch doesn't seem to do that. Can we recognize somehow whether the card is bifurcated or not? Is it sufficient to just look at the Maximum Link Width in the Link Capabilities Register? Does the SSD report x4 there if it's bifurcate

Good question. Unique to the subsystem device id?

Quoted 4 lines

Thanks,

Lukas

#10 lore raw
In reply to Lukas Wunner
Quoted 54 lines · Lukas Wunner
Quoted 42 lines · Jonathan Derrick
Quoted 33 lines · Jon Derrick
Quoted 32 lines · Lukas Wunner
Quoted 16 lines · Jon Derrick
Quoted 10 lines · Lukas Wunner
Quoted 8 lines · Jon Derrick

When an Intel P5608 SSD is bifurcated into x4x4 mode, and the upstream ports both support hotplugging on each respective x4 device, a slot management system for the SSD requires both x4 slots to have power removed via sysfs (echo 0 > slot/N/power), from the OS before it can safely turn-off physical power for the whole x8 device. The implications are that slot status will display powered off and link inactive statuses for the x4 devices where the devices are actually powered until both ports have powered off.

Quoted 4 lines

Just to get a better understanding, does the P5608 have an internal PCIe switch with hotplug capability on the Downstream Ports or does it plug into two separate PCIe slots? I recall previous patches mentioned a CEM interposer? (An lspci listing might be helpful.)

Quoted 8 lines

It looks like 2 NVMe endpoints plugged into two different root ports, ex, 80:00.0 Root port to [81-86]
80:01.0 Root port to [87-8b]
81:00.0 NVMe
87:00.0 NVMe

The x8 is bifurcated to x4x4. Physically they share the same slot power/clock/reset but are logically separate per root port.

So are these two P5608 drives attached to a single Root Port with an interposer in-between?

I assume the Root Port needs to know that it's bifurcated and has to appear as two slots on the bus. Is this configured with a BIOS setting?

Quoted 7 lines
Quoted 4 lines

If these assumptions are true, the quirk isn't really specific to the P5608 but should rather apply to the bifurcation-capable Root Port and the quirk should set the flag if the Root Port is indeed configured for bifurcation.

It's a function of the slot + card combination, but we can't distinguish this slot's special power handling behavior from the vanilla behavior. It's modified to handle power on the logically bifurcated, singular physical device.

Quoted 7 lines

Hi Bjorn, Lukas,

I need to resubmit this.

Besides the 'pdev->shared_pcc_and_link_slot = false', addition mentioned above, is there anything else that should be changed or any reason this wouldn't be accepted?

Another report has cropped up of spurious DLLSC events:
https://bugzilla.kernel.org/show_bug.cgi?id=216511

Quoted 8 lines

That other case differs from yours in that a spurious DLLSC event is seen on plugging *in* a card, whereas in your case the event seems to occur on *removing* a card. In both cases, the spurious event is seen on the hotplug port's sibling.

I'm starting to think that we should probably disable DLLSC events entirely if they're known to be unreliable. The hotplug port solely relies on PDC events then. Otherwise we'd have to clutter

But when we have ATTN, we don't subscribe to PDC events correct? Can we always rely on PDC changes?

Unless you have a PowerControl, PDC can also be spurious correct? and so will be DLLSC.. But I agreem, the hotplug code is getting hairy complex and can help some simplification. A long time ago I remember I worked out a state diagram in google draw for how these states transition to guide the code, but I don't remember where they went.

Quoted 19 lines

the event handling with all sorts of special cases. The code would become fairly difficult to follow.

I've attached an experimental patch to the bug report which disables DLLSC events on a hotplug port if a P5608 SSD is plugged into it:
https://bugzilla.kernel.org/attachment.cgi?id=301845

Would this approach work for you?

One other question: What if the SSD is not bifurcated (i.e. x8 instead of x4x4), don't we need avoid applying the quirk in that case? Your patch doesn't seem to do that. Can we recognize somehow whether the card is bifurcated or not? Is it sufficient to just look at the Maximum Link Width in the Link Capabilities Register? Does the SSD report x4 there if it's bifurcated?

Thanks,

Lukas