UI becomes unresponsive every couple of seconds

Bug 215533 Bjorn Helgaas 2 participants 4 messages Jan 26, 2022 – Feb 10, 2022 1 message not in the archive PCI

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

Bjorn Helgaas

author

2022-01-26 12:12 UTC #1 lore raw

[+cc Jan, author of 0e8ae5a6ff59, linux-pci]

Quoted 52 lines · On Wed, Jan 26, 2022 at 08:18:12AM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=215533
--- Comment #1 from [email protected] ---
I accidentally sent the report prematurely. So here come my findings:
Since 5.16
(1) my system becomes unresponsive every couple of seconds (micro lags), which makes it more or less unusable.
(2) wrong(?) CPU frequencies are reported.
- 5.15 works fine.
- Starting from some commit in 5.17, it seems (1) is fixed (unsure), but definitely not (2).
I have bisected the kernel between 5.15 and 5.16, and found that the offending commit is 0e8ae5a6ff5952253cd7cc0260df838ab4c21009 ("PCI/portdrv: Do not setup up IRQs if there are no users"). Bisection log attached.
Reverting this commit on linux-git[1] fixes both (1) and (2).
Important notes:
- This regression was reported on a DELL XPS 9550 laptop by two users [2], so it might be related strictly to that model.
- According to user mallocman, the issue can also be fixed by reverting the BIOS version of the laptop to v1.12.
- The issue ONLY occurs when AC is plugged in (and stays there even when I unplug it).
- When booting on battery power, there is no issue at all.
You can easily observe the regression via:
watch cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_fre
As soon as I plug in AC, all frequencies go up to values around 3248338 and stay there even if I unplug AC. This does not happen at all when booted on battery power.
Also note:
- the laptop's fans are not really affected by the high frequencies.
- setting the governor to "powersave" has no effect on the frequencies (as compared to when on battery power).
- lowering the maximum frequency manually works, but does not fix (1).
[1] https://aur.archlinux.org/pkgbase/linux-git/ (pulled commits up to 0280e3c58f92b2fe0e8fbbdf8d386449168de4a8).
[2] https://bbs.archlinux.org/viewtopic.php?id=273330
Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

Bjorn Helgaas

author

2022-02-07 22:45 UTC #2 lore raw

[+cc linux-kernel for visibility]

Quoted 44 lines · Bjorn Helgaas

Quoted 43 lines · On Wed, Jan 26, 2022 at 08:18:12AM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=215533
--- Comment #1 from [email protected] ---
I accidentally sent the report prematurely. So here come my findings:
Since 5.16
(1) my system becomes unresponsive every couple of seconds (micro lags), which makes it more or less unusable.
(2) wrong(?) CPU frequencies are reported.
- 5.15 works fine.
- Starting from some commit in 5.17, it seems (1) is fixed (unsure), but definitely not (2).
I have bisected the kernel between 5.15 and 5.16, and found that the offending commit is 0e8ae5a6ff5952253cd7cc0260df838ab4c21009 ("PCI/portdrv: Do not setup up IRQs if there are no users"). Bisection log attached.
Reverting this commit on linux-git[1] fixes both (1) and (2).
Important notes:
- This regression was reported on a DELL XPS 9550 laptop by two users [2], so it might be related strictly to that model.
- According to user mallocman, the issue can also be fixed by reverting the BIOS version of the laptop to v1.12.
- The issue ONLY occurs when AC is plugged in (and stays there even when I unplug it).
- When booting on battery power, there is no issue at all.
You can easily observe the regression via:
watch cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_fre
As soon as I plug in AC, all frequencies go up to values around 3248338 and stay there even if I unplug AC. This does not happen at all when booted on battery power.
Also note:
- the laptop's fans are not really affected by the high frequencies.
- setting the governor to "powersave" has no effect on the frequencies (as compared to when on battery power).
- lowering the maximum frequency manually works, but does not fix (1).

Quoted 3 lines

Quoted 3 lines
[1] https://aur.archlinux.org/pkgbase/linux-git/ (pulled commits up to 0280e3c58f92b2fe0e8fbbdf8d386449168de4a8).
[2] https://bbs.archlinux.org/viewtopic.php?id=273330

I hope we can find a better solution, but since the responsiveness issue is a significant regression, I queued up a revert of 0e8ae5a6ff59 ("PCI/portdrv: Do not setup up IRQs if there are no users") in case we don't find one.

If/when we get to the bottom of this, I'll replace the revert with the solution. 0e8ae5a6ff59 appeared in v5.16, so we'll have to make sure we fix that as well.

Bjorn

Jan Kiszka

2022-02-08 06:35 UTC #3 lore raw

Quoted 47 lines · Bjorn Helgaas

[+cc linux-kernel for visibility]
Quoted 44 lines · Bjorn Helgaas
Quoted 43 lines · On Wed, Jan 26, 2022 at 08:18:12AM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=215533
--- Comment #1 from [email protected] ---
I accidentally sent the report prematurely. So here come my findings:
Since 5.16
(1) my system becomes unresponsive every couple of seconds (micro lags), which makes it more or less unusable.
(2) wrong(?) CPU frequencies are reported.
- 5.15 works fine.
- Starting from some commit in 5.17, it seems (1) is fixed (unsure), but definitely not (2).
I have bisected the kernel between 5.15 and 5.16, and found that the offending commit is 0e8ae5a6ff5952253cd7cc0260df838ab4c21009 ("PCI/portdrv: Do not setup up IRQs if there are no users"). Bisection log attached.
Reverting this commit on linux-git[1] fixes both (1) and (2).
Important notes:
- This regression was reported on a DELL XPS 9550 laptop by two users [2], so it might be related strictly to that model.
- According to user mallocman, the issue can also be fixed by reverting the BIOS version of the laptop to v1.12.
- The issue ONLY occurs when AC is plugged in (and stays there even when I unplug it).
- When booting on battery power, there is no issue at all.
You can easily observe the regression via:
watch cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_fre
As soon as I plug in AC, all frequencies go up to values around 3248338 and stay there even if I unplug AC. This does not happen at all when booted on battery power.
Also note:
- the laptop's fans are not really affected by the high frequencies.
- setting the governor to "powersave" has no effect on the frequencies (as compared to when on battery power).
- lowering the maximum frequency manually works, but does not fix (1).

Quoted 8 lines

Quoted 3 lines
Quoted 3 lines
[1] https://aur.archlinux.org/pkgbase/linux-git/ (pulled commits up to 0280e3c58f92b2fe0e8fbbdf8d386449168de4a8).
[2] https://bbs.archlinux.org/viewtopic.php?id=273330
I hope we can find a better solution, but since the responsiveness issue is a significant regression, I queued up a revert of 0e8ae5a6ff59 ("PCI/portdrv: Do not setup up IRQs if there are no users") in case we don't find one.

Likely best for now.

Quoted 4 lines

If/when we get to the bottom of this, I'll replace the revert with the solution. 0e8ae5a6ff59 appeared in v5.16, so we'll have to make sure we fix that as well.

If you could give some feedback/hints on the questions I posted last week on the original patch, that might accelerate understanding the real issue.

Thanks,
Jan

Signature · 2 lines

Siemens AG, Technology
Competence Center Embedded Linux

Bjorn Helgaas

author

2022-02-10 22:27 UTC #4 lore raw

In reply to Bjorn Helgaas

Quoted 48 lines · Bjorn Helgaas

Quoted 47 lines · Bjorn Helgaas
Quoted 46 lines · On Wed, Jan 26, 2022 at 08:18:12AM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=215533
--- Comment #1 from [email protected] ---
I accidentally sent the report prematurely. So here come my findings:
Since 5.16
(1) my system becomes unresponsive every couple of seconds (micro lags), which makes it more or less unusable.
(2) wrong(?) CPU frequencies are reported.
- 5.15 works fine.
- Starting from some commit in 5.17, it seems (1) is fixed (unsure), but definitely not (2).
I have bisected the kernel between 5.15 and 5.16, and found that the offending commit is 0e8ae5a6ff5952253cd7cc0260df838ab4c21009 ("PCI/portdrv: Do not setup up IRQs if there are no users"). Bisection log attached.
Reverting this commit on linux-git[1] fixes both (1) and (2).
Important notes:
- This regression was reported on a DELL XPS 9550 laptop by two users [2], so it might be related strictly to that model.
- According to user mallocman, the issue can also be fixed by reverting the BIOS version of the laptop to v1.12.
- The issue ONLY occurs when AC is plugged in (and stays there even when I unplug it).
- When booting on battery power, there is no issue at all.
You can easily observe the regression via:
watch cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_fre
As soon as I plug in AC, all frequencies go up to values around 3248338 and stay there even if I unplug AC. This does not happen at all when booted on battery power.
Also note:
- the laptop's fans are not really affected by the high frequencies.
- setting the governor to "powersave" has no effect on the frequencies (as compared to when on battery power).
- lowering the maximum frequency manually works, but does not fix (1).

Quoted 8 lines

Quoted 3 lines
Quoted 3 lines
[1] https://aur.archlinux.org/pkgbase/linux-git/ (pulled commits up to 0280e3c58f92b2fe0e8fbbdf8d386449168de4a8).
[2] https://bbs.archlinux.org/viewtopic.php?id=273330
I hope we can find a better solution, but since the responsiveness issue is a significant regression, I queued up a revert of 0e8ae5a6ff59 ("PCI/portdrv: Do not setup up IRQs if there are no users") in case we don't find one.

Here's the revert I plan to merge for v5.17. As mentioned at [3], I think the problem is a BIOS defect that leaves interrupts enabled when handing off to the kernel.

Bjorn

[3] https://lore.kernel.org/r/20220208185658.GA489586@bhelgaas

commit b139e2632409 ("Revert "PCI/portdrv: Do not setup up IRQs if there are no users"")

Author: Bjorn Helgaas <[email protected]>
Date:   Mon Feb 7 16:33:30 2022 -0600

Revert "PCI/portdrv: Do not setup up IRQs if there are no users"

This reverts commit 0e8ae5a6ff5952253cd7cc0260df838ab4c21009.

0e8ae5a6ff59 ("PCI/portdrv: Do not setup up IRQs if there are no users") reduced usage of IRQs when we don't think we need them. But several people reported unresponsive systems and bisected it to this commit.

Revert 0e8ae5a6ff59 until we figure out a better solution.

Reported-by Joey Corleone <[email protected]>

Reported-by Sergiu Deitsch <[email protected]>

Reported-by David Spencer <[email protected]>

Link https://bugzilla.kernel.org/show_bug.cgi?id=215533

Signed-off-by Bjorn Helgaas <[email protected]>

Cc [email protected] # v5.16+

Cc Jan Kiszka <[email protected]>

drivers/pci/pcie/portdrv_core.c +17 ~~-30~~

index bda630889f95..604feeb84ee4 100644
@@ -166,6 +166,9 @@ static int pcie_init_service_irqs(struct pci_dev *dev, int *irqs, int mask)
 {
 	int ret, i;
 
+	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
+		irqs[i] = -1;
+
 	/*
 	 * If we support PME but can't use MSI/MSI-X for it, we have to
 	 * fall back to INTx or other interrupts, e.g., a system shared

@@ -314,10 +317,8 @@ static int pcie_device_init(struct pci_dev *pdev, int service, int irq)
  */
 int pcie_port_device_register(struct pci_dev *dev)
 {
-	int status, capabilities, irq_services, i, nr_service;
-	int irqs[PCIE_PORT_DEVICE_MAXSERVICES] = {
-		[0 ... PCIE_PORT_DEVICE_MAXSERVICES-1] = -1
-	};
+	int status, capabilities, i, nr_service;
+	int irqs[PCIE_PORT_DEVICE_MAXSERVICES];
 
 	/* Enable PCI Express port device */
 	status = pci_enable_device(dev);

@@ -330,32 +331,18 @@ int pcie_port_device_register(struct pci_dev *dev)
 		return 0;
 
 	pci_set_master(dev);
-
-	irq_services = 0;
-	if (IS_ENABLED(CONFIG_PCIE_PME))
-		irq_services |= PCIE_PORT_SERVICE_PME;
-	if (IS_ENABLED(CONFIG_PCIEAER))
-		irq_services |= PCIE_PORT_SERVICE_AER;
-	if (IS_ENABLED(CONFIG_HOTPLUG_PCI_PCIE))
-		irq_services |= PCIE_PORT_SERVICE_HP;
-	if (IS_ENABLED(CONFIG_PCIE_DPC))
-		irq_services |= PCIE_PORT_SERVICE_DPC;
-	irq_services &= capabilities;
-
-	if (irq_services) {
-		/*
-		 * Initialize service IRQs. Don't use service devices that
-		 * require interrupts if there is no way to generate them.
-		 * However, some drivers may have a polling mode (e.g.
-		 * pciehp_poll_mode) that can be used in the absence of IRQs.
-		 * Allow them to determine if that is to be used.
-		 */
-		status = pcie_init_service_irqs(dev, irqs, irq_services);
-		if (status) {
-			irq_services &= PCIE_PORT_SERVICE_HP;
-			if (!irq_services)
-				goto error_disable;
-		}
+	/*
+	 * Initialize service irqs. Don't use service devices that
+	 * require interrupts if there is no way to generate them.
+	 * However, some drivers may have a polling mode (e.g. pciehp_poll_mode)
+	 * that can be used in the absence of irqs.  Allow them to determine
+	 * if that is to be used.
+	 */
+	status = pcie_init_service_irqs(dev, irqs, capabilities);
+	if (status) {
+		capabilities &= PCIE_PORT_SERVICE_HP;
+		if (!capabilities)
+			goto error_disable;
 	}
 
 	/* Allocate child services if any */