New: ehci-hcd.c ISR
https://bugzilla.kernel.org/show_bug.cgi?id=214789
Bug ID: 214789
Summary: ehci-hcd.c ISR
Product: Drivers
Version: 2.5
Kernel Version: 5.11+
Hardware: x86-64
OS: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: USB
Assignee: [email protected]
Reporter: [email protected]
Regression: NoChange in ehci_irq from spin_lock_irqsave/irqrestore to spin_lock/unlock broke shared IRQ's
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Quoted 3 lines · On Thu, Oct 21, 2021 at 04:29:42AM +0000, [email protected]…
Change in
ehci_irqfrom spin_lock_irqsave/irqrestore to spin_lock/unlock broke shared IRQ's
What shared irq broke exactly? For what hardware platform? And what kernel version worked for you and now does not work?
thanks,
greg k-h
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #1 from Greg Kroah-Hartman ([email protected]) ---
Quoted 4 lines · On Thu, Oct 21, 2021 at 04:29:42AM +0000, [email protected]…
Change in
ehci_irqfrom spin_lock_irqsave/irqrestore to spin_lock/unlock broke
shared IRQ's
What shared irq broke exactly? For what hardware platform? And what kernel version worked for you and now does not work?
thanks,
greg k-h
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #2 from Scott Arnold ([email protected]) --- Hello,
The driver is for a Symmetricom bc635pcie Time and Frequency Processor. Driver developed in house for ISS simulations.
Works fine on 5.3 and prior kernels.
Works fine with 5.11 if not using shared IRQ's. No interrupts with shared IRQ's.
Machine is a HP dl580 G9 w/ 72 Intel(R) Xeon(R) CPU E7-8867 v4 @ 2.40GHz. We only have a few of these machines the rest are G10's and work fine (not sharing IRQ's with timing cards).
I am having the card moved to another slot to try and get a unshared IRQ. Unfortunately MSI does not seem possible with this card.
Thanks
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, October 21, 2021 12:56 AM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #1 from Greg Kroah-Hartman ([email protected]) --- On Thu, Oct 21, 2021 at 04:29:42AM +0000, [email protected]
Quoted 3 lines · wrote:
Change in
ehci_irqfrom spin_lock_irqsave/irqrestore to spin_lock/unlock broke shared IRQ's
What shared irq broke exactly? For what hardware platform? And what kernel version worked for you and now does not work?
thanks,
greg k-h
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #3 from Alan Stern ([email protected]) ---
Can you provide more information about exactly what goes wrong? And why shared IRQs should make any difference?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #4 from Scott Arnold ([email protected]) --- Hello,
Timing card is not receiving interrupts when sharing a IRQ (16) with the ehci-hcd driver and the 5.11.17 (also tried 5.14.13) kernel.
I changed the spin_lock/spin_unlock back to spin_lock_irqsave/irqrestore in ehci_irq in the 5.14.13 kernel and timing card now getting interrupts as expected.
The 5.3 (and probably prior although I have not checked) are using the irqsave/restore versions of spinlock in ehci_irq.
From 5.3:
/*
* For threadirqs option we use spin_lock_irqsave() variant to prevent
* deadlock with ehci hrtimer callback, because hrtimer callbacks run
* in interrupt context even when threadirqs is specified. We can go
* back to spin_lock() variant when hrtimer callbacks become threaded. */
spin_lock_irqsave(&ehci->lock, flags);
Thanks
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, October 21, 2021 11:03 AM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #3 from Alan Stern ([email protected]) --- Can you provide more information about exactly what goes wrong? And why shared IRQs should make any difference?
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #5 from Alan Stern ([email protected]) --- Okay, but _why_ don't the timing card's interrupts get handled when ehci_irq uses spin_lock? And _why_ does changing to spin_lock_irqsave make a difference?
Do all of the card's interrupt requests get lost or only some of them?
Are you somehow getting recursive (nested) interrupts for the same IRQ line?
Is ehci_irq somehow getting called with interrupts enabled?
I don't want to make any changes to the driver until we know the answers to these questions.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #6 from Scott Arnold ([email protected]) --- Hello,
I don't know why it makes a difference but according to /proc/interrupts IRQ16 gets about 90 interrupts and stops, when working properly card generates 240 interrupts/second.
Works fine with irqsave/restore in the ehci-hcd isr.
Ehci-hcd is built in and not a module in our configuration.
I am having the card moved to another slot now.
Thanks
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, October 21, 2021 2:28 PM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #5 from Alan Stern ([email protected]) --- Okay, but _why_ don't the timing card's interrupts get handled when ehci_irq uses spin_lock? And _why_ does changing to spin_lock_irqsave make a difference?
Do all of the card's interrupt requests get lost or only some of them?
Are you somehow getting recursive (nested) interrupts for the same IRQ line?
Is ehci_irq somehow getting called with interrupts enabled?
I don't want to make any changes to the driver until we know the answers to these questions.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #7 from Scott Arnold ([email protected]) --- Anything you want me to try while I have the hardware available?
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, October 21, 2021 2:28 PM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #5 from Alan Stern ([email protected]) --- Okay, but _why_ don't the timing card's interrupts get handled when ehci_irq uses spin_lock? And _why_ does changing to spin_lock_irqsave make a difference?
Do all of the card's interrupt requests get lost or only some of them?
Are you somehow getting recursive (nested) interrupts for the same IRQ line?
Is ehci_irq somehow getting called with interrupts enabled?
I don't want to make any changes to the driver until we know the answers to these questions.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #8 from Alan Stern ([email protected]) --- How many of those 90 interrupts were issued by the EHCI host controller as opposed to the card?
Are there any USB devices attached to the host controller? What does /sys/kernel/debug/usb/devices have to say?
Does the card use edge-triggered interrupts rather than level-triggered?
Have you tried adding any debugging messages to ehci_irq to find out what's going on when it runs?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #9 from Scott Arnold ([email protected]) --- Hello,
The timing card driver receives no interrupts according to debug.
IRQ16: IO-APIC 16-fasteoi ehci_hcd:usb1, hpilo, rt_pcclk
Ehci_hcd entries in /sys/kernel/debug/usb/devices:
T: Bus=01 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=480 MxCh= 2 B: Alloc= 0/800 us ( 0%), #Int= 4, #Iso= 0 D: Ver= 2.00 Cls=09(hub ) Sub=00 Prot=00 MxPS=64 #Cfgs= 1 P: Vendor=1d6b ProdID=0002 Rev= 5.11 S: Manufacturer=Linux 5.11.17_OBCS_1 ehci_hcd S: Product=EHCI Host Controller S: SerialNumber=0000:00:1a.0
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr= 0mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub
E: Ad=81(I) Atr=03(Int.) MxPS= 4 Ivl=256ms
T: Bus=02 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=480 MxCh= 2 B: Alloc= 0/800 us ( 0%), #Int= 0, #Iso= 0 D: Ver= 2.00 Cls=09(hub ) Sub=00 Prot=00 MxPS=64 #Cfgs= 1 P: Vendor=1d6b ProdID=0002 Rev= 5.11 S: Manufacturer=Linux 5.11.17_OBCS_1 ehci_hcd S: Product=EHCI Host Controller S: SerialNumber=0000:00:1d.0
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr= 0mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub
E: Ad=81(I) Atr=03(Int.) MxPS= 4 Ivl=256ms
I have not tried enabling debug in ehci_hcd yet. Moving card to another slot did not help, still sharing IRQ 16
Thanks
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, October 21, 2021 2:46 PM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #8 from Alan Stern ([email protected]) --- How many of those 90 interrupts were issued by the EHCI host controller as opposed to the card?
Are there any USB devices attached to the host controller? What does /sys/kernel/debug/usb/devices have to say?
Does the card use edge-triggered interrupts rather than level-triggered?
Have you tried adding any debugging messages to ehci_irq to find out what's going on when it runs?
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 214789] ehci-hcd.c ISR
Quoted 6 lines · On Thu, Oct 21, 2021 at 08:04:06PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #9 from Scott Arnold ([email protected]) --- Hello,
The timing card driver receives no interrupts according to debug.
IRQ16: IO-APIC 16-fasteoi ehci_hcd:usb1, hpilo,rt_pcclk
So the IRQ is also shared with some (mainline) HP management-processor driver (hpilo).
Quoted 9 lines
Ehci_hcdentries in/sys/kernel/debug/usb/devices:T: Bus=01 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=480 MxCh= 2 B: Alloc= 0/800 us ( 0%), #Int= 4, #Iso= 0 D: Ver= 2.00 Cls=09(hub ) Sub=00 Prot=00 MxPS=64 #Cfgs= 1 P: Vendor=1d6b ProdID=0002 Rev= 5.11 S: Manufacturer=Linux 5.11.17_OBCS_1 ehci_hcd S: Product=EHCI Host Controller S: SerialNumber=0000:00:1a.0
Quoted 3 lines
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr= 0mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub
E: Ad=81(I) Atr=03(Int.) MxPS= 4 Ivl=256ms
Just to be clear: Are there any USB devices physically connected to this bus?
What does "lsusb -s1:" say?
Quoted 1 line
I have not tried enabling debug in
ehci_hcdyet.
Try adding
WARN_ON_ONCE(!irqs_disabled());
at the start of ehci_irq() before grabbing the lock.
That should give us a stack dump in case there's someone calling the interrupt handler with interrupts enabled (which seems to be the case).
Johan
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #10 from Johan Hovold ([email protected]) ---
Quoted 6 lines · On Thu, Oct 21, 2021 at 08:04:06PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #9 from Scott Arnold ([email protected]) --- Hello,
The timing card driver receives no interrupts according to debug.
IRQ16: IO-APIC 16-fasteoi ehci_hcd:usb1, hpilo,rt_pcclk
So the IRQ is also shared with some (mainline) HP management-processor driver (hpilo).
Quoted 9 lines
Ehci_hcdentries in/sys/kernel/debug/usb/devices:T: Bus=01 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=480 MxCh= 2 B: Alloc= 0/800 us ( 0%), #Int= 4, #Iso= 0 D: Ver= 2.00 Cls=09(hub ) Sub=00 Prot=00 MxPS=64 #Cfgs= 1 P: Vendor=1d6b ProdID=0002 Rev= 5.11 S: Manufacturer=Linux 5.11.17_OBCS_1 ehci_hcd S: Product=EHCI Host Controller S: SerialNumber=0000:00:1a.0
Quoted 3 lines
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr= 0mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub
E: Ad=81(I) Atr=03(Int.) MxPS= 4 Ivl=256ms
Just to be clear: Are there any USB devices physically connected to this bus?
What does "lsusb -s1:" say?
Quoted 1 line
I have not tried enabling debug in
ehci_hcdyet.
Try adding
WARN_ON_ONCE(!irqs_disabled());
at the start of ehci_irq() before grabbing the lock.
That should give us a stack dump in case there's someone calling the interrupt handler with interrupts enabled (which seems to be the case).
Johan
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 214789] ehci-hcd.c ISR
[ Adding back bugzilla and linux-usb on CC. ]
Quoted 2 lines · INC]
I added the WARN_ON_ONCE(!irqs_disabled()); at the beginning of ehci-irq before the lock and did not notice anything.
Ok, so interrupts are already disabled as they should be.
Quoted 62 lines
However after looking at the logs I discovered:
[ 5.189043] irq 16: nobody cared (try booting with the "irqpoll" option) [ 5.189112] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.14.13_OBCS_1 #4 [ 5.189180] Hardware name: HP ProLiant DL580 Gen9/ProLiant DL580 Gen9, BIOS U17 01/22/2018 [ 5.189261] Call Trace: [ 5.189324] <IRQ> [ 5.189381] ? dump_stack_lvl+0x33/0x42 [ 5.189445] ? __report_bad_irq+0x32/0xac [ 5.189505] ? note_interrupt.cold.11+0xa/0x63 [ 5.189562] ? handle_irq_event_percpu+0x65/0x70 [ 5.189623] ? handle_irq_event+0x32/0x50 [ 5.189681] ? handle_fasteoi_irq+0xa1/0x160 [ 5.189740] ? __common_interrupt+0x3c/0xa0 [ 5.189798] ? common_interrupt+0x7a/0xa0 [ 5.189859] </IRQ> [ 5.189913] ? asm_common_interrupt+0x1b/0x40 [ 5.189973] ? mwait_idle+0x50/0x70 [ 5.190031] ? default_idle+0x10/0x10 [ 5.190088] ? default_idle_call+0x1f/0x30 [ 5.190147] ? do_idle+0x1df/0x1f0 [ 5.190207] ? cpu_startup_entry+0x14/0x20 [ 5.190267] ? start_kernel+0x616/0x63d [ 5.190328] ? secondary_startup_64_no_verify+0xb0/0xbb [ 5.190388] handlers: [ 5.190442] [<00000000da7aaaea>] usb_hcd_irq [ 5.190504] Disabling IRQ #16[ 5.201827] irq 23: nobody cared (try booting with the "irqpoll" option) [ 5.201885] CPU: 1 PID: 8 Comm: kworker/u145:0 Not tainted 5.14.13_OBCS_1 #4 [ 5.201942] Hardware name: HP ProLiant DL580 Gen9/ProLiant DL580 Gen9, BIOS U17 01/22/2018 [ 5.202010] Workqueue: events_unbound async_run_entry_fn [ 5.202069] Call Trace: [ 5.202119] <IRQ> [ 5.202168] ? dump_stack_lvl+0x33/0x42 [ 5.202223] ? __report_bad_irq+0x32/0xac [ 5.202277] ? note_interrupt.cold.11+0xa/0x63 [ 5.202332] ? handle_irq_event_percpu+0x65/0x70 [ 5.202386] ? handle_irq_event+0x32/0x50 [ 5.202441] ? handle_fasteoi_irq+0xa1/0x160 [ 5.202495] ? __common_interrupt+0x3c/0xa0 [ 5.202548] ? common_interrupt+0x7a/0xa0 [ 5.202603] </IRQ> [ 5.202652] ? asm_common_interrupt+0x1b/0x40 [ 5.202707] ? inflate_fast+0x118/0x5e0 [ 5.202764] ? zlib_inflate+0x3d1/0x1770 [ 5.202817] ? do_copy+0xed/0x109 [ 5.202869] ? write_buffer+0x22/0x32 [ 5.202921] ? initrd_load+0x268/0x268 [ 5.202975] ? write_buffer+0x32/0x32 [ 5.203026] ? __gunzip+0x244/0x310 [ 5.203083] ? decompress_method+0x3c/0x3c [ 5.203137] ? initrd_load+0x268/0x268 [ 5.203190] ? gunzip+0xe/0x11 [ 5.203243] ? initrd_load+0x268/0x268 [ 5.203296] ? unpack_to_rootfs+0x14f/0x285 [ 5.203349] ? initrd_load+0x268/0x268 [ 5.203402] ? do_populate_rootfs+0x6c/0x160 [ 5.203455] ? async_run_entry_fn+0x1b/0xa0 [ 5.203508] ? process_one_work+0x1d1/0x330
Quoted 8 lines
[ 5.203563] ? worker_thread+0x28/0x3d0 [ 5.203615] ? mod_delayed_work_on+0x90/0x90 [ 5.203668] ? kthread+0x120/0x150 [ 5.203720] ? set_kthread_struct+0x30/0x30 [ 5.203773] ? ret_from_fork+0x22/0x30 [ 5.203826] handlers: [ 5.203875] [<00000000da7aaaea>] usb_hcd_irq [ 5.203930] Disabling IRQ #23
So this happens also for another EHCI bus IRQ. Is this IRQ also shared with something?
Quoted 28 lines
[ 62.407444] irq 16: nobody cared (try booting with the "irqpoll" option) [ 62.407474] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.14.13_OBCS_1 #4 [ 62.407499] Hardware name: HP ProLiant DL580 Gen9/ProLiant DL580 Gen9, BIOS U17 01/22/2018 [ 62.407527] Call Trace: [ 62.407538] <IRQ> [ 62.407547] ? dump_stack_lvl+0x33/0x42 [ 62.407569] ? __report_bad_irq+0x32/0xac [ 62.407588] ? note_interrupt.cold.11+0xa/0x63 [ 62.407606] ? handle_irq_event_percpu+0x65/0x70 [ 62.407626] ? handle_irq_event+0x32/0x50 [ 62.407642] ? handle_fasteoi_irq+0xa1/0x160 [ 62.408250] ? __common_interrupt+0x3c/0xa0 [ 62.408820] ? common_interrupt+0x7a/0xa0 [ 62.409386] </IRQ> [ 62.409934] ? asm_common_interrupt+0x1b/0x40 [ 62.410483] ? mwait_idle+0x50/0x70 [ 62.411026] ? default_idle+0x10/0x10 [ 62.411565] ? default_idle_call+0x1f/0x30 [ 62.412102] ? do_idle+0x1df/0x1f0 [ 62.412634] ? cpu_startup_entry+0x14/0x20 [ 62.413164] ? start_kernel+0x616/0x63d [ 62.413694] ? secondary_startup_64_no_verify+0xb0/0xbb [ 62.414218] handlers: [ 62.414734] [<00000000da7aaaea>] usb_hcd_irq [ 62.415257] [<000000008857253d>] ilo_isr [hpilo] [ 62.415775] Disabling IRQ #16
Quoted 3 lines
There is one usb device " Bus 001 Device 003: ID 14dd:1007 Raritan Computer, Inc. D2CIM-VUSB KVM connector" and it has disappeared (i.e.not working)
Thanks for confirming.
Quoted 1 line
This does not occur without the irqsave/restore in the ehci-hcd.
Now why would simply saving the interrupt state in ehci_irq() prevent these spurious IRQs? There's something fishy going on alright.
Quoted 1 line
My timercard driver is not loaded. This is with a 5.14.13 kernel.
Are you able to reproduce this on a machine without the timer card present at all?
Quoted 6 lines
Lsusb -s1:
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hubLsb -s2 and -s3 are blank.
Looks like you forgot the colon in "lsusb -s1:" so this lists the devices with number 1 instead of the devices connected to bus 1.
Quoted 8 lines
On another identical machine (almost has 92 cores instead of 72) running 5.3.6 kernel:
Lsusb -s1:Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Quoted 8 lines
Lsusb -s2:
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching HubLsusb -s3:
Bus 002 Device 003: ID 0424:2660 Microchip Technology, Inc. (formerly SMSC) Hub Bus 001 Device 003: ID 14dd:1007 Raritan Computer, Inc. D2CIM-VUSB KVM connector
Ok, but there is a device connected to bus 1 as you also mentioned above.
Quoted 4 lines · INC]
Just as another datapoint I put the
ehci-hcd.cfile from the 5.3.6 kernel into the 5.14.13 kernel.
No more "nobody cared" messages but neither timer card or USB is working now.
Yeah, that probably not going to work.
Quoted 11 lines
[ 6.798509] usb 2-1: new high-speed USB device number 4 using ehci-pci [ 6.798586] usb 1-1: new high-speed USB device number 4 using ehci-pci [ 7.238498] usb 1-1: device not accepting address 4, error -32 [ 7.238562] usb 2-1: device not accepting address 4, error -32 [ 7.388499] usb 1-1: new high-speed USB device number 5 using ehci-pci [ 7.388561] usb 2-1: new high-speed USB device number 5 using ehci-pci [ 7.828496] usb 1-1: device not accepting address 5, error -32 [ 7.828557] usb 2-1: device not accepting address 5, error -32 [ 7.828618] usb usb1-port1: unable to enumerate USB device [ 7.828678] usb usb2-port1: unable to enumerate USB device
Quoted 3 lines
/proc/interruptsfor IRQ 16 is stuck at 50.Some combination of these two may solve problem.
It would be good if we could rule out the timer card being involved in this (e.g. since the driver is out of tree).
Johan
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #11 from Johan Hovold ([email protected]) --- [ Adding back bugzilla and linux-usb on CC. ]
Quoted 2 lines · INC]
I added the WARN_ON_ONCE(!irqs_disabled()); at the beginning of ehci-irq before the lock and did not notice anything.
Ok, so interrupts are already disabled as they should be.
Quoted 65 lines
However after looking at the logs I discovered:
[ 5.189043] irq 16: nobody cared (try booting with the "irqpoll" option) [ 5.189112] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.14.13_OBCS_1 #4 [ 5.189180] Hardware name: HP ProLiant DL580 Gen9/ProLiant DL580 Gen9, BIOS U17 01/22/2018 [ 5.189261] Call Trace: [ 5.189324] <IRQ> [ 5.189381] ? dump_stack_lvl+0x33/0x42 [ 5.189445] ? __report_bad_irq+0x32/0xac [ 5.189505] ? note_interrupt.cold.11+0xa/0x63 [ 5.189562] ? handle_irq_event_percpu+0x65/0x70 [ 5.189623] ? handle_irq_event+0x32/0x50 [ 5.189681] ? handle_fasteoi_irq+0xa1/0x160 [ 5.189740] ? __common_interrupt+0x3c/0xa0 [ 5.189798] ? common_interrupt+0x7a/0xa0 [ 5.189859] </IRQ> [ 5.189913] ? asm_common_interrupt+0x1b/0x40 [ 5.189973] ? mwait_idle+0x50/0x70 [ 5.190031] ? default_idle+0x10/0x10 [ 5.190088] ? default_idle_call+0x1f/0x30 [ 5.190147] ? do_idle+0x1df/0x1f0 [ 5.190207] ? cpu_startup_entry+0x14/0x20 [ 5.190267] ? start_kernel+0x616/0x63d [ 5.190328] ? secondary_startup_64_no_verify+0xb0/0xbb [ 5.190388] handlers: [ 5.190442] [<00000000da7aaaea>] usb_hcd_irq [ 5.190504] Disabling IRQ #16[ 5.201827] irq 23: nobody cared (try booting with the "irqpoll" option) [ 5.201885] CPU: 1 PID: 8 Comm: kworker/u145:0 Not tainted 5.14.13_OBCS_1 #4 [ 5.201942] Hardware name: HP ProLiant DL580 Gen9/ProLiant DL580 Gen9, BIOS U17 01/22/2018 [ 5.202010] Workqueue: events_unbound async_run_entry_fn [ 5.202069] Call Trace: [ 5.202119] <IRQ> [ 5.202168] ? dump_stack_lvl+0x33/0x42 [ 5.202223] ? __report_bad_irq+0x32/0xac [ 5.202277] ? note_interrupt.cold.11+0xa/0x63 [ 5.202332] ? handle_irq_event_percpu+0x65/0x70 [ 5.202386] ? handle_irq_event+0x32/0x50 [ 5.202441] ? handle_fasteoi_irq+0xa1/0x160 [ 5.202495] ? __common_interrupt+0x3c/0xa0 [ 5.202548] ? common_interrupt+0x7a/0xa0 [ 5.202603] </IRQ> [ 5.202652] ? asm_common_interrupt+0x1b/0x40 [ 5.202707] ? inflate_fast+0x118/0x5e0 [ 5.202764] ? zlib_inflate+0x3d1/0x1770 [ 5.202817] ? do_copy+0xed/0x109 [ 5.202869] ? write_buffer+0x22/0x32 [ 5.202921] ? initrd_load+0x268/0x268 [ 5.202975] ? write_buffer+0x32/0x32 [ 5.203026] ? __gunzip+0x244/0x310 [ 5.203083] ? decompress_method+0x3c/0x3c [ 5.203137] ? initrd_load+0x268/0x268 [ 5.203190] ? gunzip+0xe/0x11 [ 5.203243] ? initrd_load+0x268/0x268 [ 5.203296] ? unpack_to_rootfs+0x14f/0x285 [ 5.203349] ? initrd_load+0x268/0x268 [ 5.203402] ? do_populate_rootfs+0x6c/0x160 [ 5.203455] ? async_run_entry_fn+0x1b/0xa0 [ 5.203508] ? process_one_work+0x1d1/0x330
Quoted 8 lines
[ 5.203563] ? worker_thread+0x28/0x3d0 [ 5.203615] ? mod_delayed_work_on+0x90/0x90 [ 5.203668] ? kthread+0x120/0x150 [ 5.203720] ? set_kthread_struct+0x30/0x30 [ 5.203773] ? ret_from_fork+0x22/0x30 [ 5.203826] handlers: [ 5.203875] [<00000000da7aaaea>] usb_hcd_irq [ 5.203930] Disabling IRQ #23
So this happens also for another EHCI bus IRQ. Is this IRQ also shared with something?
Quoted 29 lines
[ 62.407444] irq 16: nobody cared (try booting with the "irqpoll" option) [ 62.407474] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.14.13_OBCS_1 #4 [ 62.407499] Hardware name: HP ProLiant DL580 Gen9/ProLiant DL580 Gen9, BIOS U17 01/22/2018 [ 62.407527] Call Trace: [ 62.407538] <IRQ> [ 62.407547] ? dump_stack_lvl+0x33/0x42 [ 62.407569] ? __report_bad_irq+0x32/0xac [ 62.407588] ? note_interrupt.cold.11+0xa/0x63 [ 62.407606] ? handle_irq_event_percpu+0x65/0x70 [ 62.407626] ? handle_irq_event+0x32/0x50 [ 62.407642] ? handle_fasteoi_irq+0xa1/0x160 [ 62.408250] ? __common_interrupt+0x3c/0xa0 [ 62.408820] ? common_interrupt+0x7a/0xa0 [ 62.409386] </IRQ> [ 62.409934] ? asm_common_interrupt+0x1b/0x40 [ 62.410483] ? mwait_idle+0x50/0x70 [ 62.411026] ? default_idle+0x10/0x10 [ 62.411565] ? default_idle_call+0x1f/0x30 [ 62.412102] ? do_idle+0x1df/0x1f0 [ 62.412634] ? cpu_startup_entry+0x14/0x20 [ 62.413164] ? start_kernel+0x616/0x63d [ 62.413694] ? secondary_startup_64_no_verify+0xb0/0xbb [ 62.414218] handlers: [ 62.414734] [<00000000da7aaaea>] usb_hcd_irq [ 62.415257] [<000000008857253d>] ilo_isr [hpilo] [ 62.415775] Disabling IRQ #16
Quoted 3 lines
There is one usb device " Bus 001 Device 003: ID 14dd:1007 Raritan Computer, Inc. D2CIM-VUSB KVM connector" and it has disappeared (i.e.not working)
Thanks for confirming.
Quoted 1 line
This does not occur without the irqsave/restore in the ehci-hcd.
Now why would simply saving the interrupt state in ehci_irq() prevent these spurious IRQs? There's something fishy going on alright.
Quoted 1 line
My timercard driver is not loaded. This is with a 5.14.13 kernel.
Are you able to reproduce this on a machine without the timer card present at all?
Quoted 6 lines
Lsusb -s1:
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hubLsb -s2 and -s3 are blank.
Looks like you forgot the colon in "lsusb -s1:" so this lists the devices with number 1 instead of the devices connected to bus 1.
Quoted 12 lines
On another identical machine (almost has 92 cores instead of 72) running 5.3.6 kernel:
Lsusb -s1:Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb -s2:
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Quoted 6 lines
Lsusb -s3:
Bus 002 Device 003: ID 0424:2660 Microchip Technology, Inc. (formerly SMSC) Hub
Bus 001 Device 003: ID 14dd:1007 Raritan Computer, Inc. D2CIM-VUSB KVM connector
Ok, but there is a device connected to bus 1 as you also mentioned above.
Quoted 4 lines · INC]
Just as another datapoint I put the
ehci-hcd.cfile from the 5.3.6 kernel into the 5.14.13 kernel.
No more "nobody cared" messages but neither timer card or USB is working now.
Yeah, that probably not going to work.
Quoted 11 lines
[ 6.798509] usb 2-1: new high-speed USB device number 4 using ehci-pci [ 6.798586] usb 1-1: new high-speed USB device number 4 using ehci-pci [ 7.238498] usb 1-1: device not accepting address 4, error -32 [ 7.238562] usb 2-1: device not accepting address 4, error -32 [ 7.388499] usb 1-1: new high-speed USB device number 5 using ehci-pci [ 7.388561] usb 2-1: new high-speed USB device number 5 using ehci-pci [ 7.828496] usb 1-1: device not accepting address 5, error -32 [ 7.828557] usb 2-1: device not accepting address 5, error -32 [ 7.828618] usb usb1-port1: unable to enumerate USB device [ 7.828678] usb usb2-port1: unable to enumerate USB device
Quoted 3 lines
/proc/interruptsfor IRQ 16 is stuck at 50.Some combination of these two may solve problem.
It would be good if we could rule out the timer card being involved in this (e.g. since the driver is out of tree).
Johan
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #12 from Scott Arnold ([email protected]) --- Hello,
Sorry for the late reply, Outlook sometimes put emails in "Other" for some reason.
I just reverted that machine back to the 5.3.6 kernel.
Now IRQ 16 has:
16: IO-APIC 16-fasteoi ehci_hcd:usb1, uhci_hcd:usb3, hpilo, rt_pcclk
"uhci_hcd:usb3" does not appear with the 5.11+ kernels (with or without rt_pcclk), maybe the issue is with uhci_hcd.
Timer card works fine in this config.
Thanks
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Monday, October 25, 2021 8:20 AM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #11 from Johan Hovold ([email protected]) --- [ Adding back bugzilla and linux-usb on CC. ]
Quoted 2 lines · INC]
I added the WARN_ON_ONCE(!irqs_disabled()); at the beginning of ehci-irq before the lock and did not notice anything.
Ok, so interrupts are already disabled as they should be.
Quoted 65 lines
However after looking at the logs I discovered:
[ 5.189043] irq 16: nobody cared (try booting with the "irqpoll" option) [ 5.189112] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.14.13_OBCS_1 #4 [ 5.189180] Hardware name: HP ProLiant DL580 Gen9/ProLiant DL580 Gen9, BIOS U17 01/22/2018 [ 5.189261] Call Trace: [ 5.189324] <IRQ> [ 5.189381] ? dump_stack_lvl+0x33/0x42 [ 5.189445] ? __report_bad_irq+0x32/0xac [ 5.189505] ? note_interrupt.cold.11+0xa/0x63 [ 5.189562] ? handle_irq_event_percpu+0x65/0x70 [ 5.189623] ? handle_irq_event+0x32/0x50 [ 5.189681] ? handle_fasteoi_irq+0xa1/0x160 [ 5.189740] ? __common_interrupt+0x3c/0xa0 [ 5.189798] ? common_interrupt+0x7a/0xa0 [ 5.189859] </IRQ> [ 5.189913] ? asm_common_interrupt+0x1b/0x40 [ 5.189973] ? mwait_idle+0x50/0x70 [ 5.190031] ? default_idle+0x10/0x10 [ 5.190088] ? default_idle_call+0x1f/0x30 [ 5.190147] ? do_idle+0x1df/0x1f0 [ 5.190207] ? cpu_startup_entry+0x14/0x20 [ 5.190267] ? start_kernel+0x616/0x63d [ 5.190328] ? secondary_startup_64_no_verify+0xb0/0xbb [ 5.190388] handlers: [ 5.190442] [<00000000da7aaaea>] usb_hcd_irq [ 5.190504] Disabling IRQ #16[ 5.201827] irq 23: nobody cared (try booting with the "irqpoll" option) [ 5.201885] CPU: 1 PID: 8 Comm: kworker/u145:0 Not tainted 5.14.13_OBCS_1 #4 [ 5.201942] Hardware name: HP ProLiant DL580 Gen9/ProLiant DL580 Gen9, BIOS U17 01/22/2018 [ 5.202010] Workqueue: events_unbound async_run_entry_fn [ 5.202069] Call Trace: [ 5.202119] <IRQ> [ 5.202168] ? dump_stack_lvl+0x33/0x42 [ 5.202223] ? __report_bad_irq+0x32/0xac [ 5.202277] ? note_interrupt.cold.11+0xa/0x63 [ 5.202332] ? handle_irq_event_percpu+0x65/0x70 [ 5.202386] ? handle_irq_event+0x32/0x50 [ 5.202441] ? handle_fasteoi_irq+0xa1/0x160 [ 5.202495] ? __common_interrupt+0x3c/0xa0 [ 5.202548] ? common_interrupt+0x7a/0xa0 [ 5.202603] </IRQ> [ 5.202652] ? asm_common_interrupt+0x1b/0x40 [ 5.202707] ? inflate_fast+0x118/0x5e0 [ 5.202764] ? zlib_inflate+0x3d1/0x1770 [ 5.202817] ? do_copy+0xed/0x109 [ 5.202869] ? write_buffer+0x22/0x32 [ 5.202921] ? initrd_load+0x268/0x268 [ 5.202975] ? write_buffer+0x32/0x32 [ 5.203026] ? __gunzip+0x244/0x310 [ 5.203083] ? decompress_method+0x3c/0x3c [ 5.203137] ? initrd_load+0x268/0x268 [ 5.203190] ? gunzip+0xe/0x11 [ 5.203243] ? initrd_load+0x268/0x268 [ 5.203296] ? unpack_to_rootfs+0x14f/0x285 [ 5.203349] ? initrd_load+0x268/0x268 [ 5.203402] ? do_populate_rootfs+0x6c/0x160 [ 5.203455] ? async_run_entry_fn+0x1b/0xa0 [ 5.203508] ? process_one_work+0x1d1/0x330
Quoted 8 lines
[ 5.203563] ? worker_thread+0x28/0x3d0 [ 5.203615] ? mod_delayed_work_on+0x90/0x90 [ 5.203668] ? kthread+0x120/0x150 [ 5.203720] ? set_kthread_struct+0x30/0x30 [ 5.203773] ? ret_from_fork+0x22/0x30 [ 5.203826] handlers: [ 5.203875] [<00000000da7aaaea>] usb_hcd_irq [ 5.203930] Disabling IRQ #23
So this happens also for another EHCI bus IRQ. Is this IRQ also shared with something?
Quoted 29 lines
[ 62.407444] irq 16: nobody cared (try booting with the "irqpoll" option) [ 62.407474] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.14.13_OBCS_1 #4 [ 62.407499] Hardware name: HP ProLiant DL580 Gen9/ProLiant DL580 Gen9, BIOS U17 01/22/2018 [ 62.407527] Call Trace: [ 62.407538] <IRQ> [ 62.407547] ? dump_stack_lvl+0x33/0x42 [ 62.407569] ? __report_bad_irq+0x32/0xac [ 62.407588] ? note_interrupt.cold.11+0xa/0x63 [ 62.407606] ? handle_irq_event_percpu+0x65/0x70 [ 62.407626] ? handle_irq_event+0x32/0x50 [ 62.407642] ? handle_fasteoi_irq+0xa1/0x160 [ 62.408250] ? __common_interrupt+0x3c/0xa0 [ 62.408820] ? common_interrupt+0x7a/0xa0 [ 62.409386] </IRQ> [ 62.409934] ? asm_common_interrupt+0x1b/0x40 [ 62.410483] ? mwait_idle+0x50/0x70 [ 62.411026] ? default_idle+0x10/0x10 [ 62.411565] ? default_idle_call+0x1f/0x30 [ 62.412102] ? do_idle+0x1df/0x1f0 [ 62.412634] ? cpu_startup_entry+0x14/0x20 [ 62.413164] ? start_kernel+0x616/0x63d [ 62.413694] ? secondary_startup_64_no_verify+0xb0/0xbb [ 62.414218] handlers: [ 62.414734] [<00000000da7aaaea>] usb_hcd_irq [ 62.415257] [<000000008857253d>] ilo_isr [hpilo] [ 62.415775] Disabling IRQ #16
Quoted 3 lines
There is one usb device " Bus 001 Device 003: ID 14dd:1007 Raritan Computer, Inc. D2CIM-VUSB KVM connector" and it has disappeared (i.e.not working)
Thanks for confirming.
Quoted 1 line
This does not occur without the irqsave/restore in the ehci-hcd.
Now why would simply saving the interrupt state in ehci_irq() prevent these spurious IRQs? There's something fishy going on alright.
Quoted 1 line
My timercard driver is not loaded. This is with a 5.14.13 kernel.
Are you able to reproduce this on a machine without the timer card present at all?
Quoted 6 lines
Lsusb -s1:
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hubLsb -s2 and -s3 are blank.
Looks like you forgot the colon in "lsusb -s1:" so this lists the devices with number 1 instead of the devices connected to bus 1.
Quoted 14 lines
On another identical machine (almost has 92 cores instead of 72) running
5.3.6 kernel:
Lsusb -s1:Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb -s2:
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Quoted 5 lines
Lsusb -s3:
Bus 002 Device 003: ID 0424:2660 Microchip Technology, Inc. (formerly SMSC) Hub Bus 001 Device 003: ID 14dd:1007 Raritan Computer, Inc. D2CIM-VUSB KVM connector
Ok, but there is a device connected to bus 1 as you also mentioned above.
Quoted 4 lines · INC]
Just as another datapoint I put the
ehci-hcd.cfile from the 5.3.6 kernel into the 5.14.13 kernel.
No more "nobody cared" messages but neither timer card or USB is working now.
Yeah, that probably not going to work.
Quoted 11 lines
[ 6.798509] usb 2-1: new high-speed USB device number 4 using ehci-pci [ 6.798586] usb 1-1: new high-speed USB device number 4 using ehci-pci [ 7.238498] usb 1-1: device not accepting address 4, error -32 [ 7.238562] usb 2-1: device not accepting address 4, error -32 [ 7.388499] usb 1-1: new high-speed USB device number 5 using ehci-pci [ 7.388561] usb 2-1: new high-speed USB device number 5 using ehci-pci [ 7.828496] usb 1-1: device not accepting address 5, error -32 [ 7.828557] usb 2-1: device not accepting address 5, error -32 [ 7.828618] usb usb1-port1: unable to enumerate USB device [ 7.828678] usb usb2-port1: unable to enumerate USB device
Quoted 3 lines
/proc/interruptsfor IRQ 16 is stuck at 50.Some combination of these two may solve problem.
It would be good if we could rule out the timer card being involved in this (e.g. since the driver is out of tree).
Johan
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #13 from Alan Stern ([email protected]) --- What does the kernel log from a recent kernel say about UHCI? If it is present in the older kernel then it should still show up in the recent kernel.
And if the problem is related to uhci-hcd, why would patching ehci-hcd make it go away?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #14 from Scott Arnold ([email protected]) --- Hello,
Sorry for the confusion.
While patching echi-hcd.c using meld to change spin_locks " current_status = ehci_readl(ehci, &ehci->regs->status);" got changed to "status = ehci_readl(ehci, &ehci->regs->status);" (~line 723).
As a result current_status was used un-initialized (I would have thought gcc 8.3.1 would have warned about that).
This caused the "nobody cared" messages on IRQ's 16 and 23. So ehci-hcd did not load/allocate the IRQ's.Spin_locks are not the problem.
If ehci-hcd does not load the timer card works fine.
If ehci-hcd loads the timer card will not work even if ehci-hcd is unloaded. IRQ count on 16 stuck at ~90 in /proc/interrupts.
Seems like IRQ 16 is somehow left disabled. Could be why uhci-hcd does not load ?
All works fine with the 5.3.6 kernel, broken in 5.11+.
Lsudb with 5.3.6:
Bus 002 Device 003: ID 0424:2660 Microchip Technology, Inc. (formerly SMSC) Hub Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 003: ID 14dd:1007 Raritan Computer, Inc. D2CIM-VUSB KVM connector
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Thanks
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, October 26, 2021 9:06 PM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #13 from Alan Stern ([email protected]) --- What does the kernel log from a recent kernel say about UHCI? If it is present in the older kernel then it should still show up in the recent kernel.
And if the problem is related to uhci-hcd, why would patching ehci-hcd make it go away?
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #15 from Alan Stern ([email protected]) --- Can you try testing some kernel versions between 5.3 and 5.11 to see exactly at what point the problem was introduced?
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #16 from Scott Arnold ([email protected]) --- Hello,
Not easily, the machine is back in general development use. I'll see what I can do.
Thanks
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Wednesday, October 27, 2021 3:07 PM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #15 from Alan Stern ([email protected]) --- Can you try testing some kernel versions between 5.3 and 5.11 to see exactly at what point the problem was introduced?
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #17 from Scott Arnold ([email protected]) --- Hello,
5.4 and 5.9 look the most suspicious based on release notes. I will try those when I get the OK.
Thanks
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Wednesday, October 27, 2021 3:07 PM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #15 from Alan Stern ([email protected]) --- Can you try testing some kernel versions between 5.3 and 5.11 to see exactly at what point the problem was introduced?
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #18 from Scott Arnold ([email protected]) --- Hello,
So far 5.9.1 is broken, 5.4.1 is ok.
I will try 5.8 and 5.5 next.
Thanks
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Wednesday, October 27, 2021 3:07 PM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #15 from Alan Stern ([email protected]) --- Can you try testing some kernel versions between 5.3 and 5.11 to see exactly at what point the problem was introduced?
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #19 from Scott Arnold ([email protected]) --- Hello,
5.6.1 kernel OK.
5.7.1+ kernel not OK
Thanks
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Wednesday, October 27, 2021 3:07 PM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #15 from Alan Stern ([email protected]) --- Can you try testing some kernel versions between 5.3 and 5.11 to see exactly at what point the problem was introduced?
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #20 from Alan Stern ([email protected]) --- There were no significant changes at all to the ehci-hcd driver between 5.6.1 and 5.7.1. Which indicates that the cause of the problem lies somewhere else in the kernel.
At this point, your best approach would be to carry out a git bisect between those two kernel versions. Or maybe just between 5.6 and 5.7 (I assume that 5.6 is okay, just like 5.6.1, and 5.7 is bad, just like 5.7.1). That would let you identify the exact commit where the problem started.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #21 from Scott Arnold ([email protected]) --- Hello,
I noticed there was not much diff in drivers/usb using meld on 5.6.1 and 5.7.1. Git not easy due to network restrictions.
I am going to see if 4.8.5 compiler vs 8.3.1 makes any difference. Also going to check and see if any bios settings could re-arrange the IRQ's to work around problem.
Thanks
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Monday, November 1, 2021 2:41 PM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #20 from Alan Stern ([email protected]) --- There were no significant changes at all to the ehci-hcd driver between 5.6.1 and 5.7.1. Which indicates that the cause of the problem lies somewhere else in the kernel.
At this point, your best approach would be to carry out a git bisect between those two kernel versions. Or maybe just between 5.6 and 5.7 (I assume that 5.6 is okay, just like 5.6.1, and 5.7 is bad, just like 5.7.1). That would let you identify the exact commit where the problem started.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #22 from Scott Arnold ([email protected]) --- Hello,
This caused problem:
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Monday, November 1, 2021 2:41 PM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #20 from Alan Stern ([email protected]) --- There were no significant changes at all to the ehci-hcd driver between 5.6.1 and 5.7.1. Which indicates that the cause of the problem lies somewhere else in the kernel.
At this point, your best approach would be to carry out a git bisect between those two kernel versions. Or maybe just between 5.6 and 5.7 (I assume that 5.6 is okay, just like 5.6.1, and 5.7 is bad, just like 5.7.1). That would let you identify the exact commit where the problem started.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #23 from Scott Arnold ([email protected]) --- pci=noioapicquirk fixes it.
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Monday, November 1, 2021 2:41 PM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
--- Comment #20 from Alan Stern ([email protected]) --- There were no significant changes at all to the ehci-hcd driver between 5.6.1 and 5.7.1. Which indicates that the cause of the problem lies somewhere else in the kernel.
At this point, your best approach would be to carry out a git bisect between those two kernel versions. Or maybe just between 5.6 and 5.7 (I assume that 5.6 is okay, just like 5.6.1, and 5.7 is bad, just like 5.7.1). That would let you identify the exact commit where the problem started.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 214789] ehci-hcd.c ISR
In reply to [email protected]Quoted 7 lines · On Fri, Nov 05, 2021 at 05:10:05PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #22 from Scott Arnold ([email protected]) --- Hello,
This caused problem:
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
Scott
This is commit b88bf6c3b6ff ("PCI: Add boot interrupt quirk mechanism for Xeon chipsets").
Sean and linux-pci readers, please take a look at this bug report (Bugzilla #214789).
Alan Stern
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #24 from Alan Stern ([email protected]) ---
Quoted 8 lines · On Fri, Nov 05, 2021 at 05:10:05PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #22 from Scott Arnold ([email protected]) --- Hello,
This caused problem:https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
Scott
This is commit b88bf6c3b6ff ("PCI: Add boot interrupt quirk mechanism for Xeon chipsets").
Sean and linux-pci readers, please take a look at this bug report (Bugzilla #214789).
Alan Stern
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 214789] ehci-hcd.c ISR
In reply to Alan Stern[+CC Sean using his other e-mail address]
Hi Alan,
Quoted 8 lines
Quoted 7 lines
https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #22 from Scott Arnold ([email protected]) --- Hello,
This caused problem:
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
Scott
Quoted 5 lines
This is commit
b88bf6c3b6ff("PCI: Add boot interrupt quirk mechanism for Xeon chipsets").Sean and linux-pci readers, please take a look at this bug report (Bugzilla #214789).
Sean might not have been able to see this message, as he is no longer working for Intel and his old e-mail would bounce.
I am added Sean using a different one I've found, perhaps it would work.
Krzysztof
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #25 from Krzysztof Wilczyński ([email protected]) --- [+CC Sean using his other e-mail address]
Hi Alan,
Quoted 6 lines
Quoted 6 lines
https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #22 from Scott Arnold ([email protected]) --- Hello,
This caused problem:
Quoted 8 lines
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
Quoted 1 line
Scott
This is commit
b88bf6c3b6ff("PCI: Add boot interrupt quirk mechanism for Xeon chipsets").Sean and linux-pci readers, please take a look at this bug report (Bugzilla #214789).
Sean might not have been able to see this message, as he is no longer working for Intel and his old e-mail would bounce.
I am added Sean using a different one I've found, perhaps it would work.
Krzysztof
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
Sean V Kelley ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #26 from Sean V Kelley ([email protected]) --- Hi,
Thanks for reaching out Krzysztof.
So this platform is a Xeon E7-8867. which makes it a broadwell based Xeon.
I don't see it in the logs, what is the reported device ID? Those Xeon have the capability disable the route to the ICH, which is these quirks are meant to work with.
It also looks like you are using an out of tree driver? Is that correct?
Are you unable to use MSI with this device?
Sean
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #27 from Sean V Kelley ([email protected]) --- It just seems like I'm with this particular vendor card and out of tree driver, there is some sort of expectation that the IRQ will always be routed to the ICH/PCH, which the quirk is attempting to prevent to avoid disabling valid interrupts with the rerouting. Hence the question on MSI.
Sean
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #28 from Sean V Kelley ([email protected]) --- (In reply to Scott Arnold from comment #23)
Quoted 2 lines
pci=noioapicquirk fixes it.
As mentioned in the patch originally:
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
+The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable +(or disable) the redirection of the interrupt handler to the PCH interrupt +line. The option can be overridden by either pci=ioapicreroute or +pci=noioapicreroute.[3]
+[3] https://lore.kernel.org/lkml/[email protected]/
In the absence of the ability to convert your out-of-tree driver for this hardware to MSI, then can you give pci=noioapicreroute a try?
Sean
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #29 from Sean V Kelley ([email protected]) --- (In reply to Sean V Kelley from comment #28)
Quoted 17 lines
(In reply to Scott Arnold from comment #23)
Quoted 2 lines
pci=noioapicquirk fixes it.
As mentioned in the patch originally:
https://patchwork.kernel.org/project/linux-pci/patch/20200214213313.66622-3-
[email protected]/(quoted diff) +5
-0+The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable +(or disable) the redirection of the interrupt handler to the PCH interrupt +line. The option can be overridden by either pci=ioapicreroute or +pci=noioapicreroute.[3] +[3] https://lore.kernel.org/lkml/[email protected]/
Quoted 2 lines
In the absence of the ability to convert your out-of-tree driver for this hardware to MSI, then can you give pci=noioapicreroute a try?
Sorry, full of typos today. I'm wondering if you could give pci=ioapicreroute a try.
I no longer work at Intel but am happy to help try to resolve your issue as best I can.
Best regards,
Sean
Quoted 3 lines
Sean
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 214789] ehci-hcd.c ISR
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=214789
--- Comment #30 from Scott Arnold ([email protected]) --- Hello,
The device does not appear to support MSI. pci_enable_msi and pci_alloc_irq_vectors fail.
From lspci (devices uses a bridge chip):
87:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode])
Physical Slot: 5
Flags: bus master, fast devsel, latency 0, IRQ 16, NUMA node 2
Bus: primary=87, secondary=88, subordinate=88, sec-latency=64 I/O behind bridge: 0000a000-0000afff
Memory behind bridge: ca100000-ca1fffff
Capabilities: [40] Power Management version 2
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [60] Express PCI-Express to PCI/PCI-X Bridge, MSI 00
Capabilities: [100] Power Budgeting <?>
88:00.0 Signal processing controller: Datum Inc. Bancomm-Timing Division Device 4013 (rev 20)
Subsystem: Datum Inc. Bancomm-Timing Division Device 4013
Flags: medium devsel, IRQ 16, NUMA node 2
I/O ports at a000 [size=128]
Memory at ca101000 (32-bit, non-prefetchable) [size=2K]
Memory at ca100000 (32-bit, non-prefetchable) [size=2K]This is a custom driver for our simulation environment.
I will have to schedule time on the machine to try pci=noioapicreroute.
Scott
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Friday, December 3, 2021 6:03 PM
To: Arnold, Scott C. (JSC-CD13)[SGT, INC] <[email protected]>
Subject: [EXTERNAL] [Bug 214789] ehci-hcd.c ISR
Sean V Kelley ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #26 from Sean V Kelley ([email protected]) --- Hi,
Thanks for reaching out Krzysztof.
So this platform is a Xeon E7-8867. which makes it a broadwell based Xeon.
I don't see it in the logs, what is the reported device ID? Those Xeon have the capability disable the route to the ICH, which is these quirks are meant to work with.
It also looks like you are using an out of tree driver? Is that correct?
Are you unable to use MSI with this device?
Sean
--
You may reply to this email to add a comment.
You are receiving this mail because:
You reported the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.