Kernel oops from pci_disable_msi
A message that is not in the archive (sent off-list or not yet received): 528a1bb6.6a88700a.28c9.ffff824aSMTPIN_ADDED_MISSING@mx.google.com
A message that is not in the archive (sent off-list or not yet received): CAErSpo4TbUuq0wb06JV9Xchmcjsk9q3cm7+XO-dOSiJAAhXPMA@mail.gmail.com
Bjorn Helgaas
maintainer[+cc linux-pci for real]
Quoted 100 lines · Bjorn Helgaas
[+cc linux-pci]
Quoted 5 lines · Bjørn Erik Nilsen
I just hit an kernel oops related to PCI (in dw_msi_teardown_irq()/clear_irq() (pcie-designware))
Linux version 3.12.0-next-20131105 (bnilsen@bnilsen) (gcc version 4.7.2 (GCC) )
Problem seem to be dereferencing a null pointer returned from
irq_desc_get_msi_desc(desc)(see attached backtrace).Included oops inline for ease of viewing/searching. Jingooo, I assume you'll investigate this. Let me know if otherwise.
Unable to handle kernel NULL pointer dereference at virtual address 00000020 pgd = 80004000 [00000020] *pgd=00000000 Internal error: Oops: 17 [#1] SMP ARM Modules linked in: sxdma(O) CPU: 1 PID: 569 Comm: i2cipc.B3 Tainted: G O 3.12.0-next-20131105 #8 task: 9efcb600 ti: 9ec8c000 task.ti: 9ec8c000 PC is at dw_msi_teardown_irq+0x40/0x118 LR is at trace_hardirqs_on_caller+0xf4/0x1c0 pc : [<802c405c>] lr : [<80078370>] psr: 20000013sp :
9ec8dca8ip :9ec8dbe0fp :9ec8dccc
r10:9fa58800r9 : 00000000 r8 : 00000001
r7 :9f9fecc0r6 : 00000188 r5 :9ecdb580r4 : 00000000 r3 :9efcb600r2 : 00000000 r1 : 00000001 r0 : 00000012
Flags: nzCv IRQs on FIQs on ModeSVC_32ISA ARM Segment user
Control:10c5387dTable:2ed1804aDAC: 00000015 Process i2cipc.B3 (pid: 569, stack limit =0x9ec8c240)Stack: (0x9ec8dca8 to 0x9ec8e000)dca0:
8005d1a4802844c8000001889ecdb580000000049fa58c74
dcc0:9ec8dce49ec8dcd0802c1844802c402800000000 000000029ec8dd0c9ec8dce8
dce0:802c1a08802c181400000000 000000049fa58c60000000049fa58c7400000001
dd00:9ec8dd1c9ec8dd10802c1a34802c19ac9ec8dd449ec8dd20802c1ad0802c1a30
dd20:9fa588008084a41000047c8c0000016c0000016c000000089ec8dd5c9ec8dd48
dd40:802c2570802c1a44000000009d5800009ec8dd749ec8dd607f0057d4802c2534
dd60: 000000009d5800009ec8dd9c9ec8dd787f000a387f00579c7f0007b89f3fa200
dd80:9f5a2bd09f0c9cd09f383d509f383d509ec8dddc9ec8dda0800d4d407f0007c4
dda0: 00000000 00000000800cf4789f3fa2089f05b9c09efcb8f880891d409e3aab00
ddc0:9efcb6009ec8c0009ec8c0009d4502549ec8ddec9ec8dde0800d4f18800d4cb4
dde0:9ec8de0c9ec8ddf08003d7cc800d4f149efcb908000000019efcb6009d450200
de00:9ec8de449ec8de10800254648003d72c9ec8de3c00000001 800783708007814c
de20:9e0b26049ec8decc9ec8c0009e0b26049ec8de4c9e0b26049ec8de649ec8de48
de40:80025b9c800251cc9ec8c0009ec8decc9ec8c0009e0b26049ec8deb49ec8de68
de60: 8003149080025b6480073404 800733309ec8de949ec8de80805faf2c9ec8deb8
de80:418004fc9d46e6809ec8debc9ec8dfb09ec8c000fffffdfc9ec8deb876d533d0
dea0:9ec8c00076d533d49ec8df7c9ec8deb88001153c80031278 800783708007814c
dec0: 400000139f0727288083ffc000000009 00000000 00000000 00000000 00000000
dee0:9ec8df049ec8def0805faf1c800784489f07240081215fc09ec8df2c9ec8df08
df00:8004c6dc805faef0000000009f072400000000006deff4c8000003e200000000
df20:9ec8df8c9ec8c00080011ab89efcb600000000019ec8dfb09ec8c00000000000
df40:9ec8df6c9ec8df50800783708007814c9ec8c0009ec8c0009ec8c0009ec8c000
df60:8000ea649ec8dfb09ec8c000000000009ec8dfac9ec8df8080011ad4800112fc
df80: 00000000 000000018000e8ac000000000341cc1800000001000000a88000ea64
dfa0: 000000009ec8dfb08000e8f880011a5874ffeaa000000002 ffffffff012f7314
dfc0: 000000000341cc1800000001000000a874ffeb980341cd800341bcf074ffece4
dfe0: 0000000074ffea7076d533c476d533d08000001074ffeaa000000000 00000000Backtrace: [<802c401c>] (dw_msi_teardown_irq+0x0/0x118) from [<802c1844>] (arch_teardown_msi_irq+0x3c/0x40) r7:9fa58c74 r6:00000004 r5:9ecdb580 r4:00000188 [<802c1808>] (arch_teardown_msi_irq+0x0/0x40) from [<802c1a08>] (default_teardown_msi_irqs+0x68/0x84) r4:00000002 r3:00000000 [<802c19a0>] (default_teardown_msi_irqs+0x0/0x84) from [<802c1a34>] (arch_teardown_msi_irqs+0x10/0x14) r8:00000001 r7:9fa58c74 r6:00000004 r5:9fa58c60 r4:00000004r3:00000000
[<802c1a24>] (arch_teardown_msi_irqs+0x0/0x14) from [<802c1ad0>] (free_msi_irqs+0x98/0x144)
[<802c1a38>] (free_msi_irqs+0x0/0x144) from [<802c2570>] (pci_disable_msi+0x48/0x60)
[<802c2528>] (pci_disable_msi+0x0/0x60) from [<7f0057d4>] (sxdma_irq_free+0x44/0x48[sxdma])
r4:9d580000 r3:00000000
[<7f005790>] (sxdma_irq_free+0x0/0x48[sxdma]) from [<7f000a38>] (sxdma_dev_release+0x280/0x2b0[sxdma])
r4:9d580000 r3:00000000
[<7f0007b8>] (sxdma_dev_release+0x0/0x2b0[sxdma]) from [<800d4d40>] (__fput+0x98/0x214)
r8:9f383d50 r7:9f383d50 r6:9f0c9cd0 r5:9f5a2bd0 r4:9f3fa200
r3:7f0007b8
[<800d4ca8>] (__fput+0x0/0x214) from [<800d4f18>] (____fput+0x10/0x14) [<800d4f08>] (____fput+0x0/0x14) from [<8003d7cc>] (task_work_run+0xac/0xe0) [<8003d720>] (task_work_run+0x0/0xe0) from [<80025464>] (do_exit+0x2a4/0x8c8)
r7:9d450200 r6:9efcb600 r5:00000001 r4:9efcb908
[<800251c0>] (do_exit+0x0/0x8c8) from [<80025b9c>] (do_group_exit+0x44/0xb8)
r7:9e0b2604
[<80025b58>] (do_group_exit+0x0/0xb8) from [<80031490>] (get_signal_to_deliver+0x224/0x5b4)
r7:9e0b2604 r6:9ec8c000 r5:9ec8decc r4:9ec8c000
[<8003126c>] (get_signal_to_deliver+0x0/0x5b4) from [<8001153c>] (do_signal+0x24c/0x41c)
[<800112f0>] (do_signal+0x0/0x41c) from [<80011ad4>] (do_work_pending+0x88/0xc8) [<80011a4c>] (do_work_pending+0x0/0xc8) from [<8000e8f8>] (work_pending+0xc/0x20)
r8:8000ea64 r7:000000a8 r6:00000001 r5:0341cc18 r4:00000000Code: 0a000032 e5944024 e3540000 0a00002c (e5943020) ---[ end trace bb8d943ca1a10994 ]---Fixing recursive fault but reboot is needed!
Quoted 5 lines · Bjorn Helgaas
Quoted 4 lines · Bjorn Helgaas
Quoted 3 lines · Bjørn Erik Nilsen
I just hit an kernel oops related to PCI (in dw_msi_teardown_irq()/clear_irq() (pcie-designware))
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Linux version 3.12.0-next-20131105 (bnilsen@bnilsen) (gcc version 4.7.2 (GCC) )
Problem seem to be dereferencing a null pointer returned from
irq_desc_get_msi_desc(desc)(see attached backtrace).Included oops inline for ease of viewing/searching. Jingooo, I assume you'll investigate this. Let me know if otherwise.
(+cc Marek Vasut, Pratyush Anand, Kishon Vijay Abraham I,
Mohit KUMAR DCG, Ajay KHANDELWAL, Tim Harvey)
Sorry, I will not investigate this.
Bjørn Erik Nilsen,
Would you let us know the ARM platform and LAN card? If you let us know them, one of these pcie-designware related people would reproduce and look at the issue.
Best regards,
Jingoo Han
Quoted 89 lines
Quoted 89 lines
Unable to handle kernel NULL pointer dereference at virtual address 00000020 pgd = 80004000 [00000020] *pgd=00000000 Internal error: Oops: 17 [#1] SMP ARM Modules linked in: sxdma(O) CPU: 1 PID: 569 Comm: i2cipc.B3 Tainted: G O 3.12.0-next-20131105 #8 task: 9efcb600 ti: 9ec8c000 task.ti: 9ec8c000 PC is at dw_msi_teardown_irq+0x40/0x118 LR is at trace_hardirqs_on_caller+0xf4/0x1c0 pc : [<802c405c>] lr : [<80078370>] psr: 20000013sp :
9ec8dca8ip :9ec8dbe0fp :9ec8dccc
r10:9fa58800r9 : 00000000 r8 : 00000001
r7 :9f9fecc0r6 : 00000188 r5 :9ecdb580r4 : 00000000 r3 :9efcb600r2 : 00000000 r1 : 00000001 r0 : 00000012
Flags: nzCv IRQs on FIQs on ModeSVC_32ISA ARM Segment user
Control:10c5387dTable:2ed1804aDAC: 00000015 Process i2cipc.B3 (pid: 569, stack limit =0x9ec8c240)Stack: (0x9ec8dca8 to 0x9ec8e000)dca0:
8005d1a4802844c8000001889ecdb580000000049fa58c74
dcc0:9ec8dce49ec8dcd0802c1844802c402800000000 000000029ec8dd0c9ec8dce8
dce0:802c1a08802c181400000000 000000049fa58c60000000049fa58c7400000001
dd00:9ec8dd1c9ec8dd10802c1a34802c19ac9ec8dd449ec8dd20802c1ad0802c1a30
dd20:9fa588008084a41000047c8c0000016c0000016c000000089ec8dd5c9ec8dd48
dd40:802c2570802c1a44000000009d5800009ec8dd749ec8dd607f0057d4802c2534
dd60: 000000009d5800009ec8dd9c9ec8dd787f000a387f00579c7f0007b89f3fa200
dd80:9f5a2bd09f0c9cd09f383d509f383d509ec8dddc9ec8dda0800d4d407f0007c4
dda0: 00000000 00000000800cf4789f3fa2089f05b9c09efcb8f880891d409e3aab00
ddc0:9efcb6009ec8c0009ec8c0009d4502549ec8ddec9ec8dde0800d4f18800d4cb4
dde0:9ec8de0c9ec8ddf08003d7cc800d4f149efcb908000000019efcb6009d450200
de00:9ec8de449ec8de10800254648003d72c9ec8de3c00000001 800783708007814c
de20:9e0b26049ec8decc9ec8c0009e0b26049ec8de4c9e0b26049ec8de649ec8de48
de40:80025b9c800251cc9ec8c0009ec8decc9ec8c0009e0b26049ec8deb49ec8de68
de60: 8003149080025b6480073404 800733309ec8de949ec8de80805faf2c9ec8deb8
de80:418004fc9d46e6809ec8debc9ec8dfb09ec8c000fffffdfc9ec8deb876d533d0
dea0:9ec8c00076d533d49ec8df7c9ec8deb88001153c80031278 800783708007814c
dec0: 400000139f0727288083ffc000000009 00000000 00000000 00000000 00000000
dee0:9ec8df049ec8def0805faf1c800784489f07240081215fc09ec8df2c9ec8df08
df00:8004c6dc805faef0000000009f072400000000006deff4c8000003e200000000
df20:9ec8df8c9ec8c00080011ab89efcb600000000019ec8dfb09ec8c00000000000
df40:9ec8df6c9ec8df50800783708007814c9ec8c0009ec8c0009ec8c0009ec8c000
df60:8000ea649ec8dfb09ec8c000000000009ec8dfac9ec8df8080011ad4800112fc
df80: 00000000 000000018000e8ac000000000341cc1800000001000000a88000ea64
dfa0: 000000009ec8dfb08000e8f880011a5874ffeaa000000002 ffffffff012f7314
dfc0: 000000000341cc1800000001000000a874ffeb980341cd800341bcf074ffece4
dfe0: 0000000074ffea7076d533c476d533d08000001074ffeaa000000000 00000000Backtrace: [<802c401c>] (dw_msi_teardown_irq+0x0/0x118) from [<802c1844>] (arch_teardown_msi_irq+0x3c/0x40) r7:9fa58c74 r6:00000004 r5:9ecdb580 r4:00000188 [<802c1808>] (arch_teardown_msi_irq+0x0/0x40) from [<802c1a08>] (default_teardown_msi_irqs+0x68/0x84) r4:00000002 r3:00000000 [<802c19a0>] (default_teardown_msi_irqs+0x0/0x84) from [<802c1a34>] (arch_teardown_msi_irqs+0x10/0x14) r8:00000001 r7:9fa58c74 r6:00000004 r5:9fa58c60 r4:00000004r3:00000000
[<802c1a24>] (arch_teardown_msi_irqs+0x0/0x14) from [<802c1ad0>] (free_msi_irqs+0x98/0x144)
[<802c1a38>] (free_msi_irqs+0x0/0x144) from [<802c2570>] (pci_disable_msi+0x48/0x60)
[<802c2528>] (pci_disable_msi+0x0/0x60) from [<7f0057d4>] (sxdma_irq_free+0x44/0x48[sxdma])
r4:9d580000 r3:00000000
[<7f005790>] (sxdma_irq_free+0x0/0x48[sxdma]) from [<7f000a38>] (sxdma_dev_release+0x280/0x2b0[sxdma])
r4:9d580000 r3:00000000
[<7f0007b8>] (sxdma_dev_release+0x0/0x2b0[sxdma]) from [<800d4d40>] (__fput+0x98/0x214)
r8:9f383d50 r7:9f383d50 r6:9f0c9cd0 r5:9f5a2bd0 r4:9f3fa200
r3:7f0007b8
[<800d4ca8>] (__fput+0x0/0x214) from [<800d4f18>] (____fput+0x10/0x14) [<800d4f08>] (____fput+0x0/0x14) from [<8003d7cc>] (task_work_run+0xac/0xe0) [<8003d720>] (task_work_run+0x0/0xe0) from [<80025464>] (do_exit+0x2a4/0x8c8)
r7:9d450200 r6:9efcb600 r5:00000001 r4:9efcb908
[<800251c0>] (do_exit+0x0/0x8c8) from [<80025b9c>] (do_group_exit+0x44/0xb8)
r7:9e0b2604
[<80025b58>] (do_group_exit+0x0/0xb8) from [<80031490>] (get_signal_to_deliver+0x224/0x5b4)
r7:9e0b2604 r6:9ec8c000 r5:9ec8decc r4:9ec8c000
[<8003126c>] (get_signal_to_deliver+0x0/0x5b4) from [<8001153c>] (do_signal+0x24c/0x41c)
[<800112f0>] (do_signal+0x0/0x41c) from [<80011ad4>] (do_work_pending+0x88/0xc8) [<80011a4c>] (do_work_pending+0x0/0xc8) from [<8000e8f8>] (work_pending+0xc/0x20)
r8:8000ea64 r7:000000a8 r6:00000001 r5:0341cc18 r4:00000000Code: 0a000032 e5944024 e3540000 0a00002c (e5943020) ---[ end trace bb8d943ca1a10994 ]---Fixing recursive fault but reboot is needed!
Dear Jingoo Han,
Quoted 3 lines · Bjorn Helgaas
Quoted 2 lines · Bjorn Helgaas
Quoted 1 line
On Mon, Nov 18, 2013 at 6:53 AM, Bjørn Erik Nilsen <[email protected]>
Quoted 17 lines · wrote:
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
I just hit an kernel oops related to PCI (in dw_msi_teardown_irq()/clear_irq() (pcie-designware))
Linux version 3.12.0-next-20131105 (bnilsen@bnilsen) (gcc version 4.7.2 (GCC) )
Problem seem to be dereferencing a null pointer returned from
irq_desc_get_msi_desc(desc)(see attached backtrace).Included oops inline for ease of viewing/searching. Jingooo, I assume you'll investigate this. Let me know if otherwise.
(+cc Marek Vasut, Pratyush Anand, Kishon Vijay Abraham I,
Mohit KUMAR DCG, Ajay KHANDELWAL, Tim Harvey)Sorry, I will not investigate this.
Quoted 17 lines
Bjørn Erik Nilsen,
Would you let us know the ARM platform and LAN card? If you let us know them, one of these pcie-designware related people would reproduce and look at the issue.
Best regards,
Jingoo HanQuoted 8 lines
Quoted 8 lines
Unable to handle kernel NULL pointer dereference at virtual address 00000020 pgd = 80004000 [00000020] *pgd=00000000 Internal error: Oops: 17 [#1] SMP ARM Modules linked in: sxdma(O) CPU: 1 PID: 569 Comm: i2cipc.B3 Tainted: G O 3.12.0-next-20131105 #8 task: 9efcb600 ti: 9ec8c000 task.ti: 9ec8c000 PC is at dw_msi_teardown_irq+0x40/0x118
see drivers/pci/host/pcie-designware.c :
336 static void dw_msi_teardown_irq(struct msi_chip *chip, unsigned int irq) 337 {
338 clear_irq(irq);
339 }
So, add such a print before the clear_irq() call:
pr_err("%i %i\n", chip != NULL, irq);
And let us know the result please.
Best regards,
Marek Vasut
A message that is not in the archive (sent off-list or not yet received): 1384861142.3682.1.camel@bnilsen-HP-EliteBook-8760w
A message that is not in the archive (sent off-list or not yet received): [email protected]
Dear Bjørn Erik Nilsen,
Quoted 6 lines · Bjørn Erik Nilsen
Quoted 5 lines · Marek Vasut
Quoted 4 lines
Dear Jingoo Han,
Quoted 2 lines · Bjorn Helgaas
Quoted 1 line
On Mon, Nov 18, 2013 at 2:01 PM, Bjorn Helgaas <[email protected]>
Quoted 55 lines · wrote:
Quoted 55 lines
Quoted 47 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Mon, Nov 18, 2013 at 6:53 AM, Bjørn Erik Nilsen <[email protected]>
Quoted 18 lines · wrote:
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
I just hit an kernel oops related to PCI (in dw_msi_teardown_irq()/clear_irq() (pcie-designware))
Linux version 3.12.0-next-20131105 (bnilsen@bnilsen) (gcc version 4.7.2 (GCC) )
Problem seem to be dereferencing a null pointer returned from
irq_desc_get_msi_desc(desc)(see attached backtrace).Included oops inline for ease of viewing/searching. Jingooo, I assume you'll investigate this. Let me know if otherwise.
(+cc Marek Vasut, Pratyush Anand, Kishon Vijay Abraham I,
Mohit KUMAR DCG, Ajay KHANDELWAL, Tim Harvey)
Sorry, I will not investigate this.
Quoted 17 lines
Bjørn Erik Nilsen,
Would you let us know the ARM platform and LAN card? If you let us know them, one of these pcie-designware related people would reproduce and look at the issue.
Best regards,
Jingoo HanQuoted 8 lines
Quoted 8 lines
Unable to handle kernel NULL pointer dereference at virtual address 00000020 pgd = 80004000 [00000020] *pgd=00000000 Internal error: Oops: 17 [#1] SMP ARM Modules linked in: sxdma(O) CPU: 1 PID: 569 Comm: i2cipc.B3 Tainted: G O 3.12.0-next-20131105 #8 task: 9efcb600 ti: 9ec8c000 task.ti: 9ec8c000 PC is at dw_msi_teardown_irq+0x40/0x118see
drivers/pci/host/pcie-designware.c:336 static void
dw_msi_teardown_irq(struct msi_chip *chip, unsigned int irq)337 {
338clear_irq(irq);
339 }Quoted 5 lines
So, add such a print before the
clear_irq()call:
pr_err("%i %i\n", chip != NULL, irq);And let us know the result please.
Here's what I get:
Quoted 7 lines
Quoted 2 lines
1 391
1 392Also worth to mention is that I trigger this behavior by removing the device:
echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove
Just for completeness, is this pure next or something else, like the boundarydevices's kernel ?
Best regards,
Marek Vasut
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Bjørn Erik Nilsen,
Quoted 73 lines · Marek Vasut
Quoted 72 lines
Dear Bjørn Erik Nilsen,
Quoted 7 lines · Bjørn Erik Nilsen
Quoted 6 lines · Marek Vasut
Quoted 5 lines
Dear Jingoo Han,
Quoted 3 lines · Bjorn Helgaas
Quoted 2 lines
On Mon, Nov 18, 2013 at 2:01 PM, Bjorn Helgaas
<[email protected]>Quoted 61 lines · wrote:
Quoted 49 lines
Quoted 49 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Mon, Nov 18, 2013 at 6:53 AM, Bjørn Erik Nilsen <[email protected]>
Quoted 18 lines · wrote:
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
I just hit an kernel oops related to PCI (in dw_msi_teardown_irq()/clear_irq() (pcie-designware))
Linux version 3.12.0-next-20131105 (bnilsen@bnilsen) (gcc version 4.7.2 (GCC) )
Problem seem to be dereferencing a null pointer returned from
irq_desc_get_msi_desc(desc)(see attached backtrace).Included oops inline for ease of viewing/searching. Jingooo, I assume you'll investigate this. Let me know if otherwise.
(+cc Marek Vasut, Pratyush Anand, Kishon Vijay Abraham I,
Mohit KUMAR DCG, Ajay KHANDELWAL, Tim Harvey)
Sorry, I will not investigate this.
Quoted 17 lines
Bjørn Erik Nilsen,
Would you let us know the ARM platform and LAN card? If you let us know them, one of these pcie-designware related people would reproduce and look at the issue.
Best regards,
Jingoo HanQuoted 8 lines
Quoted 8 lines
Unable to handle kernel NULL pointer dereference at virtual address 00000020 pgd = 80004000 [00000020] *pgd=00000000 Internal error: Oops: 17 [#1] SMP ARM Modules linked in: sxdma(O) CPU: 1 PID: 569 Comm: i2cipc.B3 Tainted: G O 3.12.0-next-20131105 #8 task: 9efcb600 ti: 9ec8c000 task.ti: 9ec8c000 PC is at dw_msi_teardown_irq+0x40/0x118see
drivers/pci/host/pcie-designware.c:336 static void
dw_msi_teardown_irq(struct msi_chip *chip, unsigned int irq)337 {
338clear_irq(irq);
339 }So, add such a print before the
clear_irq()call:Quoted 8 lines
Quoted 3 lines
pr_err("%i %i\n", chip != NULL, irq);And let us know the result please.
Here's what I get:
1 391
1 392Also worth to mention is that I trigger this behavior by removing the device:
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove
Just for completeness, is this pure next or something else, like the boundarydevices's kernel ?
It's a boundary device kernel (boundary-imx_3.12.0):
https://github.com/boundarydevices/linux-imx6/tree/boundary-imx_3.12.0
OK, thanks. Jingoo, can you please try if this also happens on Exynos?
A message that is not in the archive (sent off-list or not yet received): [email protected]
Dear Bjørn Erik Nilsen,
Quoted 87 lines · Marek Vasut
Quoted 86 lines
Hi Bjørn Erik Nilsen,
Quoted 83 lines · Marek Vasut
Quoted 71 lines
Dear Bjørn Erik Nilsen,
Quoted 7 lines · Bjørn Erik Nilsen
Quoted 6 lines · Marek Vasut
Quoted 5 lines
Dear Jingoo Han,
Quoted 3 lines · Bjorn Helgaas
Quoted 2 lines
On Mon, Nov 18, 2013 at 2:01 PM, Bjorn Helgaas
<[email protected]>Quoted 60 lines · wrote:
Quoted 59 lines
Quoted 49 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Mon, Nov 18, 2013 at 6:53 AM, Bjørn Erik Nilsen <[email protected]>
Quoted 20 lines · wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
I just hit an kernel oops related to PCI (in dw_msi_teardown_irq()/clear_irq() (pcie-designware))
Linux version 3.12.0-next-20131105 (bnilsen@bnilsen) (gcc version 4.7.2 (GCC) )
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Problem seem to be dereferencing a null pointer returned from
irq_desc_get_msi_desc(desc)(see attached backtrace).Included oops inline for ease of viewing/searching. Jingooo, I assume you'll investigate this. Let me know if otherwise.
(+cc Marek Vasut, Pratyush Anand, Kishon Vijay Abraham I,
Mohit KUMAR DCG, Ajay KHANDELWAL, Tim Harvey)
Sorry, I will not investigate this.
Quoted 17 lines
Bjørn Erik Nilsen,
Would you let us know the ARM platform and LAN card? If you let us know them, one of these pcie-designware related people would reproduce and look at the issue.
Best regards,
Jingoo HanQuoted 8 lines
Quoted 8 lines
Unable to handle kernel NULL pointer dereference at virtual address 00000020 pgd = 80004000 [00000020] *pgd=00000000 Internal error: Oops: 17 [#1] SMP ARM Modules linked in: sxdma(O) CPU: 1 PID: 569 Comm: i2cipc.B3 Tainted: G O 3.12.0-next-20131105 #8 task: 9efcb600 ti: 9ec8c000 task.ti: 9ec8c000 PC is at dw_msi_teardown_irq+0x40/0x118see
drivers/pci/host/pcie-designware.c:336 static void
dw_msi_teardown_irq(struct msi_chip *chip, unsigned int irq)337 {
338clear_irq(irq);
339 }Quoted 5 lines
So, add such a print before the
clear_irq()call:
pr_err("%i %i\n", chip != NULL, irq);And let us know the result please.
Here's what I get:
1 391
1 392Quoted 7 lines
Quoted 4 lines
Also worth to mention is that I trigger this behavior by removing the device:
echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove
Just for completeness, is this pure next or something else, like the boundarydevices's kernel ?
It's a boundary device kernel (boundary-imx_3.12.0):
https://github.com/boundarydevices/linux-imx6/tree/boundary-imx_3.12.0
Quoted 8 lines
Quoted 1 line
OK, thanks. Jingoo, can you please try if this also happens on Exynos?
Sorry, I need to clarify the steps leading to the oops. It's actually not removing the device itself which is the trigger point. The kernel module (sxdma) registers a PCI driver and creates a char device (
/dev/sxdma). When this device is openedpci_enable_msiis called, and when it is closedsxdma_dev_releaseis called (which in turn callspci_disable_msias shown in the bt).
Uh, what's this 'sxdma' ?
Quoted 4 lines
I can therefore reproduce the issue by simply opening and closing
/dev/sxdma.Hope this clarifies and did not cause too much confusion.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Dear Bjørn Erik Nilsen,
Quoted 106 lines · Marek Vasut
Quoted 94 lines
Dear Bjørn Erik Nilsen,
Quoted 92 lines · Marek Vasut
Quoted 90 lines
Hi Bjørn Erik Nilsen,
Quoted 78 lines · Marek Vasut
Quoted 77 lines
Dear Bjørn Erik Nilsen,
Quoted 7 lines · Bjørn Erik Nilsen
Quoted 6 lines · Marek Vasut
Quoted 5 lines
Dear Jingoo Han,
Quoted 3 lines · Bjorn Helgaas
Quoted 2 lines
On Mon, Nov 18, 2013 at 2:01 PM, Bjorn Helgaas
<[email protected]>Quoted 65 lines · wrote:
Quoted 52 lines
Quoted 52 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Mon, Nov 18, 2013 at 6:53 AM, Bjørn Erik Nilsen <[email protected]>
Quoted 20 lines · wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
I just hit an kernel oops related to PCI (in
dw_msi_teardown_irq()/clear_irq() (pcie-designware))Linux version 3.12.0-next-20131105 (bnilsen@bnilsen)
(gcc version 4.7.2 (GCC) )Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Problem seem to be dereferencing a null pointer
returned from irq_desc_get_msi_desc(desc) (see
attached backtrace).Included oops inline for ease of viewing/searching. Jingooo, I assume you'll investigate this. Let me know if otherwise.
(+cc Marek Vasut, Pratyush Anand, Kishon Vijay Abraham I,
Mohit KUMAR DCG, Ajay KHANDELWAL, Tim Harvey)
Sorry, I will not investigate this.
Quoted 18 lines
Bjørn Erik Nilsen,
Would you let us know the ARM platform and LAN card? If you let us know them, one of these pcie-designware related people would reproduce and look at the issue.
Best regards,
Jingoo HanQuoted 9 lines
Quoted 9 lines
Unable to handle kernel NULL pointer dereference at virtual address 00000020 pgd = 80004000 [00000020] *pgd=00000000 Internal error: Oops: 17 [#1] SMP ARM Modules linked in: sxdma(O) CPU: 1 PID: 569 Comm: i2cipc.B3 Tainted: G O 3.12.0-next-20131105 #8 task: 9efcb600 ti: 9ec8c000task.ti:
9ec8c000PC is atdw_msi_teardown_irq+0x40/0x118see
drivers/pci/host/pcie-designware.c:336 static void
dw_msi_teardown_irq(struct msi_chip *chip, unsigned int irq)337 {
338clear_irq(irq);
339 }So, add such a print before the
clear_irq()call:Quoted 8 lines
Quoted 3 lines
pr_err("%i %i\n", chip != NULL, irq);And let us know the result please.
Here's what I get:
1 391
1 392Also worth to mention is that I trigger this behavior by removing the device:
echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove
Quoted 7 lines
Quoted 2 lines
Just for completeness, is this pure next or something else, like the boundarydevices's kernel ?
It's a boundary device kernel (boundary-imx_3.12.0):
https://github.com/boundarydevices/linux-imx6/tree/boundary-imx_3.1
2.0OK, thanks. Jingoo, can you please try if this also happens on Exynos?
Quoted 8 lines
Quoted 6 lines
Sorry, I need to clarify the steps leading to the oops. It's actually not removing the device itself which is the trigger point. The kernel module (sxdma) registers a PCI driver and creates a char device (
/dev/sxdma). When this device is openedpci_enable_msiis called, and when it is closedsxdma_dev_releaseis called (which in turn callspci_disable_msias shown in the bt).Uh, what's this 'sxdma' ?
It's the driver for the PCI device:
Quoted 10 lines
01:00.0 Memory controller: Barco Graphics NV Device ba01 (rev 01)
Flags: bus master, fast devsel, latency 0, IRQ 155
Memory at 01000000 (32-bit, non-prefetchable) [size=2M]
Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [200] Vendor Specific Information: ID=1172 Rev=0 Len=044
<?>
Kernel driver in use: sxdma
Sure, I just cannot find such driver anywhere in the kernel tree for some reason.
A message that is not in the archive (sent off-list or not yet received): 1385036087.3945.28.camel@bnilsen-HP
A message that is not in the archive (sent off-list or not yet received): 1385039987.6020.5.camel@bnilsen-HP
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 175 lines · Bjørn Erik Nilsen
Quoted 128 lines · Bjørn Erik Nilsen
Quoted 127 lines · Bjørn Erik Nilsen
Quoted 109 lines · Marek Vasut
Quoted 98 lines · Marek Vasut
Quoted 86 lines
Quoted 86 lines · Marek Vasut
Quoted 84 lines
Quoted 74 lines · Marek Vasut
Quoted 73 lines
Quoted 5 lines · Bjørn Erik Nilsen
Quoted 4 lines · Marek Vasut
Quoted 3 lines
On Tuesday, November 19, 2013 6:03 AM, Bjorn Helgaas wrote:
On Mon, Nov 18, 2013 at 2:01 PM, Bjorn Helgaas
<[email protected]>Quoted 65 lines · wrote:
Quoted 52 lines
Quoted 52 lines
On Mon, Nov 18, 2013 at 6:53 AM, Bjørn Erik Nilsen
<[email protected]>wrote:
I just hit an kernel oops related to PCI (in
dw_msi_teardown_irq()/clear_irq() (pcie-designware))Linux version 3.12.0-next-20131105 (bnilsen@bnilsen)
(gcc version 4.7.2 (GCC) )Problem seem to be dereferencing a null pointer
returned from irq_desc_get_msi_desc(desc) (see
attached backtrace).Included oops inline for ease of viewing/searching.
Jingooo, I assume you'll investigate this. Let me
know if otherwise.(+cc Marek Vasut, Pratyush Anand, Kishon Vijay Abraham I,
Mohit KUMAR DCG, Ajay KHANDELWAL, Tim Harvey)
Sorry, I will not investigate this.
Bjørn Erik Nilsen,
Would you let us know the ARM platform and LAN card?
If you let us know them, one of these pcie-designware
related people would reproduce and look at the issue.Best regards,
Jingoo HanUnable to handle kernel NULL pointer dereference at
virtual address 00000020 pgd = 80004000
[00000020] *pgd=00000000
Internal error: Oops: 17 [#1] SMP ARM
Modules linked in: sxdma(O)
CPU: 1 PID: 569 Comm: i2cipc.B3 Tainted: G O
3.12.0-next-20131105 #8 task: 9efcb600 ti: 9ec8c000
task.ti: 9ec8c000 PC is at
dw_msi_teardown_irq+0x40/0x118see drivers/pci/host/pcie-designware.c :
336 static void dw_msi_teardown_irq(struct msi_chip *chip,
unsigned int irq) 337 {
338 clear_irq(irq);
339 }So, add such a print before the clear_irq() call:
Quoted 8 lines
Quoted 3 lines
pr_err("%i %i\n", chip != NULL, irq);
And let us know the result please.
Here's what I get:
1 391
1 392Also worth to mention is that I trigger this behavior by removing the device:
echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove
Quoted 7 lines
Quoted 2 lines
Just for completeness, is this pure next or something else, like the boundarydevices's kernel ?
It's a boundary device kernel (boundary-imx_3.12.0):
https://github.com/boundarydevices/linux-imx6/tree/boundary-imx_3.1
2.0OK, thanks. Jingoo, can you please try if this also happens on Exynos?
Quoted 8 lines
Quoted 6 lines
Sorry, I need to clarify the steps leading to the oops. It's actually not removing the device itself which is the trigger point. The kernel module (sxdma) registers a PCI driver and creates a char device (
/dev/sxdma). When this device is openedpci_enable_msiis called, and when it is closedsxdma_dev_releaseis called (which in turn callspci_disable_msias shown in the bt).Uh, what's this 'sxdma' ?
It's the driver for the PCI device:
Quoted 10 lines
01:00.0 Memory controller: Barco Graphics NV Device ba01 (rev 01)
Flags: bus master, fast devsel, latency 0, IRQ 155
Memory at 01000000 (32-bit, non-prefetchable) [size=2M]
Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [200] Vendor Specific Information: ID=1172 Rev=0 Len=044
<?>
Kernel driver in use: sxdmaQuoted 2 lines
Sure, I just cannot find such driver anywhere in the kernel tree for some reason.
The sxdma kernel module is developed outside the kernel tree, however I can give you the source (GPL V2) if you are interested or think it is important for this issue. Please let me know.
The background is that a successful call to
pci_enable_msifollowed bypci_disable_msitriggers the crash. It used to work just fine before the pci-imx6/pcie-designware code was upstreamed.I guess you already figured this out, but to clarify my findings so far:
As mentioned earlier, the real hot spot is that
irq_desc_get_msi_desc(called fromclear_irqinpcie-designware.c) returns NULL and its return value is unconditionally dereferenced.Quoted 8 lines
Quoted 4 lines
Judging from the code it looks to me that it should never return NULL in that case (because there is no check against NULL before dereferencing it). So the nut to crack is to find out why it returns NULL in this particular case, and why it used to work before.
As far as I can tell,
irq_set_msi_desc(irq, desc)is never called which is whyirq_desc_get_msi_descreturns NULL. The following patch fixes the problem (I also simplified the retrieval of themsi_desc):I jumped on the conclusion a bit too fast with my previous patch as it only removes the artifact in one particular case rather than fixing the real problem.
I did more extensive testing and here's my new findings:
irq_set_msi_desc()is indeed called from dw_msi_setup_irq/assign_irq, however when I looked at the implementation ofirq_set_msi_desc()I noticed how it actually updates the entry's irq. So after four consecutive calls to the function, the irq points to the last irq:irq_set_msi_desc(388, desc); -> desc->irq = 388 ... irq_set_msi_desc(391, desc); -> desc->irq = 391This is an important detail, because from
msi.cteardown_msi_irq()thedesc->irqis used as the base address for the teardown. I.e.:for (int i = 0; i < number of irqs; ++i)
teardown_msi_irq(desc->irq + i);teardown(391)
teardown(392)
teardown(393)
teardown(394)Bummer.
And now you probably understand why my previous patch fixed an artifact, because what it really did was updating the
desc->irqto point to 388 (instead of 391):irq_set_msi_desc(388, desc); -> desc->irq = 388
Quoted 7 lines
We therefore got somewhat closer to a clean teardown. The next problem is that each allocated irq depend on each other as they all share the same
msi_descpointer, which means they cannot be freed independently (at least so I think). My idea was therefore to allocate all irqs at once fromsetup()and free them all at once fromteardown(). My new patch makes my system stable again and fixes all the issues (not only one particular issue):
Hi Bjørn Erik Nilsen,
Thank you for your effort!
I reproduced this kernel panic on Exynos platform with LAN card. And then, I tested your patch and checked this kernel panic is resolved.
Marek Vasut,
Will you test Bjørn Erik Nilsen's patch with your i.MX platform?
Pratyush Anand,
Would you confirm Bjørn Erik Nilsen's patch?
I will do more extensive testing.
Thank you.
Best regards,
Jingoo Han
Quoted 57 lines
pcie-designware.c renamed from pcie-designware.c.orig +13
-6@@ -242,11 +242,12 @@ static int assign_irq(int no_irqs, strucif (!irq) goto no_valid_irq; +irq_alloc_descs (irq, irq, no_irqs,0 ); + i =0 ;while (i < no_irqs) {set_bit (pos0 + i, pp->msi_irq_in_use); -irq_alloc_descs ((irq + i), (irq + i),1 ,0 ); -irq_set_msi_desc (irq + i, desc); +irq_set_msi_desc_off (irq + i, i, desc);/*Enable corresponding interrupt in MSI interrupt controller */ res = ((pos0 + i) /32 ) *12 ; bit = (pos0 + i) %32 ;@@ -267,14 +268,12 @@ no_valid_irq:static void clear_irq (unsigned int irq) {int res, bit, val, pos; -struct irq_desc *desc;struct msi_desc *msi;struct pcie_port *pp;struct irq_data *data =irq_get_irq_data (irq);/* get the port structure */ - desc =irq_to_desc (irq); - msi =irq_desc_get_msi_desc (desc); + msi = data->msi_desc; pp =sys_to_pcie (msi->dev->bus->sysdata);if (!pp) {BUG ();@@ -283,7 +282,15 @@ static void clear_irq(unsigned int irq) pos = data->hwirq; -irq_free_desc (irq); +if (irq == msi->irq) { +/* +* This is the irq returned from dw_msi_setup_irq where +* 'nvec' number of desc where allocated. Now it is time +* to free them all. +*/ +unsigned int nvec =1 << msi->msi_attrib.multiple; +irq_free_descs (irq, nvec); + }clear_bit (pos, pp->msi_irq_in_use);Best regards,
Bjørn Erik Nilsen
A message that is not in the archive (sent off-list or not yet received): 1385118399.3944.32.camel@bnilsen-HP
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 1 line · Bjørn Erik Nilsen
On Fri, 2013-11-22 at 12:06 +0100, Bjørn Erik Nilsen wrote:
[....]
Quoted 21 lines
Quoted 17 lines
The patch does indeed fix the crash, but there are more subtle issues lurking around. I noticed how irq numbers were constantly increasing and I found at least one stupid mistake that I made.
Review
Quoted 2 lines
(quoted diff) +1
-1- irq_set_msi_desc(irq + i, desc); + irq_set_msi_desc_off(irq + i, i, desc);That should be '
irq_set_msi_desc_off(irq, i, desc)'(I'm really puzzled why this didn't cause other oops ...)
I also realized that
teardown()is only called for the irq returned fromsetup(), which kind of makes sense since the others aredestroyed()in the first call to teardown. This means we need to iterate all irqs in the same fashion as we allocate in setup/assign_irq. Basically unrolling what was done there.Quoted 3 lines
I'm hacking on this solution now but it doesn't quite take me to where I want at the moment, so it would be nice if someone with a better understanding of the code could pitch in.
Quoted 6 lines
I gave it another shot and now it starts to look like something. At least I get consistent irq numbers and my system is very stable in general.
My new patch does exactly the opposite in
teardown()of what is done insetup(), which in itself is a good sign.
Hi Bjørn Erik Nilsen,
I tested this patch with Exynos platform.
I checked that the kernel panic is resolved.
By the way, would you change your mail to text mode? Your mails have not been submitted to PCI-mailing list.
Marek Vasut, Pratyush Anand, Kishon Vijay Abraham I Would you review Bjørn Erik Nilsen's new patch?
Quoted 21 lines
pcie-designware.c 2013-11-22 16:32:30.360954591 +0100 renamed from pcie-designware.c.orig 2013-11-21 14:02:03.656007695 +0100 +6
-2@@ -242,11 +242,15 @@ static int assign_irq(int no_irqs, struc if (!irq) goto no_valid_irq; + /* + * irq_domain_add_linear (called from dw_pcie_host_init) pre-allocates + * descs so there is no need to call irq_alloc_descs here. + */ + i = 0; while (i < no_irqs) { set_bit(pos0 + i, pp->msi_irq_in_use); - irq_alloc_descs((irq + i), (irq + i), 1, 0); - irq_set_msi_desc(irq + i, desc); + irq_set_msi_desc_off(irq, i, desc); /*Enable corresponding interrupt in MSI interrupt controller */ res = ((pos0 + i) / 32) * 12; bit = (pos0 + i) % 32;
pcie-designware.c 2013-11-22 16:32:30.360954591 +0100 no_valid_irq: · line 269
Quoted 4 lines
pcie-designware.c 2013-11-22 16:32:30.360954591 +0100 renamed from pcie-designware.c.orig 2013-11-21 14:02:03.656007695 +0100 +0
-1@@ -266,7 +270,7 @@ no_valid_irq: static void clear_irq(unsigned int irq) { - int res, bit, val, pos;
The build errors happens.
Thus, the following lines are necessary
+ int res, bit, pos, nvec, i;
+ u32 val;
Best regards,
Jingoo Han
Quoted 62 lines
pcie-designware.c 2013-11-22 16:32:30.360954591 +0100 renamed from pcie-designware.c.orig 2013-11-21 14:02:03.656007695 +0100 +23
-14struct irq_desc *desc; struct msi_desc *msi; struct pcie_port *pp;@@ -281,18 +285,28 @@ static void clear_irq(unsigned int irq) return; } + /* undo what was done in assign_irq */ pos = data->hwirq; + nvec = 1 << msi->msi_attrib.multiple; - irq_free_desc(irq); - - clear_bit(pos, pp->msi_irq_in_use); + i = 0; + while (i < nvec) { + clear_bit(pos + i, pp->msi_irq_in_use); + irq_set_msi_desc_off(irq, i, NULL); + /* Disable corresponding interrupt on MSI interrupt controller */ + res = ((pos + i) / 32) * 12; + bit = (pos + i) % 32; + dw_pcie_rd_own_conf(pp, PCIE_MSI_INTR0_ENABLE + res, 4, &val); + val &= ~(1 << bit); + dw_pcie_wr_own_conf(pp, PCIE_MSI_INTR0_ENABLE + res, 4, val); + ++i; + } - /* Disable corresponding interrupt on MSI interrupt controller */ - res = (pos / 32) * 12; - bit = pos % 32; - dw_pcie_rd_own_conf(pp, PCIE_MSI_INTR0_ENABLE + res, 4, &val); - val &= ~(1 << bit); - dw_pcie_wr_own_conf(pp, PCIE_MSI_INTR0_ENABLE + res, 4, val); + msi->msg.address_lo = 0; + msi->msg.address_hi = 0; + msi->msg.data = 0; + msi->irq = 0; + msi->msi_attrib.multiple = 0; } static int dw_msi_setup_irq(struct msi_chip *chip, struct pci_dev *pdev,@@ -320,15 +334,14 @@ static int dw_msi_setup_irq(struct msi_c if (irq < 0) return irq; - msg_ctr &= ~PCI_MSI_FLAGS_QSIZE; - msg_ctr |= msgvec << 4; - pci_write_config_word(pdev, desc->msi_attrib.pos + PCI_MSI_FLAGS, - msg_ctr); desc->msi_attrib.multiple = msgvec; - msg.address_lo = virt_to_phys((void *)pp->msi_data); msg.address_hi = 0x0; msg.data = pos; + /* + * write_msi_msg() will update PCI_MSI_FLAGS so there + * is no need to explicitly call pci_write_config here + */ write_msi_msg(irq, &msg); return 0;
A message that is not in the archive (sent off-list or not yet received): [email protected]
Marek Vasut
Tested-byDear Bjørn Erik Nilsen,
[...]
Quoted 52 lines
Quoted 31 lines
Quoted 5 lines
Thank you for your effort!
I reproduced this kernel panic on Exynos platform with LAN card. And then, I tested your patch and checked this kernel panic is resolved.Quoted 8 lines
Marek Vasut,
Will you test Bjørn Erik Nilsen's patch with your i.MX platform?Pratyush Anand,
Would you confirm Bjørn Erik Nilsen's patch?I will do more extensive testing.
Thank you.The patch does indeed fix the crash, but there are more subtle issues lurking around. I noticed how irq numbers were constantly increasing and I found at least one stupid mistake that I made.
Review
Quoted 2 lines
(quoted diff) +1
-1- irq_set_msi_desc(irq + i, desc); + irq_set_msi_desc_off(irq + i, i, desc);That should be '
irq_set_msi_desc_off(irq, i, desc)'(I'm really puzzled why this didn't cause other oops ...)
I also realized that
teardown()is only called for the irq returned fromsetup(), which kind of makes sense since the others aredestroyed()in the first call to teardown. This means we need to iterate all irqs in the same fashion as we allocate in setup/assign_irq. Basically unrolling what was done there.Quoted 3 lines
I'm hacking on this solution now but it doesn't quite take me to where I want at the moment, so it would be nice if someone with a better understanding of the code could pitch in.
I gave it another shot and now it starts to look like something. At least I get consistent irq numbers and my system is very stable in general.
My new patch does exactly the opposite in
teardown()of what is done insetup(), which in itself is a good sign.pcie-designware.c 2013-11-22 16:32:30.360954591 +0100 renamed from pcie-designware.c.orig 2013-11-21 14:02:03.656007695 +0100 +2
-0@@ -242,11 +242,15 @@ static int assign_irq(int no_irqs, struc if (!irq) goto no_valid_irq; + /* + * irq_domain_add_linear (called from dw_pcie_host_init)pre-allocates + * descs so there is no need to call
irq_alloc_descs
here. + */
pcie-designware.c 2013-11-22 16:32:30.360954591 +0100
Quoted 7 lines
pcie-designware.c 2013-11-22 16:32:30.360954591 +0100 renamed from pcie-designware.c.orig 2013-11-21 14:02:03.656007695 +0100 +2
-2+ i = 0; while (i < no_irqs) { set_bit(pos0 + i, pp->msi_irq_in_use); - irq_alloc_descs((irq + i), (irq + i), 1, 0); - irq_set_msi_desc(irq + i, desc); + irq_set_msi_desc_off(irq, i, desc);
Why do you not allocate the descs anymore ?
Quoted 71 lines
/*Enable corresponding interrupt in MSI interrupt controller */ res = ((pos0 + i) / 32) * 12; bit = (pos0 + i) % 32;pcie-designware.c 2013-11-22 16:32:30.360954591 +0100 renamed from pcie-designware.c.orig 2013-11-21 14:02:03.656007695 +0100 +20
-15@@ -266,7 +270,7 @@ no_valid_irq: static void clear_irq(unsigned int irq) { - int res, bit, val, pos; struct irq_desc *desc; struct msi_desc *msi; struct pcie_port *pp;@@ -281,18 +285,28 @@ static void clear_irq(unsigned int irq) return; } + /* undo what was done in assign_irq */ pos = data->hwirq; + nvec = 1 << msi->msi_attrib.multiple; - irq_free_desc(irq); - - clear_bit(pos, pp->msi_irq_in_use); + i = 0; + while (i < nvec) { + clear_bit(pos + i, pp->msi_irq_in_use); + irq_set_msi_desc_off(irq, i, NULL); + /* Disable corresponding interrupt on MSI interrupt controller */ + res = ((pos + i) / 32) * 12; + bit = (pos + i) % 32; + dw_pcie_rd_own_conf(pp, PCIE_MSI_INTR0_ENABLE + res, 4, &val); + val &= ~(1 << bit); + dw_pcie_wr_own_conf(pp, PCIE_MSI_INTR0_ENABLE + res, 4, val); + ++i; + } - /* Disable corresponding interrupt on MSI interrupt controller */ - res = (pos / 32) * 12; - bit = pos % 32; - dw_pcie_rd_own_conf(pp, PCIE_MSI_INTR0_ENABLE + res, 4, &val); - val &= ~(1 << bit); - dw_pcie_wr_own_conf(pp, PCIE_MSI_INTR0_ENABLE + res, 4, val); + msi->msg.address_lo = 0; + msi->msg.address_hi = 0; + msi->msg.data = 0; + msi->irq = 0; + msi->msi_attrib.multiple = 0; } static int dw_msi_setup_irq(struct msi_chip *chip, struct pci_dev *pdev,@@ -320,15 +334,14 @@ static int dw_msi_setup_irq(struct msi_c if (irq < 0) return irq; - msg_ctr &= ~PCI_MSI_FLAGS_QSIZE; - msg_ctr |= msgvec << 4; - pci_write_config_word(pdev, desc->msi_attrib.pos + PCI_MSI_FLAGS, - msg_ctr); desc->msi_attrib.multiple = msgvec; - msg.address_lo = virt_to_phys((void *)pp->msi_data); msg.address_hi = 0x0; msg.data = pos; + /* + * write_msi_msg() will update PCI_MSI_FLAGS so there + * is no need to explicitly call pci_write_config here + */ write_msi_msg(irq, &msg); return 0;
Quoted 3 lines
Best regards,
Bjørn Erik Nilsen
Can you try using 'git send-email' for submitting subsequent patches please?
The patch works back and forth, I can 'remove' the card with:
echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove
and then 'rescan' the card with:
echo 1 > /sys/bus/pci/devices/0000:00:00.0/rescan
A message that is not in the archive (sent off-list or not yet received): [email protected]
Dear Bjørn Erik Nilsen,
Quoted 75 lines
26. nov. 2013 kl. 22:19 skrev Marek Vasut <[email protected]>:
Quoted 62 lines
Dear Bjørn Erik Nilsen,
[...]Quoted 59 lines
Quoted 31 lines
Quoted 5 lines
Thank you for your effort!
I reproduced this kernel panic on Exynos platform with LAN card. And then, I tested your patch and checked this kernel panic is resolved.Quoted 8 lines
Marek Vasut,
Will you test Bjørn Erik Nilsen's patch with your i.MX platform?Pratyush Anand,
Would you confirm Bjørn Erik Nilsen's patch?I will do more extensive testing.
Thank you.The patch does indeed fix the crash, but there are more subtle issues lurking around. I noticed how irq numbers were constantly increasing and I found at least one stupid mistake that I made.
Review
Quoted 2 lines
(quoted diff) +1
-1- irq_set_msi_desc(irq + i, desc); + irq_set_msi_desc_off(irq + i, i, desc);That should be '
irq_set_msi_desc_off(irq, i, desc)'(I'm really puzzled why this didn't cause other oops ...)
I also realized that
teardown()is only called for the irq returned fromsetup(), which kind of makes sense since the others aredestroyed()in the first call to teardown. This means we need to iterate all irqs in the same fashion as we allocate in setup/assign_irq. Basically unrolling what was done there.Quoted 3 lines
I'm hacking on this solution now but it doesn't quite take me to where I want at the moment, so it would be nice if someone with a better understanding of the code could pitch in.
I gave it another shot and now it starts to look like something. At least I get consistent irq numbers and my system is very stable in general.
My new patch does exactly the opposite in
teardown()of what is done insetup(), which in itself is a good sign.pcie-designware.c 2013-11-22 16:32:30.360954591 +0100 renamed from pcie-designware.c.orig 2013-11-21 14:02:03.656007695 +0100 +2
-0@@ -242,11 +242,15 @@ static int assign_irq(int no_irqs, struc if (!irq) goto no_valid_irq; + /* + * irq_domain_add_linear (called from dw_pcie_host_init)pre-allocates + * descs so there is no need to call
irq_alloc_descshere. + */
+i = 0; while (i < no_irqs) {Quoted 7 lines
pcie-designware.c 2013-11-22 16:32:30.360954591 +0100
Quoted 5 lines
set_bit(pos0 + i, pp->msi_irq_in_use);pcie-designware.c 2013-11-22 16:32:30.360954591 +0100 renamed from pcie-designware.c.orig 2013-11-21 14:02:03.656007695 +0100 +1
-2- irq_alloc_descs((irq + i), (irq + i), 1, 0); - irq_set_msi_desc(irq + i, desc); + irq_set_msi_desc_off(irq, i, desc);Why do you not allocate the descs anymore ?
Descs are allocated in
irq_create_mapping, added in commit904d0e788993Oh, and I realize now that my comment in the patch regarding this is wrong.
Quoted 7 lines
[…]
Quoted 2 lines
Can you try using 'git send-email' for submitting subsequent patches please?
I will eventually try using git send-mail for my next patch, which will correct the comment.
That'd be really helpful, thanks for spotting this btw ;-)
Dear Bjørn Erik Nilsen,
Quoted 60 lines
26. nov. 2013 kl. 22:19 skrev Marek Vasut <[email protected]>:
Quoted 59 lines
Dear Bjørn Erik Nilsen,
[...]Quoted 56 lines
Quoted 31 lines
Quoted 5 lines
Thank you for your effort!
I reproduced this kernel panic on Exynos platform with LAN card. And then, I tested your patch and checked this kernel panic is resolved.Quoted 8 lines
Marek Vasut,
Will you test Bjørn Erik Nilsen's patch with your i.MX platform?Pratyush Anand,
Would you confirm Bjørn Erik Nilsen's patch?I will do more extensive testing.
Thank you.The patch does indeed fix the crash, but there are more subtle issues lurking around. I noticed how irq numbers were constantly increasing and I found at least one stupid mistake that I made.
Review
Quoted 2 lines
(quoted diff) +1
-1- irq_set_msi_desc(irq + i, desc); + irq_set_msi_desc_off(irq + i, i, desc);That should be '
irq_set_msi_desc_off(irq, i, desc)'(I'm really puzzled why this didn't cause other oops ...)
I also realized that
teardown()is only called for the irq returned fromsetup(), which kind of makes sense since the others aredestroyed()in the first call to teardown. This means we need to iterate all irqs in the same fashion as we allocate in setup/assign_irq. Basically unrolling what was done there.Quoted 3 lines
I'm hacking on this solution now but it doesn't quite take me to where I want at the moment, so it would be nice if someone with a better understanding of the code could pitch in.
I gave it another shot and now it starts to look like something. At least I get consistent irq numbers and my system is very stable in general.
My new patch does exactly the opposite in
teardown()of what is done insetup(), which in itself is a good sign.pcie-designware.c 2013-11-22 16:32:30.360954591 +0100 renamed from pcie-designware.c.orig 2013-11-21 14:02:03.656007695 +0100 +2
-0@@ -242,11 +242,15 @@ static int assign_irq(int no_irqs, struc if (!irq) goto no_valid_irq; + /* + * irq_domain_add_linear (called from dw_pcie_host_init)pre-allocates + * descs so there is no need to call
irq_alloc_descshere. + */
+
Quoted 8 lines
Quoted 8 lines
pcie-designware.c 2013-11-22 16:32:30.360954591 +0100
Quoted 8 lines
i = 0; while (i < no_irqs) { set_bit(pos0 + i, pp->msi_irq_in_use);pcie-designware.c 2013-11-22 16:32:30.360954591 +0100 renamed from pcie-designware.c.orig 2013-11-21 14:02:03.656007695 +0100 +1
-2- irq_alloc_descs((irq + i), (irq + i), 1, 0); - irq_set_msi_desc(irq + i, desc); + irq_set_msi_desc_off(irq, i, desc);
btw you are not checking the return value of irq_set_msi_desc{,_off}(), are you sure these functions do return 0 always ? Please fix this so you also check the return value.
Best regards,
A message that is not in the archive (sent off-list or not yet received): [email protected]
Dear Bjørn Erik Nilsen,
Quoted 23 lines
Commit
904d0e7889933fb48d921c998fd1cabb3a9d6635addedirq_create_mappingwhich pre-allocates irq descs. Problem was that inassign_irqthese descs were explicitly allocated and hence also freed, resulting in a crash.Signed-off-by Bjørn Erik Nilsen <[email protected]>--- drivers/pci/host/pcie-designware.c | 50+++++++++++++++++++++++++------------- 1 file changed, 33 insertions(+), 17 deletions(-)
(quoted diff) +0
-0
b/drivers/pci/host/pcie-designware.cindex e33b68b..6c61abd 100644drivers/pci/host/pcie-designware.c +4
-0@@ -242,11 +242,18 @@ static int assign_irq(int no_irqs, struct msi_desc *desc, int *pos) if (!irq) goto no_valid_irq; +/* +* irq_create_mapping (called from dw_pcie_host_init) pre-allocates +* descs so there is no need to allocate descs here. We can therefore +* assume that if irq_find_mapping above returns non-zero, then the
drivers/pci/host/pcie-designware.c static int assign_irq(int no_irqs, struct msi_desc · line 256
Quoted 10 lines
drivers/pci/host/pcie-designware.c +5
-2+* descs are also successfully allocated. Hence no need to check the +* return value of irq_set_msi_desc_off below. +*/ + i =0 ;while (i < no_irqs) {set_bit (pos0 + i, pp->msi_irq_in_use); -irq_alloc_descs ((irq + i), (irq + i),1 ,0 ); -irq_set_msi_desc (irq + i, desc); +irq_set_msi_desc_off (irq, i, desc);
This function here returns an return value , you _must_ check it and handle the possible error accordingly .
Quoted 17 lines
drivers/pci/host/pcie-designware.c +2
-1/*Enable corresponding interrupt in MSI interrupt controller */ res = ((pos0 + i) /32 ) *12 ; bit = (pos0 + i) %32 ;@@ -266,7 +273,7 @@ no_valid_irq:static void clear_irq (unsigned int irq) { -int res, bit, val, pos; +int res, bit, val, pos, i, nvec;struct irq_desc *desc;struct msi_desc *msi;struct pcie_port *pp;@@ -281,18 +288,28 @@ static void clear_irq(unsigned int irq)return ; } +/* undo what was done in assign_irq */
drivers/pci/host/pcie-designware.c static void clear_irq(unsigned int irq) · line 298
Quoted 10 lines
drivers/pci/host/pcie-designware.c +5
-3pos = data->hwirq; + nvec =1 << msi->msi_attrib.multiple; -irq_free_desc (irq); - -clear_bit (pos, pp->msi_irq_in_use); + i =0 ; +while (i < nvec) { +clear_bit (pos + i, pp->msi_irq_in_use); +irq_set_msi_desc_off (irq, i,NULL );
DTTO here.
drivers/pci/host/pcie-designware.c static void clear_irq(unsigned int irq) · line 299
Quoted 1 line
drivers/pci/host/pcie-designware.c +1
-0+/* Disable corresponding interrupt on MSI interrupt controller
*/
Quoted 9 lines
drivers/pci/host/pcie-designware.c +7
-1+ res = ((pos + i) /32 ) *12 ; + bit = (pos + i) %32 ; +dw_pcie_rd_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , &val); + val &= ~(1 << bit); +dw_pcie_wr_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , val); + ++i; + } -/* Disable corresponding interrupt on MSI interrupt controller */
drivers/pci/host/pcie-designware.c static void clear_irq(unsigned int irq) · line 312
Quoted 10 lines
drivers/pci/host/pcie-designware.c +5
-5- res = (pos /32 ) *12 ; - bit = pos %32 ; -dw_pcie_rd_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , &val); - val &= ~(1 << bit); -dw_pcie_wr_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , val); + msi->msg.address_lo =0 ; + msi->msg.address_hi =0 ; + msi->msg.data =0 ; + msi->irq =0 ; + msi->msi_attrib.multiple =0 ;
This piece of new code could use a comment please.
Quoted 9 lines
drivers/pci/host/pcie-designware.c +0
-2} static int dw_msi_setup_irq(struct msi_chip *chip, struct pci_dev *pdev,@@ -320,12 +337,11 @@ static int dw_msi_setup_irq(struct msi_chip *chip, struct pci_dev *pdev, if (irq < 0) return irq; - msg_ctr &= ~PCI_MSI_FLAGS_QSIZE ; - msg_ctr |= msgvec <<4 ;
drivers/pci/host/pcie-designware.c static int dw_msi_setup_irq(struct msi_chip *chip, · line 346
Quoted 10 lines
drivers/pci/host/pcie-designware.c +4
-3-pci_write_config_word (pdev, desc->msi_attrib.pos +PCI_MSI_FLAGS , - msg_ctr); +/* +* write_msi_msg() will update PCI_MSI_FLAGS so there +* is no need to explicitly call pci_write_config here +*/ desc->msi_attrib.multiple = msgvec; - msg.address_lo =virt_to_phys ((void *)pp->msi_data); msg.address_hi =0x0 ;
Won't this overwrite your new code above ?
Quoted 1 line
drivers/pci/host/pcie-designware.c +0
-0msg.data = pos;
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines · Bjørn Erik Nilsen
Commit
904d0e7889933fb48d921c998fd1cabb3a9d6635added
Please replace it with the following.
Commit 904d0e7889933fb48d921c998fd1cabb3a9d6635
"PCI: designware: Add irq_create_mapping()" added
Also, the subject of the commit is not good.
Usually, 'verb + objective + etc.' looks good.
Ex. PCI: designware: Remove irq_alloc_descs()
I tested this patch on Exynos platform; it works properly. I will wait for v5 patch which addresses Marek Vasut's comments. Thank you.
Best regards,
Jingoo Han
Quoted 101 lines
irq_create_mappingwhich pre-allocates irq descs. Problem was that inassign_irqthese descs were explicitly allocated and hence also freed, resulting in a crash.Signed-off-by Bjørn Erik Nilsen <[email protected]>--- drivers/pci/host/pcie-designware.c | 50 +++++++++++++++++++++++++------------- 1 file changed, 33 insertions(+), 17 deletions(-)drivers/pci/host/pcie-designware.c +33
-18@@ -242,11 +242,18 @@ static int assign_irq(int no_irqs, struct msi_desc *desc, int *pos)if (!irq) goto no_valid_irq; +/* +* irq_create_mapping (called from dw_pcie_host_init) pre-allocates +* descs so there is no need to allocate descs here. We can therefore +* assume that if irq_find_mapping above returns non-zero, then the +* descs are also successfully allocated. Hence no need to check the +* return value of irq_set_msi_desc_off below. +*/ + i =0 ;while (i < no_irqs) {set_bit (pos0 + i, pp->msi_irq_in_use); -irq_alloc_descs ((irq + i), (irq + i),1 ,0 ); -irq_set_msi_desc (irq + i, desc); +irq_set_msi_desc_off (irq, i, desc);/*Enable corresponding interrupt in MSI interrupt controller */ res = ((pos0 + i) /32 ) *12 ; bit = (pos0 + i) %32 ;@@ -266,7 +273,7 @@ no_valid_irq:static void clear_irq (unsigned int irq) { -int res, bit, val, pos; +int res, bit, val, pos, i, nvec;struct irq_desc *desc;struct msi_desc *msi;struct pcie_port *pp;@@ -281,18 +288,28 @@ static void clear_irq(unsigned int irq)return ; } +/* undo what was done in assign_irq */ pos = data->hwirq; + nvec =1 << msi->msi_attrib.multiple; -irq_free_desc (irq); - -clear_bit (pos, pp->msi_irq_in_use); + i =0 ; +while (i < nvec) { +clear_bit (pos + i, pp->msi_irq_in_use); +irq_set_msi_desc_off (irq, i,NULL ); +/* Disable corresponding interrupt on MSI interrupt controller */ + res = ((pos + i) /32 ) *12 ; + bit = (pos + i) %32 ; +dw_pcie_rd_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , &val); + val &= ~(1 << bit); +dw_pcie_wr_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , val); + ++i; + } -/* Disable corresponding interrupt on MSI interrupt controller */ - res = (pos /32 ) *12 ; - bit = pos %32 ; -dw_pcie_rd_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , &val); - val &= ~(1 << bit); -dw_pcie_wr_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , val); + msi->msg.address_lo =0 ; + msi->msg.address_hi =0 ; + msi->msg.data =0 ; + msi->irq =0 ; + msi->msi_attrib.multiple =0 ; } static int dw_msi_setup_irq(struct msi_chip *chip, struct pci_dev *pdev,@@ -320,12 +337,11 @@ static int dw_msi_setup_irq(struct msi_chip *chip, struct pci_dev *pdev,if (irq <0 )return irq; - msg_ctr &= ~PCI_MSI_FLAGS_QSIZE ; - msg_ctr |= msgvec <<4 ; -pci_write_config_word (pdev, desc->msi_attrib.pos +PCI_MSI_FLAGS , - msg_ctr); +/* +* write_msi_msg() will update PCI_MSI_FLAGS so there +* is no need to explicitly call pci_write_config here +*/ desc->msi_attrib.multiple = msgvec; - msg.address_lo =virt_to_phys ((void *)pp->msi_data); msg.address_hi =0x0 ; msg.data = pos; --1.8.4.4
This patchset address Marek Vasut's comments on the previous patches. I decided to split into two separate commits as they are unrelated.
Bjørn Erik Nilsen (2):
PCI: designware: Fix crash in dw_msi_teardown_irq
PCI: designware: Remove redundant call to pci_write_config
drivers/pci/host/pcie-designware.c | 57 +++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 16 deletions(-)
Signature · 1 line
1.8.4.4
PATCH v5 1/2 PCI: designware: Fix crash in dw_msi_teardown_irq
"PCI: designware: Add irq_create_mapping()" resulted in pre-allocated irq descs. Problem was that in assign_irq() these descs were explicitly allocated and hence also freed, resulting in a crash. We also need to clear the entire irq range in teardown. With this commit the teardown basically does exactly the opposite of what was done in setup.
--- drivers/pci/host/pcie-designware.c | 49 ++++++++++++++++++++++++++++---------- 1 file changed, 37 insertions(+), 12 deletions(-)
drivers/pci/host/pcie-designware.c +37 -12
@@ -209,6 +209,25 @@ static int find_valid_pos0(struct pcie_port *pp, int msgvec, int pos, int *pos0)return 0 ; } +static void clear_irq_range (struct pcie_port *pp,unsigned int irq_base, +unsigned int nvec,unsigned int pos) +{ +unsigned int i, res, bit, val; + + i =0 ; +while (i < nvec) { +irq_set_msi_desc_off (irq_base, i,NULL ); +clear_bit (pos + i, pp->msi_irq_in_use); +/* Disable corresponding interrupt on MSI interrupt controller */ + res = ((pos + i) /32 ) *12 ; + bit = (pos + i) %32 ; +dw_pcie_rd_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , &val); + val &= ~(1 << bit); +dw_pcie_wr_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , val); + ++i; + } +} +static int assign_irq (int no_irqs,struct msi_desc *desc,int *pos) {int res, bit, irq, pos0, pos1, i;
@@ -242,11 +261,20 @@ static int assign_irq(int no_irqs, struct msi_desc *desc, int *pos)if (!irq) goto no_valid_irq; +/* +* irq_create_mapping (called from dw_pcie_host_init) pre-allocates +* descs so there is no need to allocate descs here. We can therefore +* assume that if irq_find_mapping above returns non-zero, then the +* descs are also successfully allocated. +*/ + i =0 ;while (i < no_irqs) { +if (irq_set_msi_desc_off (irq, i, desc) !=0 ) { +clear_irq_range (pp, irq, i, pos0); + goto no_valid_irq; + }set_bit (pos0 + i, pp->msi_irq_in_use); -irq_alloc_descs ((irq + i), (irq + i),1 ,0 ); -irq_set_msi_desc (irq + i, desc);/*Enable corresponding interrupt in MSI interrupt controller */ res = ((pos0 + i) /32 ) *12 ; bit = (pos0 + i) %32 ;
@@ -266,7 +294,7 @@ no_valid_irq:static void clear_irq (unsigned int irq) { -int res, bit, val, pos; +unsigned int pos, nvec;struct irq_desc *desc;struct msi_desc *msi;struct pcie_port *pp;
@@ -281,18 +309,15 @@ static void clear_irq(unsigned int irq)return ; } +/* undo what was done in assign_irq */ pos = data->hwirq; + nvec =1 << msi->msi_attrib.multiple; -irq_free_desc (irq); - -clear_bit (pos, pp->msi_irq_in_use); +clear_irq_range (pp, irq, nvec, pos); -/* Disable corresponding interrupt on MSI interrupt controller */ - res = (pos /32 ) *12 ; - bit = pos %32 ; -dw_pcie_rd_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , &val); - val &= ~(1 << bit); -dw_pcie_wr_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , val); +/* all irqs cleared; reset attributes */ + msi->irq =0 ; + msi->msi_attrib.multiple =0 ; } static int dw_msi_setup_irq(struct msi_chip *chip, struct pci_dev *pdev,
Signature · 1 line
1.8.4.4
Re: [PATCH v5 1/2] PCI: designware: Fix crash in dw_msi_teardown_irq
Dear Bjørn Erik Nilsen,
I will be a nitpicking bastard below, sorry ;-/
Quoted 8 lines
"PCI: designware: Add
irq_create_mapping()" resulted in pre-allocated irq descs. Problem was that inassign_irq()these descs were explicitly allocated and hence also freed, resulting in a crash. We also need to clear the entire irq range in teardown. With this commit the teardown basically does exactly the opposite of what was done in setup.Signed-off-by Bjørn Erik Nilsen <[email protected]>
You can stuff your CC list here, like this, if it's more convenient for you then to type all these people on the command line:
The git send-email will automatically send email to this address.
Quoted 10 lines
--- drivers/pci/host/pcie-designware.c | 49++++++++++++++++++++++++++++---------- 1 file changed, 37 insertions(+), 12 deletions(-)
(quoted diff) +0
-0
b/drivers/pci/host/pcie-designware.cindex e33b68b..61345a1 100644
drivers/pci/host/pcie-designware.c static int find_valid_pos0(struct pcie_port *pp, int · line 218
Quoted 10 lines
drivers/pci/host/pcie-designware.c +7
-0@@ -209,6 +209,25 @@ static int find_valid_pos0(struct pcie_port *pp, int msgvec, int pos, int *pos0) return 0; } +static void clear_irq_range (struct pcie_port *pp,unsigned int irq_base, +unsigned int nvec,unsigned int pos) +{ +unsigned int i, res, bit, val; + + i =0 ; +while (i < nvec) {
Won't simple
for (i = 0; i < nvec; i++) {
do here?
drivers/pci/host/pcie-designware.c static int find_valid_pos0(struct pcie_port *pp, int msgvec, int pos, int *pos0) · line 219
Quoted 1 line
drivers/pci/host/pcie-designware.c +1
-0+irq_set_msi_desc_off (irq_base, i,NULL );
This one returns a return value, please handle it. ret = irq_set(...);
if (ret)
goto fail;
drivers/pci/host/pcie-designware.c static int find_valid_pos0(struct pcie_port *pp, int · line 221
Quoted 2 lines
drivers/pci/host/pcie-designware.c +2
-0+clear_bit (pos + i, pp->msi_irq_in_use); +/* Disable corresponding interrupt on MSI interrupt controller
*/
drivers/pci/host/pcie-designware.c static int find_valid_pos0(struct pcie_port *pp, int msgvec, int pos, int *pos0) · line 228
Quoted 7 lines
drivers/pci/host/pcie-designware.c +7
-0+ res = ((pos + i) /32 ) *12 ; + bit = (pos + i) %32 ; +dw_pcie_rd_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , &val); + val &= ~(1 << bit); +dw_pcie_wr_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , val); + ++i; + }
fail:
for (--i; i >=0; i--)
irq...
return ret;
Quoted 62 lines
drivers/pci/host/pcie-designware.c +20
-12+} +static int assign_irq (int no_irqs,struct msi_desc *desc,int *pos) {int res, bit, irq, pos0, pos1, i;@@ -242,11 +261,20 @@ static int assign_irq(int no_irqs, struct msi_desc *desc, int *pos) if (!irq) goto no_valid_irq; +/* +* irq_create_mapping (called from dw_pcie_host_init) pre-allocates +* descs so there is no need to allocate descs here. We can therefore +* assume that if irq_find_mapping above returns non-zero, then the +* descs are also successfully allocated. +*/ + i =0 ;while (i < no_irqs) { +if (irq_set_msi_desc_off (irq, i, desc) !=0 ) { +clear_irq_range (pp, irq, i, pos0); + goto no_valid_irq; + }set_bit (pos0 + i, pp->msi_irq_in_use); -irq_alloc_descs ((irq + i), (irq + i),1 ,0 ); -irq_set_msi_desc (irq + i, desc);/*Enable corresponding interrupt in MSI interrupt controller */ res = ((pos0 + i) /32 ) *12 ; bit = (pos0 + i) %32 ;@@ -266,7 +294,7 @@ no_valid_irq:static void clear_irq (unsigned int irq) { -int res, bit, val, pos; +unsigned int pos, nvec;struct irq_desc *desc;struct msi_desc *msi;struct pcie_port *pp;@@ -281,18 +309,15 @@ static void clear_irq(unsigned int irq)return ; } +/* undo what was done in assign_irq */ pos = data->hwirq; + nvec =1 << msi->msi_attrib.multiple; -irq_free_desc (irq); - -clear_bit (pos, pp->msi_irq_in_use); +clear_irq_range (pp, irq, nvec, pos); -/* Disable corresponding interrupt on MSI interrupt controller */ - res = (pos /32 ) *12 ; - bit = pos %32 ; -dw_pcie_rd_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , &val); - val &= ~(1 << bit); -dw_pcie_wr_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , val); +/* all irqs cleared; reset attributes */ + msi->irq =0 ; + msi->msi_attrib.multiple =0 ; } static int dw_msi_setup_irq(struct msi_chip *chip, struct pci_dev *pdev,
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH v5 1/2] PCI: designware: Fix crash in dw_msi_teardown_irq
Dear Bjørn Erik Nilsen,
[...]
Quoted 6 lines
Quoted 5 lines
Won't simple
for (i = 0; i < nvec; i++) {
do here?
Quoted 7 lines
Yes.
The very same syntax ('while i < nvec') is used in
assign_irq. That is why I wanted to keep it like that to avoid adding too much confusion, or at least make it easy to recognize the same pattern.You still want me to change it?
Your loop does exactly what a for() loop would do, there's no need to emulate it with a while() loop. If you can fix the other one, fix the other one as well please.
Quoted 9 lines
As for the other nitpick, I don't agree with you.
In fact,
dw_msi_teardown_irqhas no return value itself. Moreover, if setting the msi desc to NULL fails, then it simply means there is no irq desc and there is nothing to unwind. Also, skipping the other unwind operations just because that single operation failed would leave the driver in a much worse state. At least in my opinion.What's your opinion on that?
I see this:
$ git grep irq_set_msi_desc_off include/ include/linux/irq.h:extern int irq_set_msi_desc_off(unsigned int irq_base, unsigned int irq_offset,
So it has a return value which needs to be handled. Sorry if I wasn't clear on the first try.
Quoted 2 lines
Now, if you want me to make another patch do you prefer a standalone patch on top of the other patches, or a completely new patchset?
Let's keep iterating :) But please wait for more feedback first.
Best regards,
Marek Vasut
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH v5 1/2] PCI: designware: Fix crash in dw_msi_teardown_irq
Dear Bjørn Erik Nilsen,
Quoted 24 lines
Dear Marek Vasut,
Quoted 15 lines · Marek Vasut
Dear Bjørn Erik Nilsen,
[...]Quoted 12 lines
Quoted 5 lines
Won't simple
for (i = 0; i < nvec; i++) {
do here?
Yes.
The very same syntax ('while i < nvec') is used in
assign_irq. That is why I wanted to keep it like that to avoid adding too much confusion, or at least make it easy to recognize the same pattern.Quoted 5 lines
Quoted 1 line
You still want me to change it?
Your loop does exactly what a
for()loop would do, there's no need to emulate it with awhile()loop. If you can fix the other one, fix the other one as well please.
Quoted 5 lines
Yes, it is syntax sugar indeed and I did it purely for readability as explained. You don't seem to agree with me on the readability argument, so I can make the new code use a for-loop. And then in an unrelated commit (which would look awfully silly) I can change the other while-loop too.
Either way is fine with me, I'm just more inclined to the for loop ;-) Let's wait for what the others have to say, they can judge this argument.
Quoted 19 lines
Quoted 10 lines
Quoted 9 lines
As for the other nitpick, I don't agree with you.
In fact,
dw_msi_teardown_irqhas no return value itself. Moreover, if setting the msi desc to NULL fails, then it simply means there is no irq desc and there is nothing to unwind. Also, skipping the other unwind operations just because that single operation failed would leave the driver in a much worse state. At least in my opinion.What's your opinion on that?
Quoted 8 lines
I see this:
$ git grep irq_set_msi_desc_off include/ include/linux/irq.h:extern int irq_set_msi_desc_off(unsigned int irq_base, unsigned int irq_offset,So it has a return value which needs to be handled. Sorry if I wasn't clear on the first try.
Quoted 8 lines
I agree that return values should be checked.
However based on my reasoning there is absolutely nothing to do with the return value in this particular case, and in fact bailing out will leave the driver in a much worse state (as other unwind operations will be skipped).
That is what I kindly asked you to comment on.
How come you cannot unwind what you did thus far and bail out ? That means the design here is seriously broken, don't you think ?
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Now, if you want me to make another patch do you prefer a standalone patch on top of the other patches, or a completely new patchset?
Let's keep iterating :) But please wait for more feedback first.
OK. I will wait for more feedback then.
ACK, thanks ;-)
Best regards,
Marek Vasut
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH v5 1/2] PCI: designware: Fix crash in dw_msi_teardown_irq
Dear Bjørn Erik Nilsen,
Quoted 72 lines
29. nov. 2013 kl. 18:02 skrev Marek Vasut <[email protected]>:
Quoted 62 lines
Dear Bjørn Erik Nilsen,
Quoted 24 lines
Dear Marek Vasut,
Quoted 15 lines · Marek Vasut
Dear Bjørn Erik Nilsen,
[...]Quoted 12 lines
Quoted 5 lines
Won't simple
for (i = 0; i < nvec; i++) {
do here?
Yes.
The very same syntax ('while i < nvec') is used in
assign_irq. That is why I wanted to keep it like that to avoid adding too much confusion, or at least make it easy to recognize the same pattern.Quoted 5 lines
Quoted 1 line
You still want me to change it?
Your loop does exactly what a
for()loop would do, there's no need to emulate it with awhile()loop. If you can fix the other one, fix the other one as well please.Quoted 5 lines
Yes, it is syntax sugar indeed and I did it purely for readability as explained. You don't seem to agree with me on the readability argument, so I can make the new code use a for-loop. And then in an unrelated commit (which would look awfully silly) I can change the other while-loop too.
Either way is fine with me, I'm just more inclined to the for loop ;-) Let's wait for what the others have to say, they can judge this argument.
Quoted 26 lines
Quoted 10 lines
Quoted 9 lines
As for the other nitpick, I don't agree with you.
In fact,
dw_msi_teardown_irqhas no return value itself. Moreover, if setting the msi desc to NULL fails, then it simply means there is no irq desc and there is nothing to unwind. Also, skipping the other unwind operations just because that single operation failed would leave the driver in a much worse state. At least in my opinion.What's your opinion on that?
Quoted 8 lines
I see this:
$ git grep irq_set_msi_desc_off include/ include/linux/irq.h:extern int irq_set_msi_desc_off(unsigned int irq_base, unsigned int irq_offset,So it has a return value which needs to be handled. Sorry if I wasn't clear on the first try.
I agree that return values should be checked.
However based on my reasoning there is absolutely nothing to do with the return value in this particular case, and in fact bailing out will leave the driver in a much worse state (as other unwind operations will be skipped).
Quoted 4 lines
Quoted 1 line
That is what I kindly asked you to comment on.
How come you cannot unwind what you did thus far and bail out ? That means the design here is seriously broken, don't you think ?
Maybe you need to take a closer look at the patch because the code in question is *the* unwinding code. So bailing out from there doesn't make much sense.
Quoted 5 lines
Also note that I do check the return value of the same function call in setup, but the one you are complaining is for unwinding stuff that has already been setup.
I hope this helps :-)
Please consider me stupid and blind, sorry. I think maybe a WARN_ON(ret) might be a good sign things have gone severely awry here at least. What do you think?
Best regards,
PATCH v5 2/2 PCI: designware: Remove redundant call to pci_write_config
In reply to Bjørn Erik Nilsenwrite_msi_msg() does exactly the same so there is no need to explicitly call pci_write_config and do the same twice.
--- drivers/pci/host/pcie-designware.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/pci/host/pcie-designware.c +4 -4
@@ -345,10 +345,10 @@ static int dw_msi_setup_irq(struct msi_chip *chip, struct pci_dev *pdev,if (irq <0 )return irq; - msg_ctr &= ~PCI_MSI_FLAGS_QSIZE ; - msg_ctr |= msgvec <<4 ; -pci_write_config_word (pdev, desc->msi_attrib.pos +PCI_MSI_FLAGS , - msg_ctr); +/* +* write_msi_msg() will update PCI_MSI_FLAGS so there is +* no need to explicitly call pci_write_config. +*/ desc->msi_attrib.multiple = msgvec; msg.address_lo =virt_to_phys ((void *)pp->msi_data);
Signature · 1 line
1.8.4.4
Quoted 11 lines · Bjørn Erik Nilsen
This patchset address Marek Vasut's comments on the previous patches. I decided to split into two separate commits as they are unrelated.
Bjørn Erik Nilsen (2):
PCI: designware: Fix crash indw_msi_teardown_irq
PCI: designware: Remove redundant call topci_write_configdrivers/pci/host/pcie-designware.c | 57 +++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 16 deletions(-)
I tested these patches on Exynos platform, and checked that the kernel panic was resolved.
Marek Vasut, Bjørn Erik Nilsen,
Is there any issue for accepting these patches?
Best regards,
Jingoo Han
Marek Vasut
Acked-byQuoted 13 lines · Jingoo Han
Quoted 11 lines · Bjørn Erik Nilsen
This patchset address Marek Vasut's comments on the previous patches. I decided to split into two separate commits as they are unrelated.
Bjørn Erik Nilsen (2):
PCI: designware: Fix crash indw_msi_teardown_irq
PCI: designware: Remove redundant call topci_write_configdrivers/pci/host/pcie-designware.c | 57+++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 16 deletions(-)
Quoted 5 lines
I tested these patches on Exynos platform, and checked that the kernel panic was resolved.
Marek Vasut, Bjørn Erik Nilsen,
Is there any issue for accepting these patches?
No issue on my end.
Best regards,
Marek Vasut
Jingoo Han
Acked-byQuoted 17 lines · Marek Vasut
Quoted 16 lines · Jingoo Han
Quoted 11 lines · Bjørn Erik Nilsen
This patchset address Marek Vasut's comments on the previous patches. I decided to split into two separate commits as they are unrelated.
Bjørn Erik Nilsen (2):
PCI: designware: Fix crash indw_msi_teardown_irq
PCI: designware: Remove redundant call topci_write_configdrivers/pci/host/pcie-designware.c | 57+++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 16 deletions(-)
I tested these patches on Exynos platform, and checked that the kernel panic was resolved.
Quoted 6 lines
Quoted 2 lines
Marek Vasut, Bjørn Erik Nilsen,
Is there any issue for accepting these patches?No issue on my end.
Acked-by Marek Vasut <[email protected]>
Best regards,
Jingoo Han
Mohit KUMAR DCG
Tested-byHello Bjørn Erik Nilsen,
Quoted 27 lines
-----Original Message-----
From: Jingoo Han [mailto:[email protected]]
Sent: Thursday, December 05, 2013 7:55 AM
To: 'Bjorn Helgaas'
Cc: 'Marek Vasut'; 'Bjørn Erik Nilsen'; 'Pratyush Anand'; linux- [email protected]; 'Kishon Vijay Abraham I'; Mohit KUMAR DCG; Ajay KHANDELWAL; 'Tim Harvey'; 'Eric Nelson'; 'Troy Kisky'; 'Jingoo Han'
Subject: Re: [PATCH v5 0/2] Kernel oops frompci_disable_msiQuoted 17 lines · Marek Vasut
Quoted 16 lines · Jingoo Han
Quoted 11 lines · Bjørn Erik Nilsen
This patchset address Marek Vasut's comments on the previous patches. I decided to split into two separate commits as they are unrelated.
Bjørn Erik Nilsen (2):
PCI: designware: Fix crash indw_msi_teardown_irq
PCI: designware: Remove redundant call topci_write_configdrivers/pci/host/pcie-designware.c | 57+++++++++++++++++++++++++++----------- 1 file changed, 41
insertions(+), 16 deletions(-)I tested these patches on Exynos platform, and checked that the kernel panic was resolved.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Marek Vasut, Bjørn Erik Nilsen,
Is there any issue for accepting these patches?No issue on my end.
Acked-by Marek Vasut <[email protected]>Acked-by Jingoo Han <[email protected]>
I tested it on SPEAr1310 platform and no kernel panic is found.
As 'write_msi_msg()' redundant call is removed so the following comment messages are not required here:
/* * write_msi_msg() will update PCI_MSI_FLAGS so there is * no need to explicitly call pci_write_config. */
Instead move it to the commit log. Same for the other patch for dw_msi_teardown_irq().
Thanks
Mohit
Bjorn Helgaas
maintainerQuoted 7 lines · Bjørn Erik Nilsen
This patchset address Marek Vasut's comments on the previous patches. I decided to split into two separate commits as they are unrelated.
Bjørn Erik Nilsen (2):
PCI: designware: Fix crash indw_msi_teardown_irq
PCI: designware: Remove redundant call topci_write_config
Hi Bjørn,
There's been a fair amount of discussion about the first patch, so I'm ignoring these for now. Please repost these (again) when that all settles down.
Bjorn
A message that is not in the archive (sent off-list or not yet received): 52a632f0.e42c980a.3d86.ffff8faeSMTPIN_ADDED_MISSING@mx.google.com
Bjorn Helgaas
maintainerQuoted 19 lines · Bjørn Erik Nilsen
Hi Bjorn,
9. des. 2013 kl. 21:43 skrev Bjorn Helgaas <[email protected]>:
Quoted 9 lines · Bjørn Erik Nilsen
Quoted 7 lines
This patchset address Marek Vasut's comments on the previous patches. I decided to split into two separate commits as they are unrelated.
Bjørn Erik Nilsen (2):
PCI: designware: Fix crash indw_msi_teardown_irq
PCI: designware: Remove redundant call topci_write_configQuoted 5 lines
Hi Bjørn,
There's been a fair amount of discussion about the first patch, so I'm ignoring these for now. Please repost these (again) when that all settles down.
Quoted 7 lines
I'm under the impression that all concerns have been addressed and that the discussion has settled down.
What are the outstanding issues to be solved regarding these patches?
I'm asking because these patches are acked-by Marek Vasut and Jingoo Han as well as tested-by Mohit Kumar (and obviously myself).
Jingoo Han confirms he can reproduce the crash on Exynos and that the first patch fixes the problem.
I didn't trace through the discussion closely enough to follow complaints and their resolution or retraction. Can you post the current version including the acks, just so I make sure I get the latest stuff?
Thanks,
Bjorn
A message that is not in the archive (sent off-list or not yet received): 52a63678.4902980a.6fd7.ffffa2b9SMTPIN_ADDED_MISSING@mx.google.com
Bjorn Helgaas
maintainerQuoted 36 lines · Bjørn Erik Nilsen
9. des. 2013 kl. 22:21 skrev Bjorn Helgaas <[email protected]>:
Quoted 24 lines · Bjørn Erik Nilsen
Quoted 23 lines
Hi Bjorn,
9. des. 2013 kl. 21:43 skrev Bjorn Helgaas <[email protected]>:
Quoted 9 lines · Bjørn Erik Nilsen
Quoted 7 lines
This patchset address Marek Vasut's comments on the previous patches. I decided to split into two separate commits as they are unrelated.
Bjørn Erik Nilsen (2):
PCI: designware: Fix crash indw_msi_teardown_irq
PCI: designware: Remove redundant call topci_write_configQuoted 5 lines
Hi Bjørn,
There's been a fair amount of discussion about the first patch, so I'm ignoring these for now. Please repost these (again) when that all settles down.
I'm under the impression that all concerns have been addressed and that the discussion has settled down.
What are the outstanding issues to be solved regarding these patches?
Quoted 8 lines
Quoted 3 lines
I'm asking because these patches are acked-by Marek Vasut and Jingoo Han as well as tested-by Mohit Kumar (and obviously myself).
Jingoo Han confirms he can reproduce the crash on Exynos and that the first patch fixes the problem.
I didn't trace through the discussion closely enough to follow complaints and their resolution or retraction. Can you post the current version including the acks, just so I make sure I get the latest stuff?
Quoted 7 lines
I'm not able to repost right now, but I can do it tomorrow morning if needed.
v5 (as posted) is the latest however, and the amendment is:
Acked-by Marek Vasut <[email protected]>Acked-by Jingoo Han <[email protected]>Tested-by Mohit Kumar <[email protected]>
OK, I added these and applied to a pci/host-designware branch for v3.14. I also added the following cleanup patch as suggested by Marek:
commit 0b8cfb6aa3aabc96177b1e68ef13d2eb5c686606
Author: Bjorn Helgaas <[email protected]> Date: Mon Dec 9 15:11:25 2013 -0700
PCI: designware: Use typical "for" loop idiom
It's conventional to use "for" rather than "while" for simple iteration. No functional change.
drivers/pci/host/pcie-designware.c +2 -6
@@ -214,8 +214,7 @@ static void clear_irq_range(struct pcie_port *pp, unsigned int irq_base, {unsigned int i, res, bit, val; - i =0 ; -while (i < nvec) { +for (i =0 ; i < nvec; i++) {irq_set_msi_desc_off (irq_base, i,NULL );clear_bit (pos + i, pp->msi_irq_in_use);/* Disable corresponding interrupt on MSI interrupt controller */
@@ -224,7 +223,6 @@ static void clear_irq_range(struct pcie_port *pp, unsigned int irq_base,dw_pcie_rd_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , &val); val &= ~(1 << bit);dw_pcie_wr_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , val); - ++i; } }
@@ -268,8 +266,7 @@ static int assign_irq(int no_irqs, struct msi_desc *desc, int *pos)* descs are also successfully allocated. */ - i =0 ; -while (i < no_irqs) { +for (i =0 ; i < no_irqs; i++) {if (irq_set_msi_desc_off (irq, i, desc) !=0 ) {clear_irq_range (pp, irq, i, pos0); goto no_valid_irq;
@@ -281,7 +278,6 @@ static int assign_irq(int no_irqs, struct msi_desc *desc, int *pos)dw_pcie_rd_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , &val); val |=1 << bit;dw_pcie_wr_own_conf (pp,PCIE_MSI_INTR0_ENABLE + res,4 , val); - i++; } *pos = pos0;