NULL pointer dereference in qla24xx_abort_command
Hi Himanshu,
could you please look into this bug?
https://www.spinics.net/lists/linux-scsi/msg138637.html
[2076301.849762] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [2076301.850021] PGD 0 P4D 0 [2076301.850109] Oops: 0002 [#1] SMP PTI [2076301.850219] CPU: 4 PID: 18992 Comm: kworker/u16:1 Not tainted 4.19.0-8-amd64 #1 Debian 4.19.98-1 [2076301.850478] Hardware name: Dell Inc. PowerEdge 2950/0JR815, BIOS 2.7.0 10/30/2010 [2076301.850720] Workqueue: scsi_tmf_4 scmd_eh_abort_handler [scsi_mod] [2076301.850936] RIP: 0010:qla24xx_async_abort_cmd+0x1b/0x250 [qla2xxx] [2076301.851130] Code: e9 19 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 41 55 41 54 55 53 4c 8b 6f 28 4c 8b 7f 20 4c 8b 77 48 <f0> 41 ff 46 04 0f a e f0 41 f6 46 24 04 74 17 f0 41 ff 4e 04 bd 02 [2076301.851663] RSP: 0018:ffffa10f8bbe7da8 EFLAGS: 00010293 [2076301.851820] RAX: 0000000000000800 RBX: ffff8ab8ddd197a8 RCX: 0000000000000070 [2076301.852036] RDX: ffff8ab8de4a8388 RSI: 0000000000000001 RDI: ffff8ab8799b8c40 [2076301.852253] RBP: ffff8ab8dc96c480 R08: ffffffffc03b7860 R09: 0000000000000000 [2076301.852469] R10: 8080808080808080 R11: 0000000000000010 R12: ffff8ab8dea00000 [2076301.852686] R13: ffff8ab8ddd197a8 R14: 0000000000000000 R15: ffff8ab8dd632000 [2076301.852902] FS: 0000000000000000(0000) GS:ffff8ab8e7b00000(0000) knlGS:0000000000000000 [2076301.853142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [2076301.853320] CR2: 0000000000000004 CR3: 00000002203dc000 CR4: 00000000000006e0 [2076301.853536] Call Trace: [2076301.853632] qla24xx_abort_command+0x218/0x2d0 [qla2xxx] [2076301.853799] ? __switch_to_asm+0x41/0x70 [2076301.853924] ? __switch_to_asm+0x35/0x70 [2076301.854056] qla2xxx_eh_abort+0x117/0x310 [qla2xxx] [2076301.854209] scmd_eh_abort_handler+0x85/0x220 [scsi_mod] [2076301.854375] process_one_work+0x1a7/0x3a0 [2076301.854506] worker_thread+0x30/0x390 [2076301.854628] ? create_worker+0x1a0/0x1a0 [2076301.854753] kthread+0x112/0x130 [2076301.854859] ? kthread_bind+0x30/0x30 [2076301.854980] ret_from_fork+0x35/0x40 [2076301.855095] Modules linked in: loop ipmi_ssif radeon coretemp ttm drm_kms_helper drm kvm i2c_algo_bit i5000_edac iTCO_wdt sg iTCO_vendor_support irqbypass evdev i5k_
amb serio_raw joydev ipmi_si rng_core pcc_cpufreq dcdbas pcspkr ipmi_devintf acpi_cpufreq ipmi_msghandler button ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb crypt o_simd cryptd glue_helper aes_x86_64 dm_service_time dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua uas usb_storage hid_generic usbhid hid sr_mod cdrom ses enc losure sd_mod scsi_transport_sas ata_generic qla2xxx ata_piix nvme_fc ehci_pci nvme_fabrics libata uhci_hcd psmouse ehci_hcd nvme_core megaraid_sas usbcore scsi_transport _fc lpc_ich mfd_core scsi_mod usb_common bnx2
[2076301.856887] CR2: 0000000000000004 [2076301.856999] ---[ end trace e9083db8fb76e126 ]--- [2076301.857151] RIP: 0010:qla24xx_async_abort_cmd+0x1b/0x250 [qla2xxx] [2076301.857345] Code: e9 19 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 41 55 41 54 55 53 4c 8b 6f 28 4c 8b 7f 20 4c 8b 77 48 <f0> 41 ff 46 04 0f a e f0 41 f6 46 24 04 74 17 f0 41 ff 4e 04 bd 02 [2076301.857878] RSP: 0018:ffffa10f8bbe7da8 EFLAGS: 00010293 [2076301.858035] RAX: 0000000000000800 RBX: ffff8ab8ddd197a8 RCX: 0000000000000070 [2076301.858251] RDX: ffff8ab8de4a8388 RSI: 0000000000000001 RDI: ffff8ab8799b8c40 [2076301.858467] RBP: ffff8ab8dc96c480 R08: ffffffffc03b7860 R09: 0000000000000000 [2076301.869384] R10: 8080808080808080 R11: 0000000000000010 R12: ffff8ab8dea00000 [2076301.880412] R13: ffff8ab8ddd197a8 R14: 0000000000000000 R15: ffff8ab8dd632000 [2076301.891483] FS: 0000000000000000(0000) GS:ffff8ab8e7b00000(0000) knlGS:0000000000000000 [2076301.902490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [2076301.913344] CR2: 0000000000000004 CR3: 00000002203dc000 CR4: 00000000000006e0 [2077225.259348] mysqld[2155]: segfault at 0 ip 000056409366ad93 sp 00007fa049514450 error 6 in mysqld[564092eb2000+805000] [2077225.270564] Code: c7 45 00 00 00 00 00 8b 7d cc 4c 89 e2 4c 89 f6 e8 62 81 84 ff 49 89 c7 49 39 c4 0f 84 f6 00 00 00 e8 e1 1c 00 00 41 8b 4d 00 <89> 08 85 c9 74 37 4
9 83 ff ff 0f 84 9d 00 00 00 f6 c3 06 75 28 4d
Signature · 1 line
Ondrej Zary
Quoted 55 lines · Ondrej Zary
Hi Himanshu,
could you please look into this bug?
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spinics.net_lists_linux-2Dscsi_msg138637.html&d=DwICAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=XB49zT5-k3oWvfMuBJCeRK5Sk6o3bkGM6ewB3QuTOME&m=ezZ2dt9wVYnM-c6jJKx9bPQ0-7vbSf5HQ0bGqri_iNE&s=wriOIaA9pVuKVrGPMvcZstNdYbzcZgFVw_OKiM0idIg&e=[2076301.849762] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [2076301.850021] PGD 0 P4D 0 [2076301.850109] Oops: 0002 [#1] SMP PTI [2076301.850219] CPU: 4 PID: 18992 Comm: kworker/u16:1 Not tainted 4.19.0-8-amd64 #1 Debian 4.19.98-1 [2076301.850478] Hardware name: Dell Inc. PowerEdge 2950/0JR815, BIOS 2.7.0 10/30/2010 [2076301.850720] Workqueue: scsi_tmf_4 scmd_eh_abort_handler [scsi_mod] [2076301.850936] RIP: 0010:qla24xx_async_abort_cmd+0x1b/0x250 [qla2xxx] [2076301.851130] Code: e9 19 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 41 55 41 54 55 53 4c 8b 6f 28 4c 8b 7f 20 4c 8b 77 48 <f0> 41 ff 46 04 0f a e f0 41 f6 46 24 04 74 17 f0 41 ff 4e 04 bd 02 [2076301.851663] RSP: 0018:ffffa10f8bbe7da8 EFLAGS: 00010293 [2076301.851820] RAX: 0000000000000800 RBX: ffff8ab8ddd197a8 RCX: 0000000000000070 [2076301.852036] RDX: ffff8ab8de4a8388 RSI: 0000000000000001 RDI: ffff8ab8799b8c40 [2076301.852253] RBP: ffff8ab8dc96c480 R08: ffffffffc03b7860 R09: 0000000000000000 [2076301.852469] R10: 8080808080808080 R11: 0000000000000010 R12: ffff8ab8dea00000 [2076301.852686] R13: ffff8ab8ddd197a8 R14: 0000000000000000 R15: ffff8ab8dd632000 [2076301.852902] FS: 0000000000000000(0000) GS:ffff8ab8e7b00000(0000) knlGS:0000000000000000 [2076301.853142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [2076301.853320] CR2: 0000000000000004 CR3: 00000002203dc000 CR4: 00000000000006e0 [2076301.853536] Call Trace: [2076301.853632] qla24xx_abort_command+0x218/0x2d0 [qla2xxx] [2076301.853799] ? __switch_to_asm+0x41/0x70 [2076301.853924] ? __switch_to_asm+0x35/0x70 [2076301.854056] qla2xxx_eh_abort+0x117/0x310 [qla2xxx] [2076301.854209] scmd_eh_abort_handler+0x85/0x220 [scsi_mod] [2076301.854375] process_one_work+0x1a7/0x3a0 [2076301.854506] worker_thread+0x30/0x390 [2076301.854628] ? create_worker+0x1a0/0x1a0 [2076301.854753] kthread+0x112/0x130 [2076301.854859] ? kthread_bind+0x30/0x30 [2076301.854980] ret_from_fork+0x35/0x40 [2076301.855095] Modules linked in: loop ipmi_ssif radeon coretemp ttm drm_kms_helper drm kvm i2c_algo_bit i5000_edac iTCO_wdt sg iTCO_vendor_support irqbypass evdev i5k_amb
serio_rawjoydevipmi_sirng_corepcc_cpufreqdcdbas pcspkripmi_devintfacpi_cpufreqipmi_msghandlerbutton ext4 crc16 mbcache jbd2crc32c_genericfscrypto ecb crypto_simdcryptdglue_helperaes_x86_64dm_service_timedm_multipathdm_modscsi_dh_rdacscsi_dh_emcscsi_dh_aluauasusb_storagehid_genericusbhid hidsr_modcdrom ses enc losuresd_modscsi_transport_sasata_genericqla2xxxata_piixnvme_fcehci_pcinvme_fabricslibatauhci_hcdpsmouseehci_hcdnvme_coremegaraid_sasusbcorescsi_transport_fclpc_ichmfd_corescsi_modusb_commonbnx2[2076301.856887] CR2: 0000000000000004 [2076301.856999] ---[ end trace e9083db8fb76e126 ]--- [2076301.857151] RIP: 0010:qla24xx_async_abort_cmd+0x1b/0x250 [qla2xxx] [2076301.857345] Code: e9 19 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 41 55 41 54 55 53 4c 8b 6f 28 4c 8b 7f 20 4c 8b 77 48 <f0> 41 ff 46 04 0f a e f0 41 f6 46 24 04 74 17 f0 41 ff 4e 04 bd 02 [2076301.857878] RSP: 0018:ffffa10f8bbe7da8 EFLAGS: 00010293 [2076301.858035] RAX: 0000000000000800 RBX: ffff8ab8ddd197a8 RCX: 0000000000000070 [2076301.858251] RDX: ffff8ab8de4a8388 RSI: 0000000000000001 RDI: ffff8ab8799b8c40 [2076301.858467] RBP: ffff8ab8dc96c480 R08: ffffffffc03b7860 R09: 0000000000000000 [2076301.869384] R10: 8080808080808080 R11: 0000000000000010 R12: ffff8ab8dea00000 [2076301.880412] R13: ffff8ab8ddd197a8 R14: 0000000000000000 R15: ffff8ab8dd632000 [2076301.891483] FS: 0000000000000000(0000) GS:ffff8ab8e7b00000(0000) knlGS:0000000000000000 [2076301.902490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [2076301.913344] CR2: 0000000000000004 CR3: 00000002203dc000 CR4: 00000000000006e0 [2077225.259348] mysqld[2155]: segfault at 0 ip 000056409366ad93 sp 00007fa049514450 error 6 in mysqld[564092eb2000+805000] [2077225.270564] Code: c7 45 00 00 00 00 00 8b 7d cc 4c 89 e2 4c 89 f6 e8 62 81 84 ff 49 89 c7 49 39 c4 0f 84 f6 00 00 00 e8 e1 1c 00 00 41 8b 4d 00 <89> 08 85 c9 74 37 4
Quoted 4 lines
9 83 ff ff 0f 84 9d 00 00 00 f6 c3 06 75 28 4d
Ondrej,
Could you pick up the latest upstream kernel and retry your test (even if it takes longer to reproduce)? That way we do not need to spend time on an issue that may have already been fixed.
Regards,
-Arun
Quoted 8 lines · Arun Easi
Quoted 7 lines · Ondrej Zary
Hi Himanshu,
could you please look into this bug?
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spinics.net_list
s_linux-2Dscsi_msg138637.html&d=DwICAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=XB49zT5- k3oWvfMuBJCeRK5Sk6o3bkGM6ewB3QuTOME&m=ezZ2dt9wVYnM-c6jJKx9bPQ0-7vbSf5HQ0bG qri_iNE&s=wriOIaA9pVuKVrGPMvcZstNdYbzcZgFVw_OKiM0idIg&e=
...
Quoted 6 lines
Ondrej,
Could you pick up the latest upstream kernel and retry your test (even if it takes longer to reproduce)? That way we do not need to spend time on an issue that may have already been fixed.
The server is a backup production server so there's no room for such experiments.
BTW. Which one is "the latest upstream kernel"? kernel.org says:
mainline: 5.6-rc7 2020-03-23 stable: 5.5.13 2020-03-25 longterm: 5.4.28 2020-03-25 longterm: 4.19.113 2020-03-25 longterm: 4.14.174 2020-03-20 longterm: 4.9.217 2020-03-20 longterm: 4.4.217 2020-03-20 longterm: 3.16.82 2020-02-11
Maybe you can point me to patches that could fix the problem?
Signature · 1 line
Ondrej Zary
Quoted 8 lines · Arun Easi
Quoted 7 lines · Ondrej Zary
Hi Himanshu,
could you please look into this bug?
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spinics.net_list
s_linux-2Dscsi_msg138637.html&d=DwICAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=XB49zT5- k3oWvfMuBJCeRK5Sk6o3bkGM6ewB3QuTOME&m=ezZ2dt9wVYnM-c6jJKx9bPQ0-7vbSf5HQ0bG qri_iNE&s=wriOIaA9pVuKVrGPMvcZstNdYbzcZgFVw_OKiM0idIg&e=
...
Quoted 9 lines
Ondrej,
Could you pick up the latest upstream kernel and retry your test (even if it takes longer to reproduce)? That way we do not need to spend time on an issue that may have already been fixed.
Regards,
-Arun
It crashed again yesterday (4.19.118-2), almost two months since the last crash :(
Now I installed 5.5.17-1~bpo10+1 from Debian backports and am going to wait another 2 months...
Signature · 1 line
Ondrej Zary
Quoted 17 lines · On Friday 15 May 2020 15:21:49 Ondrej Zary wrote:
Quoted 8 lines · Arun Easi
Quoted 7 lines · Ondrej Zary
Hi Himanshu,
could you please look into this bug?
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spinics.net_list
s_linux-2Dscsi_msg138637.html&d=DwICAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=XB49zT5- k3oWvfMuBJCeRK5Sk6o3bkGM6ewB3QuTOME&m=ezZ2dt9wVYnM-c6jJKx9bPQ0-7vbSf5HQ0bG qri_iNE&s=wriOIaA9pVuKVrGPMvcZstNdYbzcZgFVw_OKiM0idIg&e=...
Quoted 7 lines
Ondrej,
Could you pick up the latest upstream kernel and retry your test (even if it takes longer to reproduce)? That way we do not need to spend time on an issue that may have already been fixed.
Quoted 8 lines
Quoted 2 lines
Regards,
-ArunIt crashed again yesterday (4.19.118-2), almost two months since the last crash :(
Now I installed 5.5.17-1~bpo10+1 from Debian backports and am going to wait another 2 months...
It survived with this kernel (occured after 27 days uptime):
[2420004.525111] qla2xxx [0000:0e:00.0]-801c:4: Abort command issued nexus=4:0:0 -- 2003. [2420004.541110] qla2xxx [0000:0e:00.0]-8009:4: DEVICE RESET ISSUED nexus=4:0:0 cmd=00000000ce4d99b9. [2420004.546654] qla2xxx [0000:0e:00.0]-800e:4: DEVICE RESET SUCCEEDED nexus:4:0:0 cmd=00000000ce4d99b9. [2420004.546797] sd 4:0:0:0: Power-on or device reset occurred [2420004.568334] sd 3:0:0:0: Power-on or device reset occurred
Signature · 1 line
Ondrej Zary