NULL pointer dereference in qla24xx_abort_command, kernel 4.19.98 (Debian)
Hello,
a couple of days after upgrading a server from Debian 9 (kernel 4.9.210-1) to 10 (kernel 4.19.98), qla2xxx crashed, along with mysql.
There is an EMC CX3 array connected through the fibre-channel adapter. No errors are present in EMC event log.
This server was running without any problems since Debian 4. Is this a known bug?
[979178.888922] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [979178.889160] PGD 0 P4D 0 [979178.889243] Oops: 0002 [#1] SMP PTI [979178.889362] CPU: 6 PID: 11060 Comm: kworker/u16:2 Not tainted 4.19.0-8-amd64 #1 Debian 4.19.98-1 [979178.889617] Hardware name: Dell Inc. PowerEdge 2950/0JR815, BIOS 2.7.0 10/30/2010 [979178.889855] Workqueue: scsi_tmf_4 scmd_eh_abort_handler [scsi_mod] [979178.890069] RIP: 0010:qla24xx_async_abort_cmd+0x1b/0x250 [qla2xxx] [979178.890258] Code: e9 19 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 41 55 41 54 55 53 4c 8b 6f 28 4c 8b 7f 20 4c 8b 77 48 <f0> 41 ff 46 04 0f ae f0 41 f6 46 24 04 74 17 f0 41 ff 4e 04 bd 02 [979178.890801] RSP: 0018:ffffb1250ba83da8 EFLAGS: 00010293 [979178.890966] RAX: 0000000000000800 RBX: ffff93b89db837a8 RCX: 00000000000005f4 [979178.891178] RDX: ffff93b89e28afa8 RSI: 0000000000000001 RDI: ffff93b8a5018fc0 [979178.891389] RBP: ffff93b89ccb89c0 R08: ffffffffc0595860 R09: 0000000000000000 [979178.891600] R10: 8080808080808080 R11: 0000000000000010 R12: ffff93b89db82000 [979178.891811] R13: ffff93b89db837a8 R14: 0000000000000000 R15: ffff93b89d88a800 [979178.892023] FS: 0000000000000000(0000) GS:ffff93b8a7b80000(0000) knlGS:0000000000000000 [979178.892258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [979178.892430] CR2: 0000000000000004 CR3: 000000021a62a000 CR4: 00000000000006e0 [979178.892642] Call Trace: [979178.892748] qla24xx_abort_command+0x218/0x2d0 [qla2xxx] [979178.892911] ? __switch_to_asm+0x41/0x70 [979178.893031] ? __switch_to_asm+0x35/0x70 [979178.893160] qla2xxx_eh_abort+0x117/0x310 [qla2xxx] [979178.893323] scmd_eh_abort_handler+0x85/0x220 [scsi_mod] [979178.893484] process_one_work+0x1a7/0x3a0 [979178.893611] worker_thread+0x30/0x390 [979178.893727] ? create_worker+0x1a0/0x1a0 [979178.893847] kthread+0x112/0x130 [979178.893948] ? kthread_bind+0x30/0x30 [979178.894064] ret_from_fork+0x35/0x40 [979178.894174] Modules linked in: loop ipmi_ssif radeon ttm drm_kms_helper drm coretemp i2c_algo_bit iTCO_wdt iTCO_vendor_support ipmi_si joydev kvm sg evdev i5000_edac
ipmi_devintf pcc_cpufreq ipmi_msghandler rng_core i5k_amb irqbypass dcdbas serio_raw acpi_cpufreq button pcspkr ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb crypto _simd cryptd glue_helper aes_x86_64 dm_service_time dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua uas usb_storage hid_generic usbhid hid sr_mod ses cdrom encl osure sd_mod scsi_transport_sas ata_generic qla2xxx uhci_hcd ehci_pci ehci_hcd psmouse ata_piix nvme_fc libata nvme_fabrics usbcore nvme_core megaraid_sas scsi_transport_ fc scsi_mod lpc_ich mfd_core usb_common bnx2
[979178.895968] CR2: 0000000000000004 [979178.896075] ---[ end trace 4d42692cc0dc3c87 ]--- [979178.896225] RIP: 0010:qla24xx_async_abort_cmd+0x1b/0x250 [qla2xxx] [979178.896414] Code: e9 19 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 41 55 41 54 55 53 4c 8b 6f 28 4c 8b 7f 20 4c 8b 77 48 <f0> 41 ff 46 04 0f ae f0 41 f6 46 24 04 74 17 f0 41 ff 4e 04 bd 02 [979178.896956] RSP: 0018:ffffb1250ba83da8 EFLAGS: 00010293 [979178.897121] RAX: 0000000000000800 RBX: ffff93b89db837a8 RCX: 00000000000005f4 [979178.897332] RDX: ffff93b89e28afa8 RSI: 0000000000000001 RDI: ffff93b8a5018fc0 [979178.897544] RBP: ffff93b89ccb89c0 R08: ffffffffc0595860 R09: 0000000000000000 [979178.908415] R10: 8080808080808080 R11: 0000000000000010 R12: ffff93b89db82000 [979178.919419] R13: ffff93b89db837a8 R14: 0000000000000000 R15: ffff93b89d88a800 [979178.930444] FS: 0000000000000000(0000) GS:ffff93b8a7b80000(0000) knlGS:0000000000000000 [979178.941366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [979178.952142] CR2: 0000000000000004 CR3: 000000021a62a000 CR4: 00000000000006e0 [980103.072740] mysqld[2175]: segfault at 0 ip 000055bbc5cd2d93 sp 00007f2362ffb450 error 6 in mysqld[55bbc551a000+805000] [980103.083956] Code: c7 45 00 00 00 00 00 8b 7d cc 4c 89 e2 4c 89 f6 e8 62 81 84 ff 49 89 c7 49 39 c4 0f 84 f6 00 00 00 e8 e1 1c 00 00 41 8b 4d 00 <89> 08 85 c9 74 37 49 83 ff ff 0f 84 9d 00 00 00 f6 c3 06 75 28 4d
Signature · 1 line
Ondrej Zary
Quoted 8 lines · Ondrej Zary
a couple of days after upgrading a server from Debian 9 (kernel 4.9.210-1) to 10 (kernel 4.19.98), qla2xxx crashed, along with mysql.
There is an EMC CX3 array connected through the fibre-channel adapter. No errors are present in EMC event log.
This server was running without any problems since Debian 4. Is this a known bug?
Please report issues encountered with Debian kernels in the Debian bug tracker. If you want the upstream community to assist please retest with an upstream kernel.
Thanks,
Bart.
Quoted 7 lines · On Sunday 23 February 2020 20:26:39 Bart Van Assche wrote:
Quoted 6 lines · Ondrej Zary
a couple of days after upgrading a server from Debian 9 (kernel 4.9.210-1) to 10 (kernel 4.19.98), qla2xxx crashed, along with mysql.
There is an EMC CX3 array connected through the fibre-channel adapter. No errors are present in EMC event log.
Quoted 6 lines
Quoted 2 lines
This server was running without any problems since Debian 4. Is this a known bug?
Please report issues encountered with Debian kernels in the Debian bug tracker. If you want the upstream community to assist please retest with an upstream kernel.
Debian kernel does not have any patches related to qla2xxx driver:
https://salsa.debian.org/kernel-team/linux/raw/debian/4.19.98-1/debian/patches/series
It crashed after running for 11 days. Not a quick&easy test.
Signature · 1 line
Ondrej Zary
Quoted 11 lines · Ondrej Zary
Quoted 10 lines · On Sunday 23 February 2020 20:26:39 Bart Van Assche wrote:
Quoted 8 lines · Ondrej Zary
a couple of days after upgrading a server from Debian 9 (kernel 4.9.210-1) to 10 (kernel 4.19.98), qla2xxx crashed, along with mysql.
There is an EMC CX3 array connected through the fibre-channel adapter. No errors are present in EMC event log.
This server was running without any problems since Debian 4. Is this a known bug?
Quoted 8 lines
Quoted 3 lines
Please report issues encountered with Debian kernels in the Debian bug tracker. If you want the upstream community to assist please retest with an upstream kernel.
Debian kernel does not have any patches related to qla2xxx driver:
https://salsa.debian.org/kernel-team/linux/raw/debian/4.19.98-1/debian/patches/seriesIt crashed after running for 11 days. Not a quick&easy test.
It would help a lot if the crash address would be translated into a source code line number. Something like the following commands should do the trick:
$ gdb drivers/scsi/qla2xxx/qla2xxx.ko (gdb) list *(qla24xx_async_abort_cmd+0x1b)
Thanks,
Bart.
Quoted 20 lines · Bart Van Assche
Quoted 19 lines · Ondrej Zary
Quoted 7 lines · On Sunday 23 February 2020 20:26:39 Bart Van Assche wrote:
Quoted 6 lines · Ondrej Zary
a couple of days after upgrading a server from Debian 9 (kernel 4.9.210-1) to 10 (kernel 4.19.98), qla2xxx crashed, along with mysql.
There is an EMC CX3 array connected through the fibre-channel adapter. No errors are present in EMC event log.
Quoted 6 lines
Quoted 2 lines
This server was running without any problems since Debian 4. Is this a known bug?
Please report issues encountered with Debian kernels in the Debian bug tracker. If you want the upstream community to assist please retest with an upstream kernel.
Debian kernel does not have any patches related to qla2xxx driver:
https://salsa.debian.org/kernel-team/linux/raw/debian/4.19.98-1/debian/pa
tches/series
Quoted 7 lines
Quoted 1 line
It crashed after running for 11 days. Not a quick&easy test.
It would help a lot if the crash address would be translated into a source code line number. Something like the following commands should do the trick:
$ gdb drivers/scsi/qla2xxx/qla2xxx.ko (gdb) list *(qla24xx_async_abort_cmd+0x1b)
Looks like it's in some inlined function.
/usr/src/linux-source-4.19# gdb /lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko GNU gdb (Debian 8.2.1-2+b3) 8.2.1
...
Reading symbols from /lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko...Reading symbols from /usr/lib/debug//lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko...done. done.
(gdb) list *(qla24xx_async_abort_cmd+0x1b)0xf88b is in qla24xx_async_abort_cmd (./arch/x86/include/asm/atomic.h:97).
92 * 93 * Atomically increments @v by 1. 94 */
95 static __always_inline void arch_atomic_inc(atomic_t *v)
96 {
97 asm volatile(LOCK_PREFIX "incl %0"
98 : "+m" (v->counter) :: "memory");
99 }
100 #define arch_atomic_inc arch_atomic_inc
101
(gdb) list *(qla24xx_abort_command+0x218)0x22238 is in qla24xx_abort_command (./drivers/scsi/qla2xxx/qla_mbx.c:3084). 3079
3080 if (vha->flags.qpairs_available && sp->qpair)
3081 req = sp->qpair->req;
3082
3083 if (ql2xasynctmfenable)
3084 return qla24xx_async_abort_command(sp);
3085
3086 spin_lock_irqsave(&ha->hardware_lock, flags);
3087 for (handle = 1; handle < req->num_outstanding_cmds; handle++) {
3088 if (req->outstanding_cmds[handle] == sp)
(gdb) list *(qla2xxx_eh_abort+0x117)0x15e7 is in qla2xxx_eh_abort (./drivers/scsi/qla2xxx/qla_os.c:1314).
1309 /* Get a reference to the sp and drop the lock.*/
1310 sp_get(sp);
1311
1312 spin_unlock_irqrestore(&ha->hardware_lock, flags);
1313 rval = ha->isp_ops->abort_command(sp);
1314 if (rval) {1315 if (rval == QLA_FUNCTION_PARAMETER_ERROR)
1316 ret = SUCCESS;
1317 else
1318 ret = FAILED;
(gdb) disassemble qla24xx_async_abort_cmd
Dump of assembler code for function qla24xx_async_abort_cmd:
0x000000000000f870 <+0>: callq 0xf875 <qla24xx_async_abort_cmd+5> 0x000000000000f875 <+5>: push %r15 0x000000000000f877 <+7>: push %r14 0x000000000000f879 <+9>: push %r13 0x000000000000f87b <+11>: push %r12 0x000000000000f87d <+13>: push %rbp 0x000000000000f87e <+14>: push %rbx 0x000000000000f87f <+15>: mov 0x28(%rdi),%r13 0x000000000000f883 <+19>: mov 0x20(%rdi),%r15 0x000000000000f887 <+23>: mov 0x48(%rdi),%r14 0x000000000000f88b <+27>: lock incl 0x4(%r14) 0x000000000000f890 <+32>: mfence 0x000000000000f893 <+35>: testb $0x4,0x24(%r14) 0x000000000000f898 <+40>: je 0xf8b1 <qla24xx_async_abort_cmd+65> 0x000000000000f89a <+42>: lock decl 0x4(%r14) 0x000000000000f89f <+47>: mov $0x102,%ebp 0x000000000000f8a4 <+52>: pop %rbx 0x000000000000f8a5 <+53>: mov %ebp,%eax 0x000000000000f8a7 <+55>: pop %rbp 0x000000000000f8a8 <+56>: pop %r12 0x000000000000f8aa <+58>: pop %r13 0x000000000000f8ac <+60>: pop %r14 0x000000000000f8ae <+62>: pop %r15 0x000000000000f8b0 <+64>: retq 0x000000000000f8b1 <+65>: mov %rdi,%rbp 0x000000000000f8b4 <+68>: mov 0x30(%r14),%rdi 0x000000000000f8b8 <+72>: mov %esi,%r12d 0x000000000000f8bb <+75>: mov $0x6000c0,%esi 0x000000000000f8c0 <+80>: callq 0xf8c5 <qla24xx_async_abort_cmd+85> 0x000000000000f8c5 <+85>: mov %rax,%rbx 0x000000000000f8c8 <+88>: test %rax,%rax 0x000000000000f8cb <+91>: je 0xf89a <qla24xx_async_abort_cmd+42> 0x000000000000f8cd <+93>: lea 0x8(%rax),%rdi 0x000000000000f8d1 <+97>: mov %rax,%rcx 0x000000000000f8d4 <+100>: movq $0x0,(%rax) 0x000000000000f8db <+107>: mov $0xc,%edx 0x000000000000f8e0 <+112>: movq $0x0,0x180(%rax) 0x000000000000f8eb <+123>: and $0xfffffffffffffff8,%rdi 0x000000000000f8ef <+127>: xor %eax,%eax 0x000000000000f8f1 <+129>: sub %rdi,%rcx 0x000000000000f8f4 <+132>: add $0x188,%ecx 0x000000000000f8fa <+138>: shr $0x3,%ecx 0x000000000000f8fd <+141>: rep stos %rax,%es:(%rdi) 0x000000000000f900 <+144>: mov %r15,0x20(%rbx) 0x000000000000f904 <+148>: movl $0x1,0x40(%rbx) 0x000000000000f90b <+155>: mov 0x18(%r14),%rax 0x000000000000f90f <+159>: mov %dx,0x36(%rbx) 0x000000000000f913 <+163>: movq $0x0,0x38(%rbx) 0x000000000000f91b <+171>: mov %rax,0x28(%rbx) 0x000000000000f91f <+175>: lea 0x50(%rbx),%rax 0x000000000000f923 <+179>: mov %rax,0x50(%rbx) 0x000000000000f927 <+183>: mov %rax,0x58(%rbx) 0x000000000000f92b <+187>: mov 0x48(%rbp),%rax 0x000000000000f92f <+191>: mov %rax,0x48(%rbx) 0x000000000000f933 <+195>: test %r12b,%r12b 0x000000000000f936 <+198>: je 0xf941 <qla24xx_async_abort_cmd+209> 0x000000000000f938 <+200>: mov $0x40,%eax 0x000000000000f93d <+205>: mov %ax,0x34(%rbx) 0x000000000000f941 <+209>: lea 0xa0(%rbx),%rdi 0x000000000000f948 <+216>: mov $0x0,%rdx 0x000000000000f94f <+223>: mov $0x0,%rsi 0x000000000000f956 <+230>: movq $0x0,0x170(%rbx) 0x000000000000f961 <+241>: lea 0x148(%rbx),%r14 0x000000000000f968 <+248>: movl $0x0,0x98(%rbx) 0x000000000000f972 <+258>: callq 0xf977 <qla24xx_async_abort_cmd+263> 0x000000000000f977 <+263>: xor %r8d,%r8d 0x000000000000f97a <+266>: xor %ecx,%ecx 0x000000000000f97c <+268>: xor %edx,%edx 0x000000000000f97e <+270>: mov $0x0,%rsi 0x000000000000f985 <+277>: mov %r14,%rdi 0x000000000000f988 <+280>: callq 0xf98d <qla24xx_async_abort_cmd+285> 0x000000000000f98d <+285>: mov 0x0(%rip),%rax # 0xf994 <qla24xx_async_abort_cmd+292> 0x000000000000f994 <+292>: lea 0x78(%rbx),%rdi 0x000000000000f998 <+296>: mov $0x0,%rdx 0x000000000000f99f <+303>: mov $0x0,%rsi 0x000000000000f9a6 <+310>: movl $0x0,0x70(%rbx) 0x000000000000f9ad <+317>: add $0x2904,%rax 0x000000000000f9b3 <+323>: movq $0x0,0x180(%rbx) 0x000000000000f9be <+334>: mov %rax,0x158(%rbx) 0x000000000000f9c5 <+341>: callq 0xf9ca <qla24xx_async_abort_cmd+346> 0x000000000000f9ca <+346>: mov 0x28(%rbx),%rax 0x000000000000f9ce <+350>: mov 0x448(%rax),%rax 0x000000000000f9d5 <+357>: testb $0x2,0x15a(%rax) 0x000000000000f9dc <+364>: jne 0xfa80 <qla24xx_async_abort_cmd+528> 0x000000000000f9e2 <+370>: mov %r14,%rdi 0x000000000000f9e5 <+373>: callq 0xf9ea <qla24xx_async_abort_cmd+378> 0x000000000000f9ea <+378>: mov 0x30(%rbp),%r8d 0x000000000000f9ee <+382>: mov 0x48(%rbp),%rax 0x000000000000f9f2 <+386>: mov %r13,%rsi 0x000000000000f9f5 <+389>: movzwl 0x36(%rbp),%r9d 0x000000000000f9fa <+394>: mov $0x507c,%edx 0x000000000000f9ff <+399>: mov $0x2000000,%edi 0x000000000000fa04 <+404>: mov $0x0,%rcx 0x000000000000fa0b <+411>: mov %r8d,0x90(%rbx) 0x000000000000fa12 <+418>: mov 0x48(%rax),%rax 0x000000000000fa16 <+422>: movzwl 0x40(%rax),%eax 0x000000000000fa1a <+426>: movq $0x0,0x178(%rbx) 0x000000000000fa25 <+437>: mov %ax,0x96(%rbx) 0x000000000000fa2c <+444>: callq 0xfa31 <qla24xx_async_abort_cmd+449> 0x000000000000fa31 <+449>: mov %rbx,%rdi 0x000000000000fa34 <+452>: callq 0xfa39 <qla24xx_async_abort_cmd+457> 0x000000000000fa39 <+457>: mov %eax,%ebp 0x000000000000fa3b <+459>: test %eax,%eax 0x000000000000fa3d <+461>: jne 0xfa64 <qla24xx_async_abort_cmd+500> 0x000000000000fa3f <+463>: test %r12b,%r12b 0x000000000000fa42 <+466>: je 0xf8a4 <qla24xx_async_abort_cmd+52> 0x000000000000fa48 <+472>: lea 0x98(%rbx),%rdi 0x000000000000fa4f <+479>: callq 0xfa54 <qla24xx_async_abort_cmd+484> 0x000000000000fa54 <+484>: cmpw $0x0,0x94(%rbx) 0x000000000000fa5c <+492>: mov $0x102,%eax 0x000000000000fa61 <+497>: cmovne %eax,%ebp 0x000000000000fa64 <+500>: mov 0x180(%rbx),%rax 0x000000000000fa6b <+507>: mov %rbx,%rdi 0x000000000000fa6e <+510>: callq 0xfa73 <qla24xx_async_abort_cmd+515> 0x000000000000fa73 <+515>: mov %ebp,%eax 0x000000000000fa75 <+517>: pop %rbx 0x000000000000fa76 <+518>: pop %rbp 0x000000000000fa77 <+519>: pop %r12 0x000000000000fa79 <+521>: pop %r13 0x000000000000fa7b <+523>: pop %r14 0x000000000000fa7d <+525>: pop %r15 0x000000000000fa7f <+527>: retq 0x000000000000fa80 <+528>: cmpw $0xa,0x36(%rbx) 0x000000000000fa85 <+533>: jne 0xf9e2 <qla24xx_async_abort_cmd+370> 0x000000000000fa8b <+539>: lea 0xe8(%rbx),%rdi 0x000000000000fa92 <+546>: mov $0x0,%rdx 0x000000000000fa99 <+553>: mov $0x0,%rsi 0x000000000000faa0 <+560>: movl $0x0,0xe0(%rbx) 0x000000000000faaa <+570>: callq 0xfaaf <qla24xx_async_abort_cmd+575> 0x000000000000faaf <+575>: jmpq 0xf9e2 <qla24xx_async_abort_cmd+370>
End of assembler dump.
Signature · 1 line
Ondrej Zary
Quoted 25 lines · Ondrej Zary
Looks like it's in some inlined function.
/usr/src/linux-source-4.19# gdb
/lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.koGNU gdb (Debian 8.2.1-2+b3) 8.2.1
...
Reading symbols from/lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko...Readingsymbols from/usr/lib/debug//lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko...done. done.(gdb) list *(qla24xx_async_abort_cmd+0x1b)
0xf88bis inqla24xx_async_abort_cmd(./arch/x86/include/asm/atomic.h:97).92 * 93 * Atomically increments @v by 1. 94 */95 static
__always_inlinevoidarch_atomic_inc(atomic_t *v)
96 {
97 asm volatile(LOCK_PREFIX "incl %0"
98 : "+m" (v->counter) :: "memory");
99 }
100 #definearch_atomic_incarch_atomic_inc[ ... ]
(gdb) disassemble
qla24xx_async_abort_cmd
Dump of assembler code for functionqla24xx_async_abort_cmd:
Quoted 12 lines
0x000000000000f870 <+0>: callq 0xf875 <qla24xx_async_abort_cmd+5> 0x000000000000f875 <+5>: push %r15 0x000000000000f877 <+7>: push %r14 0x000000000000f879 <+9>: push %r13 0x000000000000f87b <+11>: push %r12 0x000000000000f87d <+13>: push %rbp 0x000000000000f87e <+14>: push %rbx 0x000000000000f87f <+15>: mov 0x28(%rdi),%r13 0x000000000000f883 <+19>: mov 0x20(%rdi),%r15 0x000000000000f887 <+23>: mov 0x48(%rdi),%r14 0x000000000000f88b <+27>: lock incl 0x4(%r14) 0x000000000000f890 <+32>: mfence
Thanks, this is very helpful. I think the above means that the crash is triggered by the following code:
sp = qla2xxx_get_qpair_sp(cmd_sp->qpair, cmd_sp->fcport, GFP_KERNEL);
From the start of qla2xxx_get_qpair_sp():
QLA_QPAIR_MARK_BUSY(qpair, bail);
From qla_def.h:
#define QLA_QPAIR_MARK_BUSY(__qpair, __bail) do { \
atomic_inc(&__qpair->ref_count); \
mb(); \
if (__qpair->delete_in_progress) { \
atomic_dec(&__qpair->ref_count); \
__bail = 1; \
} else { \
__bail = 0; \
} \
} while (0)One of the changes between kernel version v4.9.210 and v4.19.98 is the
following: "qla2xxx: Add multiple queue pair functionality". I think the
above information means that the cmd_sp->qpair pointer is NULL. I will
let QLogic recommend a solution.
Bart.
Quoted 62 lines · On Tuesday 25 February 2020 04:41:48 Bart Van Assche wrote:
Quoted 25 lines · Ondrej Zary
Looks like it's in some inlined function.
/usr/src/linux-source-4.19# gdb
/lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.koGNU gdb (Debian 8.2.1-2+b3) 8.2.1
...
Reading symbols from/lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko...Readingsymbols from/usr/lib/debug//lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko...done. done.(gdb) list *(qla24xx_async_abort_cmd+0x1b)
0xf88bis inqla24xx_async_abort_cmd(./arch/x86/include/asm/atomic.h:97).92 * 93 * Atomically increments @v by 1. 94 */95 static
__always_inlinevoidarch_atomic_inc(atomic_t *v)
96 {
97 asm volatile(LOCK_PREFIX "incl %0"
98 : "+m" (v->counter) :: "memory");
99 }
100 #definearch_atomic_incarch_atomic_inc[ ... ]
(gdb) disassemble
qla24xx_async_abort_cmd
Dump of assembler code for functionqla24xx_async_abort_cmd:Quoted 12 lines
0x000000000000f870 <+0>: callq 0xf875 <qla24xx_async_abort_cmd+5> 0x000000000000f875 <+5>: push %r15 0x000000000000f877 <+7>: push %r14 0x000000000000f879 <+9>: push %r13 0x000000000000f87b <+11>: push %r12 0x000000000000f87d <+13>: push %rbp 0x000000000000f87e <+14>: push %rbx 0x000000000000f87f <+15>: mov 0x28(%rdi),%r13 0x000000000000f883 <+19>: mov 0x20(%rdi),%r15 0x000000000000f887 <+23>: mov 0x48(%rdi),%r14 0x000000000000f88b <+27>: lock incl 0x4(%r14) 0x000000000000f890 <+32>: mfenceThanks, this is very helpful. I think the above means that the crash is triggered by the following code:
sp = qla2xxx_get_qpair_sp(cmd_sp->qpair, cmd_sp->fcport, GFP_KERNEL);From the start of
qla2xxx_get_qpair_sp():QLA_QPAIR_MARK_BUSY(qpair, bail);From
qla_def.h:#define QLA_QPAIR_MARK_BUSY(__qpair, __bail) do { \ atomic_inc(&__qpair->ref_count); \ mb(); \ if (__qpair->delete_in_progress) { \ atomic_dec(&__qpair->ref_count); \ __bail = 1; \ } else { \ __bail = 0; \ } \ } while (0)
Quoted 4 lines
One of the changes between kernel version v4.9.210 and v4.19.98 is the
following: "qla2xxx: Add multiple queue pair functionality". I think the
above information means that thecmd_sp->qpairpointer is NULL. I will
let QLogic recommend a solution.
Thank you very much for the analysis.
Unfortunately, QLogic does not seem to care...
Signature · 1 line
Ondrej Zary
Quoted 64 lines · On Thursday 27 February 2020 18:09:07 Ondrej Zary wrote:
Quoted 62 lines · On Tuesday 25 February 2020 04:41:48 Bart Van Assche wrote:
Quoted 25 lines · Ondrej Zary
Looks like it's in some inlined function.
/usr/src/linux-source-4.19# gdb
/lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.koGNU gdb (Debian 8.2.1-2+b3) 8.2.1
...
Reading symbols from/lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko...Readingsymbols from/usr/lib/debug//lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko...done. done.(gdb) list *(qla24xx_async_abort_cmd+0x1b)
0xf88bis inqla24xx_async_abort_cmd(./arch/x86/include/asm/atomic.h:97).92 * 93 * Atomically increments @v by 1. 94 */95 static
__always_inlinevoidarch_atomic_inc(atomic_t *v)
96 {
97 asm volatile(LOCK_PREFIX "incl %0"
98 : "+m" (v->counter) :: "memory");
99 }
100 #definearch_atomic_incarch_atomic_inc[ ... ]
(gdb) disassemble
qla24xx_async_abort_cmd
Dump of assembler code for functionqla24xx_async_abort_cmd:Quoted 12 lines
0x000000000000f870 <+0>: callq 0xf875 <qla24xx_async_abort_cmd+5> 0x000000000000f875 <+5>: push %r15 0x000000000000f877 <+7>: push %r14 0x000000000000f879 <+9>: push %r13 0x000000000000f87b <+11>: push %r12 0x000000000000f87d <+13>: push %rbp 0x000000000000f87e <+14>: push %rbx 0x000000000000f87f <+15>: mov 0x28(%rdi),%r13 0x000000000000f883 <+19>: mov 0x20(%rdi),%r15 0x000000000000f887 <+23>: mov 0x48(%rdi),%r14 0x000000000000f88b <+27>: lock incl 0x4(%r14) 0x000000000000f890 <+32>: mfenceThanks, this is very helpful. I think the above means that the crash is triggered by the following code:
sp = qla2xxx_get_qpair_sp(cmd_sp->qpair, cmd_sp->fcport, GFP_KERNEL);From the start of
qla2xxx_get_qpair_sp():QLA_QPAIR_MARK_BUSY(qpair, bail);From
qla_def.h:#define QLA_QPAIR_MARK_BUSY(__qpair, __bail) do { \ atomic_inc(&__qpair->ref_count); \ mb(); \ if (__qpair->delete_in_progress) { \ atomic_dec(&__qpair->ref_count); \ __bail = 1; \ } else { \ __bail = 0; \ } \ } while (0)
Quoted 7 lines
Quoted 4 lines
One of the changes between kernel version v4.9.210 and v4.19.98 is the
following: "qla2xxx: Add multiple queue pair functionality". I think the
above information means that thecmd_sp->qpairpointer is NULL. I will
let QLogic recommend a solution.Thank you very much for the analysis.
Unfortunately, QLogic does not seem to care...
Let's try to CC the people at Cavium that signed-off the commit.
Signature · 1 line
Ondrej Zary
Quoted 70 lines · On Monday 02 March 2020 23:26:08 Ondrej Zary wrote:
Quoted 69 lines · On Thursday 27 February 2020 18:09:07 Ondrej Zary wrote:
Quoted 66 lines · On Tuesday 25 February 2020 04:41:48 Bart Van Assche wrote:
Quoted 25 lines · Ondrej Zary
Looks like it's in some inlined function.
/usr/src/linux-source-4.19# gdb
/lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.koGNU gdb (Debian 8.2.1-2+b3) 8.2.1
...
Reading symbols from/lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko...Readingsymbols from/usr/lib/debug//lib/modules/4.19.0-8-amd64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko...done. done.(gdb) list *(qla24xx_async_abort_cmd+0x1b)
0xf88bis inqla24xx_async_abort_cmd(./arch/x86/include/asm/atomic.h:97).92 * 93 * Atomically increments @v by 1. 94 */95 static
__always_inlinevoidarch_atomic_inc(atomic_t *v)
96 {
97 asm volatile(LOCK_PREFIX "incl %0"
98 : "+m" (v->counter) :: "memory");
99 }
100 #definearch_atomic_incarch_atomic_inc[ ... ]
(gdb) disassemble
qla24xx_async_abort_cmd
Dump of assembler code for functionqla24xx_async_abort_cmd:Quoted 12 lines
0x000000000000f870 <+0>: callq 0xf875 <qla24xx_async_abort_cmd+5> 0x000000000000f875 <+5>: push %r15 0x000000000000f877 <+7>: push %r14 0x000000000000f879 <+9>: push %r13 0x000000000000f87b <+11>: push %r12 0x000000000000f87d <+13>: push %rbp 0x000000000000f87e <+14>: push %rbx 0x000000000000f87f <+15>: mov 0x28(%rdi),%r13 0x000000000000f883 <+19>: mov 0x20(%rdi),%r15 0x000000000000f887 <+23>: mov 0x48(%rdi),%r14 0x000000000000f88b <+27>: lock incl 0x4(%r14) 0x000000000000f890 <+32>: mfenceThanks, this is very helpful. I think the above means that the crash is triggered by the following code:
sp = qla2xxx_get_qpair_sp(cmd_sp->qpair, cmd_sp->fcport, GFP_KERNEL);From the start of
qla2xxx_get_qpair_sp():QLA_QPAIR_MARK_BUSY(qpair, bail);From
qla_def.h:#define QLA_QPAIR_MARK_BUSY(__qpair, __bail) do { \ atomic_inc(&__qpair->ref_count); \ mb(); \ if (__qpair->delete_in_progress) { \ atomic_dec(&__qpair->ref_count); \ __bail = 1; \ } else { \ __bail = 0; \ } \ } while (0)One of the changes between kernel version v4.9.210 and v4.19.98 is the
following: "qla2xxx: Add multiple queue pair functionality". I think the
above information means that thecmd_sp->qpairpointer is NULL. I will
let QLogic recommend a solution.
Quoted 4 lines
Quoted 2 lines
Thank you very much for the analysis.
Unfortunately, QLogic does not seem to care...Let's try to CC the people at Cavium that signed-off the commit.
No reply.
[email protected] address is dead:
Generating server: DC5-EXCH01.marvell.com [email protected]
Remote Server returned '550 5.1.1 RESOLVER.ADR.RecipNotFound; not found'
Added some more CC addresses.
Yesterday it crashed again at the same place:
[2076301.849762] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [2076301.850021] PGD 0 P4D 0 [2076301.850109] Oops: 0002 [#1] SMP PTI [2076301.850219] CPU: 4 PID: 18992 Comm: kworker/u16:1 Not tainted 4.19.0-8-amd64 #1 Debian 4.19.98-1 [2076301.850478] Hardware name: Dell Inc. PowerEdge 2950/0JR815, BIOS 2.7.0 10/30/2010 [2076301.850720] Workqueue: scsi_tmf_4 scmd_eh_abort_handler [scsi_mod] [2076301.850936] RIP: 0010:qla24xx_async_abort_cmd+0x1b/0x250 [qla2xxx] [2076301.851130] Code: e9 19 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 41 55 41 54 55 53 4c 8b 6f 28 4c 8b 7f 20 4c 8b 77 48 <f0> 41 ff 46 04 0f a e f0 41 f6 46 24 04 74 17 f0 41 ff 4e 04 bd 02 [2076301.851663] RSP: 0018:ffffa10f8bbe7da8 EFLAGS: 00010293 [2076301.851820] RAX: 0000000000000800 RBX: ffff8ab8ddd197a8 RCX: 0000000000000070 [2076301.852036] RDX: ffff8ab8de4a8388 RSI: 0000000000000001 RDI: ffff8ab8799b8c40 [2076301.852253] RBP: ffff8ab8dc96c480 R08: ffffffffc03b7860 R09: 0000000000000000 [2076301.852469] R10: 8080808080808080 R11: 0000000000000010 R12: ffff8ab8dea00000 [2076301.852686] R13: ffff8ab8ddd197a8 R14: 0000000000000000 R15: ffff8ab8dd632000 [2076301.852902] FS: 0000000000000000(0000) GS:ffff8ab8e7b00000(0000) knlGS:0000000000000000 [2076301.853142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [2076301.853320] CR2: 0000000000000004 CR3: 00000002203dc000 CR4: 00000000000006e0 [2076301.853536] Call Trace: [2076301.853632] qla24xx_abort_command+0x218/0x2d0 [qla2xxx] [2076301.853799] ? __switch_to_asm+0x41/0x70 [2076301.853924] ? __switch_to_asm+0x35/0x70 [2076301.854056] qla2xxx_eh_abort+0x117/0x310 [qla2xxx] [2076301.854209] scmd_eh_abort_handler+0x85/0x220 [scsi_mod] [2076301.854375] process_one_work+0x1a7/0x3a0 [2076301.854506] worker_thread+0x30/0x390 [2076301.854628] ? create_worker+0x1a0/0x1a0 [2076301.854753] kthread+0x112/0x130 [2076301.854859] ? kthread_bind+0x30/0x30 [2076301.854980] ret_from_fork+0x35/0x40 [2076301.855095] Modules linked in: loop ipmi_ssif radeon coretemp ttm drm_kms_helper drm kvm i2c_algo_bit i5000_edac iTCO_wdt sg iTCO_vendor_support irqbypass evdev i5k_
amb serio_raw joydev ipmi_si rng_core pcc_cpufreq dcdbas pcspkr ipmi_devintf acpi_cpufreq ipmi_msghandler button ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb crypt o_simd cryptd glue_helper aes_x86_64 dm_service_time dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua uas usb_storage hid_generic usbhid hid sr_mod cdrom ses enc losure sd_mod scsi_transport_sas ata_generic qla2xxx ata_piix nvme_fc ehci_pci nvme_fabrics libata uhci_hcd psmouse ehci_hcd nvme_core megaraid_sas usbcore scsi_transport _fc lpc_ich mfd_core scsi_mod usb_common bnx2
[2076301.856887] CR2: 0000000000000004 [2076301.856999] ---[ end trace e9083db8fb76e126 ]--- [2076301.857151] RIP: 0010:qla24xx_async_abort_cmd+0x1b/0x250 [qla2xxx] [2076301.857345] Code: e9 19 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 41 57 41 56 41 55 41 54 55 53 4c 8b 6f 28 4c 8b 7f 20 4c 8b 77 48 <f0> 41 ff 46 04 0f a e f0 41 f6 46 24 04 74 17 f0 41 ff 4e 04 bd 02 [2076301.857878] RSP: 0018:ffffa10f8bbe7da8 EFLAGS: 00010293 [2076301.858035] RAX: 0000000000000800 RBX: ffff8ab8ddd197a8 RCX: 0000000000000070 [2076301.858251] RDX: ffff8ab8de4a8388 RSI: 0000000000000001 RDI: ffff8ab8799b8c40 [2076301.858467] RBP: ffff8ab8dc96c480 R08: ffffffffc03b7860 R09: 0000000000000000 [2076301.869384] R10: 8080808080808080 R11: 0000000000000010 R12: ffff8ab8dea00000 [2076301.880412] R13: ffff8ab8ddd197a8 R14: 0000000000000000 R15: ffff8ab8dd632000 [2076301.891483] FS: 0000000000000000(0000) GS:ffff8ab8e7b00000(0000) knlGS:0000000000000000 [2076301.902490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [2076301.913344] CR2: 0000000000000004 CR3: 00000002203dc000 CR4: 00000000000006e0 [2077225.259348] mysqld[2155]: segfault at 0 ip 000056409366ad93 sp 00007fa049514450 error 6 in mysqld[564092eb2000+805000] [2077225.270564] Code: c7 45 00 00 00 00 00 8b 7d cc 4c 89 e2 4c 89 f6 e8 62 81 84 ff 49 89 c7 49 39 c4 0f 84 f6 00 00 00 e8 e1 1c 00 00 41 8b 4d 00 <89> 08 85 c9 74 37 4
9 83 ff ff 0f 84 9d 00 00 00 f6 c3 06 75 28 4d
Signature · 1 line
Ondrej Zary