Skip to contentSkip to conversation

Kernel panic w/ message request_threaded_irq -> qla2x00_request_irqs -> qla2x00_probe_one -> mod_timer

DISCUSSION TomK 1 participant 1 message Sep 29, 2019 9 messages not in the archive SCSI

lore
Share & tools
lore

TomK

author
#9 lore raw
In reply to a message not in the archive
Quoted 269 lines · TomK
Quoted 241 lines · TomK
Quoted 240 lines · Laurence Oberman
Quoted 210 lines · TomK
Quoted 113 lines · TomK
Quoted 101 lines · Laurence Oberman
Quoted 100 lines · Bart Van Assche
Quoted 73 lines · Laurence Oberman
Quoted 39 lines · TomK

Hey All,

I'm getting a kernel panic on an Gigabyte GA-890XA-UD3 motherboard
that
I've got a QLE2464 card in as a target (FC).  The kernel has
been
crashing / panicking in the last 1-2 months about once a week.  Before
that, it was rock solid for 4-5 years.  I've upgraded to kernel
4.18.19
but that hasn't made much of a difference.  Since the message
includes
qla2x00_request_irqs I thought I would try here first.

Tried to get more info on this but:

1) Keyboard doesn't work and locks up when the panic occurs.  No
USB
ports work.  Tried the PS/2 port but nothing.

2) Unable to capture a kdump.  Can't get to the kdump vmcore due
to
1).

The two screenshots is pretty much all I can capture. Tried
things
like
clocksource=rtc in the kernel parms and disabling hpet1 but
apparently I
haven't disabled it everywhere since it still shows up.

Quoted 6 lines

Wondering if anyone recognizes these messages or has any idea
what
could
be the issue here?  Even a hint would be appreciated.

Hello Tom
I have had similar issues and reported them to Himanshu@Cavium
I have kept all my target servers at kernel 4.5 as it been the only
version that has always been stable.
If your motherboard has an NMI (virtual or physical) set all of
these
in /etc/sysctl.conf
Run sysctl -a;dracut -f and reboot

kernel.nmi_watchdog = 1
kernel.panic_on_io_nmi = 1
kernel.panic_on_unrecovered_nmi =
kernel.unknown_nmi_panic = 1

When the issue shows up press the virtual/physical NMI

This is with the assumption that generic kdump is properly setup
and
dmesg | grep crash shows memory resrved by the crashkernel and that
you
have tested kdump manually.

Quoted 4 lines

Other options are use a USB serial port to capture the full log if
you
cannot get kdump to work.

That approach may provide further evidence about kernel bugs but it
is not
guaranteed that that approach will lead to a solution. It would help
if
either or both of you could do the following on a test system:
* Check out branch qla2xxx-for-next of my kernel repo on github
    (https://github.com/bvanassche/linux/tree/qla2xxx-for-next).
* Enable lockdep and KASAN in the kernel config (CONFIG_PROVE_LOCKING
and
    CONFIG_KASAN).
* Build and install that kernel.
* Run your favorite workload.

Please note that the qla2xxx-for-next branch is based on the v5.1-rc1
kernel
and hence should not be installed on any production system.

Quoted 9 lines
Quoted 3 lines

Thanks,

Bart.

Hello Bart
OK, I will get to this by Thursday, wont be able to change the targetserver kernel until then.
Regards
Laurence

Same.  I'll try this out closer to the weekend.

Quoted 10 lines

Not an NMI motherboard.  This is a 9-10 year old AMD board meant as a desktop or home server.

I'll have to read more about the USB Serial port to capture further info.  That's interesting.

For the time being, I've disabled HPET in BIOS.  ( Appears the kernel boot parameter method wasn't enough. )

Hey Guy's,
Did some of what you suggested, including the USB serial setup:
1) One of DB9 RS232 Serial Null Modem Cable F/F
2) Two of USB to RS232 Serial Port DB9 9 Pin Male however, when the kernel came down it took the USB support with it and so minicom went offline:
  CTRL-A Z for help |115200 8N1 | NOR | Minicom 2.6.2  | VT102 | Offline
But I did enable full logging for the QLA module:
echo 0x7fffffff > /sys/module/qla2xxx/parameters/ql2xextended_error_logging Did all that, minus the Kernel v5.1-rc1 implementation, and this is what was picked up from the minicom USB to Serial capture before things went south:
1235905 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-

Quoted 1 line

bufflen=512, cmd->sg_cnt=1, cmd-

dma_data_directi
                              on=1
se_cmd[0000
                         00009c9ea758]
qp
                 0
1235906 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-

Quoted 1 line

bufflen=4096, cmd->sg_cnt=0,

cmd-

Quoted 1 line

dma_data_direct

                              ion=2
se_cmd[000
                        0000096ae11b7]
q
               p 0
1235907 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-

Quoted 1 line

bufflen=20480, cmd->sg_cnt=0,

cmd
->dma_data_direc
                              tion=2
se_cmd[00
0000001738f793]
                              qp 0
1235908 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-

Quoted 1 line

bufflen=20480, cmd->sg_cnt=0,

cmd
->dma_data_direc
                              tion=2
se_cmd[00
000000e8160a90]
                              qp 0
1235909 ^MDetected MISCOMPARE for addr: 0000000033045258 buf:
00000000f9849912
1235910 ^MTarget/fileio: Send MISCOMPARE check condition and sense 1235911 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-

Quoted 1 line

bufflen=512, cmd->sg_cnt=0, cmd-

dma_data_directi
                              on=2
se_cmd[0000
                         0000363ae214]
qp
                 0
1235912 ^Mqla2xxx [0000:04:00.0]-e817: Skipping EXPLICIT_CONFORM and CTIO7_FLAGS_CONFORM_REQ
fo
              r FCP READ w/
no
                n GOOD status
1235913 ^Mqla2xxx [0000:04:00.0]-e874:2: qlt_free_cmd:
se_cmd[000000001db805fd] ox_id 00c8
1235914 ^Mqla2xxx [0000:04:00.0]-e872:2: qlt_24xx_atio_pkt_all_vps:
qla_target(0): type 6
ox_id
                  00db
1235915 ^Mqla2xxx [0000:04:00.0]-e872:2: qlt_24xx_atio_pkt_all_vps:
qla_target(0): type 6
ox_id
                  00dc
1235916 ^Mqla2xxx [0000:04:00.0]-e874:2: qlt_free_cmd:
se_cmd[00000000f67a701f] ox_id 00c9
1235917 ^Mqla2xxx [0000:04:00.0]-e872:2: qlt_24xx_atio_pkt_all_vps:
qla_target(0): type 6
ox_id
                  00dd
1235918 ^Mqla2xxx [0000:04:00.0]-e872:2: qlt_24xx_atio_pkt_all_vps:
qla_target(0): type 6
ox_id
                  00de

Quoted 5 lines

On an earlier crash, captured the attached image.  This time there was nothing on the monitor and the keyboard didn't refresh it.  No signal.
When looking this up, closest I could see online is the following:

https://target-devel.vger.kernel.narkive.com/XiM5Csx8/luns-become-unavailable-with-current-git-head

Quoted 21 lines

They too run ESXi .
To read the file I used the AnsiEsc plugin for VIM:
https://www.vim.org/scripts/script.php?script_id=302
This started to occur once had a VMware based MySQL and PostgreSQL cluster configured.  Takes a few days for the issue to occur so from that perspective, appears to be memory related.
Firmware that I'm using is:
     supported_classes   = "Class 3"
     supported_speeds    = "1 Gbit, 2 Gbit, 4 Gbit" symbolic_name       = "QLE2464 FW:v8.04.00 DVR:v10.00.00.05-k" Targetcli, rtslib and configshell versions I'm using are:

# rpm -aq|grep -Ei "targetcli|rtslib|configshell"
python-rtslib-3.0.pre4.9~g6fd0bbf-1.el6.noarch
python-configshell-1.1.fb4-1.el6.noarch
targetcli-3.0.pre4.5~ga125182-1.el6.noarch
Signature · 2 lines
Thx,
TK.
Quoted 10 lines
Quoted 4 lines

I missed this email, Been buried in customer cases. I also need to still run some tests.
Sorry, reading now

No worries.  Would be very interested to see what you find.

In the meantime later tonight I'll be trying to 1) find more recent firmware for the card, 2) try to use the newer kernel v5.1-rc1 outlined above 3) try the 4.5 kernel later on the weekend.

Trying with this F/W and driver:

symbolic_name       = "QLE2464 FW:v8.06.02 DVR:v10.00.00.07-k-debug"

[root@mbpc-pc 05-03-2019]# strings /lib/firmware/ql2400_fw.bin|grep -Ei copyright
 COPYRIGHT 2016 QLOGIC CORPORATION   ISP24xx Firmware   Version 8.06.02  $
[root@mbpc-pc 05-03-2019]#

The latest 'boot' firmware I can find is the following:

Quoted 9 lines

[root@mbpc-pc 05-03-2019]# strings Q24A7232.BIN |grep -Ei copyright Copyright (C) QLogic Corporation 1993-2015. All rights reserved. Copyright (C) QLogic Corporation 1993-2015. All rights reserved.
 COPYRIGHT 2015 QLOGIC CORPORATION   ISP24xx Firmware   Version 8.01.02  $
You have new mail in /var/spool/mail/root
[root@mbpc-pc 05-03-2019]#

This is what shows up as loaded during system boot.

So resolved it a few months back. What I did is simply to upgrade the memory on the storage server from 4GB to 8GB. Then later to 16GB. Since the time I upgrade the memory, the kernel panic didn't show up and everything's been stable.

Signature · 2 lines
Thx,
TK.