Kernel panic w/ message request_threaded_irq -> qla2x00_request_irqs -> qla2x00_probe_one -> mod_timer
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 269 lines · TomK
Quoted 241 lines · TomK
Quoted 240 lines · Laurence Oberman
Quoted 210 lines · TomK
Quoted 113 lines · TomK
Quoted 101 lines · Laurence Oberman
Quoted 100 lines · Bart Van Assche
Quoted 73 lines · Laurence Oberman
Quoted 39 lines · TomK
Hey All,
I'm getting a kernel panic on an Gigabyte GA-890XA-UD3 motherboard
that
I've got a QLE2464 card in as a target (FC). The kernel has
been
crashing / panicking in the last 1-2 months about once a week. Before
that, it was rock solid for 4-5 years. I've upgraded to kernel
4.18.19
but that hasn't made much of a difference. Since the message
includesqla2x00_request_irqsI thought I would try here first.Tried to get more info on this but:
1) Keyboard doesn't work and locks up when the panic occurs. No
USB
ports work. Tried the PS/2 port but nothing.2) Unable to capture a kdump. Can't get to the kdump vmcore due
to
1).The two screenshots is pretty much all I can capture. Tried
things
like
clocksource=rtc in the kernel parms and disabling hpet1 but
apparently I
haven't disabled it everywhere since it still shows up.Quoted 6 lines
Wondering if anyone recognizes these messages or has any idea
what
could
be the issue here? Even a hint would be appreciated.Hello Tom
I have had similar issues and reported them to Himanshu@Cavium
I have kept all my target servers at kernel 4.5 as it been the only
version that has always been stable.
If your motherboard has an NMI (virtual or physical) set all of
these
in/etc/sysctl.conf
Run sysctl -a;dracut -f and rebootkernel.nmi_watchdog = 1
kernel.panic_on_io_nmi = 1
kernel.panic_on_unrecovered_nmi =
kernel.unknown_nmi_panic = 1When the issue shows up press the virtual/physical NMI
This is with the assumption that generic kdump is properly setup
and
dmesg | grep crash shows memory resrved by the crashkernel and that
you
have tested kdump manually.Quoted 4 lines
Other options are use a USB serial port to capture the full log if
you
cannot get kdump to work.That approach may provide further evidence about kernel bugs but it
is not
guaranteed that that approach will lead to a solution. It would help
if
either or both of you could do the following on a test system:
* Check out branch qla2xxx-for-next of my kernel repo on github
(https://github.com/bvanassche/linux/tree/qla2xxx-for-next).
* Enable lockdep and KASAN in the kernel config (CONFIG_PROVE_LOCKING
and
CONFIG_KASAN).
* Build and install that kernel.
* Run your favorite workload.Please note that the qla2xxx-for-next branch is based on the v5.1-rc1
kernel
and hence should not be installed on any production system.Quoted 9 lines
Quoted 3 lines
Thanks,
Bart.
Hello Bart
OK, I will get to this by Thursday, wont be able to change the targetserver kernel until then.
Regards
LaurenceSame. I'll try this out closer to the weekend.
Quoted 10 lines
Not an NMI motherboard. This is a 9-10 year old AMD board meant as a desktop or home server.
I'll have to read more about the USB Serial port to capture further info. That's interesting.
For the time being, I've disabled HPET in BIOS. ( Appears the kernel boot parameter method wasn't enough. )
Hey Guy's,
Did some of what you suggested, including the USB serial setup:
1) One of DB9 RS232 Serial Null Modem Cable F/F
2) Two of USB to RS232 Serial Port DB9 9 Pin Male however, when the kernel came down it took the USB support with it and so minicom went offline:
CTRL-A Z for help |115200 8N1 | NOR | Minicom 2.6.2 | VT102 | Offline
But I did enable full logging for the QLA module:
echo0x7fffffff>/sys/module/qla2xxx/parameters/ql2xextended_error_loggingDid all that, minus the Kernel v5.1-rc1 implementation, and this is what was picked up from the minicom USB to Serial capture before things went south:
1235905 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-Quoted 1 line
bufflen=512, cmd->sg_cnt=1, cmd-
dma_data_directi
on=1
se_cmd[0000
00009c9ea758]
qp
0
1235906 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-Quoted 1 line
bufflen=4096, cmd->sg_cnt=0,
cmd-
Quoted 1 line
dma_data_direction=2
se_cmd[000
0000096ae11b7]
q
p 0
1235907 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-Quoted 1 line
bufflen=20480, cmd->sg_cnt=0,
cmd
->dma_data_direc
tion=2
se_cmd[000000001738f793]
qp 0
1235908 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-Quoted 1 line
bufflen=20480, cmd->sg_cnt=0,
cmd
->dma_data_direc
tion=2
se_cmd[00000000e8160a90]
qp 0
1235909 ^MDetected MISCOMPARE for addr: 0000000033045258 buf:00000000f9849912
1235910 ^MTarget/fileio: Send MISCOMPARE check condition and sense 1235911 ^Mqla2xxx [0000:04:00.0]-e818: is_send_status=1, cmd-Quoted 1 line
bufflen=512, cmd->sg_cnt=0, cmd-
dma_data_directi
on=2
se_cmd[0000
0000363ae214]
qp
0
1235912 ^Mqla2xxx [0000:04:00.0]-e817: SkippingEXPLICIT_CONFORMandCTIO7_FLAGS_CONFORM_REQ
fo
r FCP READ w/
no
n GOOD status
1235913 ^Mqla2xxx [0000:04:00.0]-e874:2:qlt_free_cmd:
se_cmd[000000001db805fd]ox_id00c8
1235914 ^Mqla2xxx [0000:04:00.0]-e872:2:qlt_24xx_atio_pkt_all_vps:qla_target(0): type 6ox_id
00db
1235915 ^Mqla2xxx [0000:04:00.0]-e872:2:qlt_24xx_atio_pkt_all_vps:qla_target(0): type 6ox_id
00dc
1235916 ^Mqla2xxx [0000:04:00.0]-e874:2:qlt_free_cmd:
se_cmd[00000000f67a701f]ox_id00c9
1235917 ^Mqla2xxx [0000:04:00.0]-e872:2:qlt_24xx_atio_pkt_all_vps:qla_target(0): type 6ox_id
00dd
1235918 ^Mqla2xxx [0000:04:00.0]-e872:2:qlt_24xx_atio_pkt_all_vps:qla_target(0): type 6ox_id
00deQuoted 5 lines
On an earlier crash, captured the attached image. This time there was nothing on the monitor and the keyboard didn't refresh it. No signal.
When looking this up, closest I could see online is the following:https://target-devel.vger.kernel.narkive.com/XiM5Csx8/luns-become-unavailable-with-current-git-head
Quoted 21 lines
They too run ESXi .
To read the file I used the AnsiEsc plugin for VIM:
https://www.vim.org/scripts/script.php?script_id=302
This started to occur once had a VMware based MySQL and PostgreSQL cluster configured. Takes a few days for the issue to occur so from that perspective, appears to be memory related.
Firmware that I'm using is:
supported_classes = "Class 3"
supported_speeds = "1 Gbit, 2 Gbit, 4 Gbit" symbolic_name = "QLE2464 FW:v8.04.00 DVR:v10.00.00.05-k" Targetcli, rtslib and configshell versions I'm using are:# rpm -aq|grep -Ei "targetcli|rtslib|configshell" python-rtslib-3.0.pre4.9~g6fd0bbf-1.el6.noarch python-configshell-1.1.fb4-1.el6.noarch targetcli-3.0.pre4.5~ga125182-1.el6.noarchSignature · 2 lines
Thx, TK.Quoted 10 lines
Quoted 4 lines
I missed this email, Been buried in customer cases. I also need to still run some tests.
Sorry, reading nowNo worries. Would be very interested to see what you find.
In the meantime later tonight I'll be trying to 1) find more recent firmware for the card, 2) try to use the newer kernel v5.1-rc1 outlined above 3) try the 4.5 kernel later on the weekend.
Trying with this F/W and driver:
symbolic_name = "QLE2464 FW:v8.06.02 DVR:v10.00.00.07-k-debug"
[root@mbpc-pc 05-03-2019]# strings /lib/firmware/ql2400_fw.bin|grep -Ei copyright
COPYRIGHT 2016 QLOGIC CORPORATION ISP24xx Firmware Version 8.06.02 $
[root@mbpc-pc 05-03-2019]#The latest 'boot' firmware I can find is the following:
Quoted 9 lines
[root@mbpc-pc 05-03-2019]# strings Q24A7232.BIN |grep -Ei copyright Copyright (C) QLogic Corporation 1993-2015. All rights reserved. Copyright (C) QLogic Corporation 1993-2015. All rights reserved.
COPYRIGHT 2015 QLOGIC CORPORATION ISP24xx Firmware Version 8.01.02 $
You have new mail in/var/spool/mail/root
[root@mbpc-pc 05-03-2019]#This is what shows up as loaded during system boot.
So resolved it a few months back. What I did is simply to upgrade the memory on the storage server from 4GB to 8GB. Then later to 16GB. Since the time I upgrade the memory, the kernel panic didn't show up and everything's been stable.
Signature · 2 lines
Thx, TK.