New: Devices disappear; on bus reset machine hangs; on I/O machine hangs
http://bugzilla.kernel.org/show_bug.cgi?id=14579
Summary: Devices disappear; on bus reset machine hangs; on I/O
machine hangs
Product: IO/Storage
Version: 2.5
Kernel Version: Linux version 2.6.30-1-amd64 (Debian
2.6.30-5+techfaklenny1)
([email protected]) (gcc version 4.3.2
(Debian 4.3.2-1.1) ) #1 SMP Sat Aug 15 11:20:08 UTC
2009
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: blocking
Priority: P1
Component: SCSI
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: NoCreated an attachment (id=23729)
--> (http://bugzilla.kernel.org/attachment.cgi?id=23729)
lspci -vvv output
On our soon-to-be primary backup machine with an Adaptec 52445 (for the disk pool) and an LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (for the Tape Library), we see random failures:
On Tape-IO (using btape from bacula), a few minutes into the writes, the changer device and both tape devices disappear from the system. Rescanning the bus sometimes helps, but sometimes results in a hardy freeze.
On Tape-IO, a few minutes into the writes, the system freezes.
lspci -vvv, lsscsi and dmesg and /proc/modules are attached, this might also be related to #14577 as it's the same machine. Attached is also a screenshot of the oops we saw on the last crash that happened durign a btape run.
The system uses stock Debian/Lenny with a custom built 2.6.30.5 kernel.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.
[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs
http://bugzilla.kernel.org/show_bug.cgi?id=14579
--- Comment #1 from Sascha Frey <[email protected]> 2009-11-10 15:29:20 --- Created an attachment (id=23730)
--> (http://bugzilla.kernel.org/attachment.cgi?id=23730)
cat /proc/modules output
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.
[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=14579
--- Comment #2 from Sascha Frey <[email protected]> 2009-11-10 15:30:02 --- Created an attachment (id=23731)
--> (http://bugzilla.kernel.org/attachment.cgi?id=23731)
lsscsi output
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.
[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=14579
--- Comment #3 from Sascha Frey <[email protected]> 2009-11-10 15:30:46 --- Created an attachment (id=23732)
--> (http://bugzilla.kernel.org/attachment.cgi?id=23732)
screenshot of last oops
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.
[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=14579
--- Comment #4 from Sascha Frey <[email protected]> 2009-11-10 15:35:41 --- Created an attachment (id=23733)
--> (http://bugzilla.kernel.org/attachment.cgi?id=23733)
dmesg output
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.
[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=14579
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
Depends on| |14577
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.
[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=14579
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected] | |d.de
--- Comment #5 from [email protected] 2009-11-18 14:05:34 --- fyi
From: "Desai, Kashyap" <[email protected]>
To: "[email protected]" <[email protected]>,
"[email protected]" <[email protected]>
CC: Lukas Kolbe <[email protected]>
Date: Fri, 13 Nov 2009 17:29:43 +0530
Subject: RE: Bug 14579 - Devices disappear... and Bug 14577 - Data
corruption with Adaptec
Message-ID: <[email protected]>
References: <[email protected]>
<[email protected]>
In-Reply-To: <[email protected]>
Subject line is related to *Adaptec* and there are some places LSI related = issue is pointed out. Little confusing to me. Is it possible to rewrite wha= t is an issue related to LSI card?
Quoted 1 line
From dmesg log I can figure out 3.04.07 is mpt fusion driver version.
Please update LSI driver using latest upstream driver version 3.04.13. And = see what a result is.
- Kashyap
-----Original Message-----
From: [email protected] [mailto:[email protected]= .org] On Behalf Of Sascha Frey
Sent: Friday, November 13, 2009 4:28 AM
To: [email protected]
Cc: Lukas Kolbe
Subject: Re: Bug 14579 - Devices disappear... and Bug 14577 - Data corrupti= on with Adaptec
Hi,
Quoted 3 lines · Lukas Kolbe
we'd really appreciate any hints and help we can get for the following bugs:
http://bugzilla.kernel.org/show_bug.cgi?id=3D14579
We've done some further testing:
it's very hard to trigger this bug. Sometimes the machine freezes after a few minutes into tape access and sometimes it works days - or even weeks - without any problem.
The bug only appears during tape I/O (regardless of which tape program is
used: btape, dd or tar).
In most cases the tape write ends with an input/output error. After this error occurred, any access to the tape library robot (connected through the SAS interface of the first drive) fails:
# mtx unload 1 1 Unloading drive 1 into Storage Element 1...mtx: Request Sense: Long Report= =3Dyes mtx: Request Sense: Valid Residual=3Dno mtx: Request Sense: Error Code=3D70 (Current) mtx: Request Sense: Sense Key=3DIllegal Request mtx: Request Sense: FileMark=3Dno mtx: Request Sense: EOM=3Dno mtx: Request Sense: ILI=3Dno mtx: Request Sense: Additional Sense Code =3D 53 mtx: Request Sense: Additional Sense Qualifier =3D 01 mtx: Request Sense: BPV=3Dno mtx: Request Sense: Error in CDB=3Dno mtx: Request Sense: SKSV=3Dno MOVE MEDIUM from Element Address 257 to 4096 Failed
After resetting the scsi bus (echo "- - -" > /sys/class/scsi_host/host5/scan) the tape drives are revitalized, but the changer device disappears. Even after a cold restart of the whole library the device keeps missing.
Yet another problem: restting the SCSI bus of the LSI SAS HBA sometimes results in a hardy freeze (console stuck; no log messages).
Quoted 5 lines
[...]
I do not believe it's a hardware fault at the moment as the machine ran OK under Solaris for a few weeks (including successful btape runs).
The very same piece of hardware worked fine using Solaris 10 with heavy disk and tape I/O at the same time for two months.
We really prefer using Linux instead, but we're in pressure of time.
We appreciate any help resolving this bug!
Regards,
Sascha Frey
From: "Desai, Kashyap" <[email protected]>
To: "[email protected]" <[email protected]>
CC: "[email protected]" <[email protected]>
Date: Wed, 18 Nov 2009 10:24:38 +0530
Subject: RE: Bug 14579 - Devices disappear... and Bug 14577 - Data
corruption with Adaptec
Message-ID: <[email protected]>
References: <[email protected]>
<[email protected]> <[email protected]> <[email protected]>
In-Reply-To: <[email protected]>
Hello Lukas,
Quoted 13 lines
-----Original Message-----
From: Lukas Kolbe [mailto:[email protected]]
Sent: Tuesday, November 17, 2009 7:53 PM
To: Desai, Kashyap
Cc: [email protected]
Subject: Re: Bug 14579 - Devices disappear... and Bug 14577 - Data corruption with Adaptec
=20
Desai, Kashyap wrote:
=20Quoted 3 lines
Subject line is related to *Adaptec* and there are some places LSI related issue is pointed out. Little confusing to me. Is it possible to rewrite what is an issue related to LSI card?
Quoted 6 lines
=20
Sorry for that one. This system has an Adaptec Controller for its Storage array and an LSI controller for the tape library. Bug 14577 is about a possible data corruption on 2.6.32-rc6 that seems to be either a hardware error (currently trying to find that out) or a regression in 2.6.32-rc6, as 2.6.30 is very happy with its storage.
OK. In data corruption condition only LSI driver and controller are involve= d? I mean can I nullify Adaptec controller's roll in your test?
Quoted 10 lines
=20
Finally, the real problem here is Bug 14579 that is about the systems problems when using the tape library.
=20Quoted 2 lines
From dmesg log I can figure out 3.04.07 is mpt fusion driver version. Please update LSI driver using latest upstream driver version 3.04.13.
And see what a result is.
=20
Thanks for the pointer. Linus' current tree contains 3.04.12 - where can I find 3.04.13?
It is there in 2.6.32-rc5. Not sure in which exact rc version it is include= d, but I have 2.6.32-rc5 tree in my setup and for that kernel mptfusion ver= sion is 3.104.13
Quoted 5 lines
=20
Quoted 1 line
- Kashyap
=20
Kind regards,
Lukas Kolbe
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.
[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=14579
--- Comment #6 from [email protected] 2009-11-18 14:07:10 --- fyi
Quoted 1 line
From [email protected] Wed Nov 18 14:39:08 2009
Date: Wed, 18 Nov 2009 14:39:09 +0100
From: Lukas Kolbe <[email protected]>
To: "Desai, Kashyap" <[email protected]>
Cc: "[email protected]" <[email protected]>
Subject: Re: Bug 14579 - Devices disappear... and Bug 14577 - Data
corruption with Adaptec
Message-ID: <[email protected]>
References: <[email protected]> <[email protected]> <[email protected]> <[email protected]> <[email protected]>
In-Reply-To: <[email protected]>
Quoted 11 lines · Kashyap
Quoted 9 lines
Quoted 3 lines
Subject line is related to *Adaptec* and there are some places LSI related issue is pointed out. Little confusing to me. Is it possible to rewrite what is an issue related to LSI card?
Sorry for that one. This system has an Adaptec Controller for its Storage array and an LSI controller for the tape library. Bug 14577 is about a possible data corruption on 2.6.32-rc6 that seems to be either a hardware error (currently trying to find that out) or a regression in 2.6.32-rc6, as 2.6.30 is very happy with its storage.
OK. In data corruption condition only LSI driver and controller are involved? I mean can I nullify Adaptec controller's roll in your test?
No, it is the other way round. We have 24 1TB Seagate harddisks connected in a RAID 60 to the adaptec controller, and a Tandberg T80 with two IBM Ultrium-HH4 tape drives connected to the LSI controller.
The system is installed on an LVM volume within the RAID 60. The data corruption occurs when we try to boot 2.6.32-rc6, we get write errors and the boot process stops somewhere. So, it seems the data corruption is related _only_ with the Adaptec Controller, the RAID array or the harddisks.
Quoted 5 lines
Quoted 5 lines
Finally, the real problem here is Bug 14579 that is about the systems problems when using the tape library.
Quoted 2 lines
From dmesg log I can figure out 3.04.07 is mpt fusion driver version. Please update LSI driver using latest upstream driver version 3.04.13.
Quoted 8 lines
Quoted 4 lines
And see what a result is.
Thanks for the pointer. Linus' current tree contains 3.04.12 - where can I find 3.04.13?
It is there in 2.6.32-rc5. Not sure in which exact rc version it is included, but I have 2.6.32-rc5 tree in my setup and for that kernel mptfusion version is 3.104.13
Okay, I grep'ed for 3.04 in the source and only got one reference to the older version number. But there lies the problem: Unless we can fix the Adaptec-Bug first (or confirm it is a hardware issue), we can't boot 2.6.32-rc on that machine to test the new LSI driver version. Is it easily possible to backport/include the mptfusion in 2.6.30?
Thanks for the help and kind regards,
Signature · 6 lines
Lukas Kolbe -- Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.
[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=14579
Bug 14579 depends on bug 14577, which changed state.
Bug 14577 Summary: Data Corruption with Adaptec 52445, Firmware 5.2-0 (17380)
http://bugzilla.kernel.org/show_bug.cgi?id=14577
What |Old Value |New Value
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |CODE_FIXSignature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.
[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=14579
Alan <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected] Kernel Version|Linux version |2.6.30-1-amd64 Debian |2.6.30-1-amd64 (Debian | |2.6.30-5+techfaklenny1) | |([email protected] | |d.DE) (gcc version 4.3.2 | |(Debian 4.3.2-1.1) ) #1 SMP | |Sat Aug 15 11:20:08 UTC | |2009 |
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.
[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=14579
--- Comment #7 from [email protected] 2010-02-10 10:10:46 --- As it turned out this machine was faulty (all 24 disks began dematerializing under our feet) so we replaced it. We do now face similar issues, though I'll open a new bug for that one to have a clear separation. Whoever has the permission to close this bug might do so, please.
Kind regards,
Lukas
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.