libata interface fatal error
Hi there,
seems I've always subscribed to SATA problems. :-)
We installed Debian Etch with the pre-compiled kernel, but when doing heavy SATA data transfer, the drives seem to make trouble. Even with the latest kernel, 2.6.21.2, we receive:
===
ata3.00: exception Emask 0x10 SAct 0x1 SErr 0x400100 action 0x2 frozen ata3.00: (irq_stat 0x08000000, interface fatal error)
ata3.00: cmd 61/80:00:00:91:91/00:00:1d:00:00/40 tag 0 cdb 0x0 data 65536 out
res 40/00:04:00:91:91/00:00:1d:00:00/40 Emask 0x10 (ATA bus error)
ata3: soft resetting port
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: configured for UDMA/133
ata3: EH complete
SCSI device sdc: 625142448 512-byte hdwr sectors (320073 MB)
sdc: Write Protect is off
sdc: Mode Sense: 00 3a 00 00
SCSI device sdc: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
===
MD5 sums of copied files are right and we experience no other problems. Is this a driver bug? If so, can I be of any help in debugging it?
lspci gives:
===
00:00.0 Host bridge: Intel Corporation P965/G965 Memory Controller Hub (rev 02) 00:01.0 PCI bridge: Intel Corporation P965/G965 PCI Express Root Port (rev 02) 00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #4 (rev 02) 00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #5 (rev 02) 00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI #2 (rev 02) 00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 02) 00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 02) 00:1c.4 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 5 (rev 02) 00:1c.5 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 6 (rev 02) 00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #1 (rev 02) 00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #2 (rev 02) 00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #3 (rev 02) 00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI #1 (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev f2)
00:1f.0 ISA bridge: Intel Corporation 82801HB/HR (ICH8/R) LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 82801HB (ICH8) SATA AHCI Controller (rev 02) 00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 02) 01:00.0 VGA compatible controller: nVidia Corporation Unknown device 016a (rev a1) 03:00.0 Ethernet controller: Marvell Technology Group Ltd. Unknown device 4364 (rev 12) 04:00.0 SATA controller: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller (rev 02) 04:00.1 IDE interface: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller (rev 02) 05:01.0 Ethernet controller: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 0c) 05:06.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller (PHY/Link)
===
Thanks
Florian
Hello,
Quoted 10 lines · Florian Effenberger
We installed Debian Etch with the pre-compiled kernel, but when doing heavy SATA data transfer, the drives seem to make trouble. Even with the latest kernel, 2.6.21.2, we receive:
===
ata3.00: exception Emask0x10SAct0x1SErr0x400100action0x2frozen ata3.00: (irq_stat0x08000000, interface fatal error) ata3.00: cmd 61/80:00:00:91:91/00:00:1d:00:00/40 tag 0 cdb0x0data 65536 outres 40/00:04:00:91:91/00:00:1d:00:00/40 Emask 0x10 (ATA bus error)
Quoted 10 lines
ata3: soft resetting port
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata3.00: configured for UDMA/133
ata3: EH complete
SCSI device sdc: 625142448 512-byte hdwr sectors (320073 MB)
sdc: Write Protect is off
sdc: Mode Sense: 00 3a 00 00
SCSI device sdc: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
===
Looks like a genuine transmission/interface error to me. How often does this occur? Please try to connect the drive to another port using and possibly different power lane. Also, testing with another drive is a good way to track down where the problem is.
Quoted 2 lines
MD5 sums of copied files are right and we experience no other problems. Is this a driver bug? If so, can I be of any help in debugging it?
Yeah, libata EH is working properly so there shouldn't be any problem other than the error messages and a bit slower transfer speed.
Signature · 1 line
tejun
Hi,
thanks for the fast reply!
Quoted 4 lines
Looks like a genuine transmission/interface error to me. How often does this occur? Please try to connect the drive to another port using and possibly different power lane. Also, testing with another drive is a good way to track down where the problem is.
it occurs as soon as the drive is being used heavily (load of about 2,x on the machine when running our test scripts). About 15 times in 2 or 3 hours. Will try to change port, power supply and drive.
Quoted 2 lines
Yeah, libata EH is working properly so there shouldn't be any problem other than the error messages and a bit slower transfer speed.
So, even if the errors are still there, there is nothing real to worry about for me?
There are now new errors with hard errors, is this still ok?
===
ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x2 frozen ata4.00: cmd 60/80:00:00:09:97/00:00:0a:00:00/40 tag 0 cdb 0x0 data 65536 in
res 40/00:04:00:67:14/00:00:1c:00:00/40 Emask 0x4 (timeout)
ata4: soft resetting port
ata4: softreset failed (1st FIS failed)
ata4: softreset failed, retrying in 5 secs
ata4: hard resetting port
ata4: port is slow to respond, please be patient (Status 0x80)
ata4: port failed to respond (30 secs, Status 0x80)
ata4: COMRESET failed (device not ready)
ata4: hardreset failed, retrying in 5 secs
ata4: hard resetting port
ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata4.00: configured for UDMA/133
ata4: EH complete
SCSI device sdd: 625142448 512-byte hdwr sectors (320073 MB)
sdd: Write Protect is off
sdd: Mode Sense: 00 3a 00 00
SCSI device sdd: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
===
Florian
Quoted 9 lines · Florian Effenberger
Quoted 4 lines
Looks like a genuine transmission/interface error to me. How often does this occur? Please try to connect the drive to another port using and possibly different power lane. Also, testing with another drive is a good way to track down where the problem is.
it occurs as soon as the drive is being used heavily (load of about 2,x on the machine when running our test scripts). About 15 times in 2 or 3 hours. Will try to change port, power supply and drive.
Quoted 5 lines
Quoted 2 lines
Yeah, libata EH is working properly so there shouldn't be any problem other than the error messages and a bit slower transfer speed.
So, even if the errors are still there, there is nothing real to worry about for me?
Data integrity wise there should be no problem but your error rate is pretty high and eventually will make libata turn off NCQ and/or speed down PHY speed.
Quoted 7 lines
There are now new errors with hard errors, is this still ok?
===
ata4.00: exception Emask0x0SAct0x1SErr0x0action0x2frozen ata4.00: cmd 60/80:00:00:09:97/00:00:0a:00:00/40 tag 0 cdb0x0data 65536 inres 40/00:04:00:67:14/00:00:1c:00:00/40 Emask 0x4 (timeout)
Yeap, your data is safe. With timeouts, data transfer speed can be much lower tho. It definitely seems something is wrong with your hardware setup.
Signature · 1 line
tejun
Hi,
Quoted 3 lines
Data integrity wise there should be no problem but your error rate is pretty high and eventually will make libata turn off NCQ and/or speed down PHY speed.
switching ports is not easy. Both on-board SATA controllers are being used, and the error seems to occur on all ports.
Quoted 2 lines
Yeap, your data is safe. With timeouts, data transfer speed can be much lower tho. It definitely seems something is wrong with your hardware setup.
I will try to use another test disk. Right now we use different models of Western Digital "RAID edition".
Any other debug information I could gather?
Florian
Quoted 8 lines · Florian Effenberger
Hi,
Quoted 3 lines
Data integrity wise there should be no problem but your error rate is pretty high and eventually will make libata turn off NCQ and/or speed down PHY speed.
switching ports is not easy. Both on-board SATA controllers are being used, and the error seems to occur on all ports.
Hmmmm...
Quoted 8 lines
Quoted 3 lines
Yeap, your data is safe. With timeouts, data transfer speed can be much lower tho. It definitely seems something is wrong with your hardware setup.
I will try to use another test disk. Right now we use different models of Western Digital "RAID edition".
Any other debug information I could gather?
If you let the system run, libata will turn off NCQ and/or lower PHY speed to 1.5Gbps. Do errors disappear after that happens?
Signature · 1 line
tejun
We just disabled the RAID (Linux software RAID, no hardware RAID) and tested with one disk only, same results
What string should I grep the logs for when things are being lowered?
Quoted 2 lines
If you let the system run, libata will turn off NCQ and/or lower PHY speed to 1.5Gbps. Do errors disappear after that happens?
Quoted 6 lines · Florian Effenberger
Hi,
Quoted 3 lines
Data integrity wise there should be no problem but your error rate is pretty high and eventually will make libata turn off NCQ and/or speed down PHY speed.
Quoted 9 lines
switching ports is not easy. Both on-board SATA controllers are being used, and the error seems to occur on all ports.
Quoted 2 lines
Yeap, your data is safe. With timeouts, data transfer speed can be much lower tho. It definitely seems something is wrong with your hardware setup.
I will try to use another test disk. Right now we use different models of Western Digital "RAID edition".
iiuc, raid editions are designed to fail fast thus allowing an alternate drive to provide the data rather than having to wait thru multiple internal retries.
Could this just be a case of the drive functioning as designed?
Greg
Signature · 3 lines
Greg Freemyer The Norcross Group Forensics for the 21st Century
Quoted 9 lines · Greg Freemyer
Quoted 3 lines
I will try to use another test disk. Right now we use different models of Western Digital "RAID edition".
iiuc, raid editions are designed to fail fast thus allowing an alternate drive to provide the data rather than having to wait thru multiple internal retries.
Could this just be a case of the drive functioning as designed?
If that's the case, the drive should be aborting commands with ICRC bit set reporting unrecoverable media error (AC_ERR_DEV | AC_ERR_MEDIA in libata terms) but the errors are fatal interface errors and timeouts, both of which are indicative of transmission problems on ATA link.
Signature · 1 line
tejun
Hi,
Quoted 5 lines
iiuc, raid editions are designed to fail fast thus allowing an alternate drive to provide the data rather than having to wait thru multiple internal retries.
Could this just be a case of the drive functioning as designed?
to be honest, I don't know. :-)
Any jumper settings to change that, or any driver settings? Are you aware of something like that?
Florian