Re: [Fwd: Re: libata , Silicon Image 3124]

Fwd: Re: libata , Silicon Image 3124 Kasimir Müller 2 participants 4 messages Jan 6, 2008 – Jan 10, 2008 1 message not in the archive libata

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Kasimir Müller

author

2008-01-06 10:55 UTC #1 lore raw

Hi Tejun,

Old communication appended below.

I wish you a Happy Xmas and a successful New Year.

I spent some time during Christmas to further investigate the problem. I bought a new 500GB disk and put all data on this disk.
This is also contineously watched by nagios and cacti
Then
1.) All 5 disks in the external case connected via Portmapper and sil24 card have excellent health-status with smartd.
2.) I get no(!!!!) errors at all if I use the disks as single drives or with lvm. I verified this by copying large amounts of data (100-200GB)
with rsync , cp-av and running bonnie++ single and simultaneously
to various combinations of drives.
3.) I get the errors as soon as I use raid. Same errors with raid0 (2 disks), 1 (2 disks), 5 (3 disks) in any combination of the drives 4.) The errors appear usually first during mkfs (same with ext3 and reiserfs) and than
after writing about 10-50 GB to the raid, and repeat then at 5 to
10 minute intervals according the disk activity.
5.) I used Kernel 2.6.23.1 with Your latest patch: same result 6.) I used kernel 2.6.24 patch rc-6 : same result
7.) during the tests I marked all files with md5-sums: No data corruption (!!!), so maybe I can live with it.

linux:/var/log # cat /proc/interrupts
CPU0

  0:   12457604    XT-PIC-XT        timer
  1:          8    XT-PIC-XT        i8042
  2:          0    XT-PIC-XT        cascade
  4:     186738    XT-PIC-XT        serial
  5:   13328470    XT-PIC-XT        sata_via, ehci_hcd:usb5, VIA8237,
fcpci, eth0
  6:          5    XT-PIC-XT        floppy
  8:          0    XT-PIC-XT        rtc
  9:          0    XT-PIC-XT        acpi
 10:    8019753    XT-PIC-XT        sata_sil24
 11:        114    XT-PIC-XT        uhci_hcd:usb1, uhci_hcd:usb2
 14:     468588    XT-PIC-XT        libata
 15:          0    XT-PIC-XT        libata, uhci_hcd:usb3, uhci_hcd:usb4
NMI:          0
LOC:   12457724
ERR:          1

complete log of kernel errors from today appended as gz:

In the opensuse mailing-list some people reported related errors.
http://lists.opensuse.org/opensuse-de/2007-12/msg00939.html

Can You make someting out of this ? Do You need any more information (please detail, because I am not a linux-guru)

Yours sincerely

Kasimir Mueller

Quoted 2 lines

[email protected] didn't work. Forwarding to the original address.

it is [email protected]

Quoted 100 lines

------------------------------------------------------------------------
Betreff:
Re: libata , Silicon Image 3124
Von:
Tejun Heo <[email protected]>
Datum:
Wed, 14 Nov 2007 11:10:57 +0900
An:
Kasimir Mueller <[email protected]>
An:
Kasimir Mueller <[email protected]>
CC:
IDE/ATA development list <[email protected]>
Return-Path:
<[email protected]>
X-Original-To:
[email protected]
Delivered-To:
[email protected]
Received:
from [127.0.0.2] (htj.dyndns.org [127.0.0.2]) by htj.dyndns.org (Postfix) with ESMTP id 50E1813A81CF; Wed, 14 Nov 2007 11:10:57 +0900 (KST)
Nachricht-ID:
<[email protected]>
User-Agent:
Thunderbird 2.0.0.6 (X11/20070801)
MIME-Version:
1.0
Referenzen:
<[email protected]> <[email protected]> <[email protected]>
In-Reply-To:
<[email protected]>
X-Enigmail-Version:
0.95.3
Content-Type:
text/plain; charset=ISO-8859-15
Content-Transfer-Encoding:
8bit
[linux-ide added back. please don't drop cc list]
Quoted 11 lines · Kasimir Mueller
Quoted 9 lines · Tejun Heo
Quoted 6 lines · Kasimir Müller
Nov 12 19:28:42 linux kernel: ata6.02: cmd
ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0 cdb 0x0 data 0
Nov 12 19:28:42 linux kernel:          res
40/00:00:09:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Quoted 7 lines
Quoted 5 lines
That's flush timing out, which isn't good.
1. Are the errors localized to ata6.02?
I think it's ata6.01
The above message says 6.02 tho. Anyways, if the errors are localized to one drive, please swap that drive with another drive. Do the errors follow the drive or stay with the slot?
Quoted 5 lines
Quoted 3 lines
2. Is FLUSH always involved (ie. is cmd always ea)?
yes
Hmmm... okay. That sounds like a dying drive to me. What does 'smartctl -a /dev/sdX' say?
Quoted 5 lines
Quoted 3 lines
3. If you disable NCQ, what do errors look like?
I put all disk-drives in the blacklist , without a change
So, NCQ isn't the problem.
ATM, it sounds like a dying drive to me. Please double check the errors are localized to one drive, check smartctl log and try to verify as written above.
Thanks.

Tejun Heo

2008-01-08 03:47 UTC #2 lore raw

Quoted 18 lines · Kasimir Müller

Hi Tejun,
Old communication appended below.
I wish you a Happy Xmas and a successful New Year.
I spent some time during Christmas to further investigate the problem. I bought a new 500GB disk and put all data on this disk.
This is also contineously watched by nagios and cacti
Then
1.) All 5 disks in the external case connected via Portmapper and sil24 card have excellent health-status with smartd.
2.) I get no(!!!!) errors at all if I use the disks as single drives or with lvm. I verified this by copying large amounts of data (100-200GB)
with rsync , cp-av and running bonnie++ single and simultaneously
to various combinations of drives.
3.) I get the errors as soon as I use raid. Same errors with raid0 (2 disks), 1 (2 disks), 5 (3 disks) in any combination of the drives

Quoted 8 lines

4.) The errors appear usually first during mkfs (same with ext3 and reiserfs) and than
after writing about 10-50 GB to the raid, and repeat then at 5 to
10 minute intervals according the disk activity.
5.) I used Kernel 2.6.23.1 with Your latest patch: same result 6.) I used kernel 2.6.24 patch rc-6 : same result
7.) during the tests I marked all files with md5-sums: No data corruption (!!!), so maybe I can live with it.

Please apply the attached patch on top of 2.6.24-rc6 and report whether anything changes.

Thanks.

Signature · 1 line

tejun

drivers/ata/libata-eh.c +15 -5

index f0124a8..74269ed 100644
@@ -1733,11 +1733,15 @@ static void ata_eh_link_autopsy(struct ata_link *link)
 		ehc->i.action &= ~ATA_EH_PERDEV_MASK;
 	}
 
-	/* consider speeding down */
+	/* propagate timeout to host link */
+	if ((all_err_mask & AC_ERR_TIMEOUT) && !ata_is_host_link(link))
+		ap->link.eh_context.i.err_mask |= AC_ERR_TIMEOUT;
+
+	/* record error and consider speeding down */
 	dev = ehc->i.dev;
-	if (!dev && ata_link_max_devices(link) == 1 &&
-	    ata_dev_enabled(link->device))
-		dev = link->device;
+	if (!dev && ((ata_link_max_devices(link) == 1 &&
+		      ata_dev_enabled(link->device))))
+	    dev = link->device;
 
 	if (dev)
 		ehc->i.action |= ata_eh_speed_down(dev, is_io, all_err_mask);

@@ -1759,8 +1763,14 @@ void ata_eh_autopsy(struct ata_port *ap)
 {
 	struct ata_link *link;
 
-	__ata_port_for_each_link(link, ap)
+	ata_port_for_each_link(link, ap)
 		ata_eh_link_autopsy(link);
+
+	/* Autopsy of fanout ports can affect host link autopsy.
+	 * Perform host link autopsy last.
+	 */
+	if (ap->nr_pmp_links)
+		ata_eh_link_autopsy(&ap->link);
 }
 
 /**

Kasimir Müller

author

2008-01-09 17:32 UTC #3 lore raw

Quoted 30 lines · Tejun Heo

Quoted 18 lines · Kasimir Müller
Hi Tejun,
Old communication appended below.
I wish you a Happy Xmas and a successful New Year.
I spent some time during Christmas to further investigate the problem. I bought a new 500GB disk and put all data on this disk.
This is also contineously watched by nagios and cacti
Then
1.) All 5 disks in the external case connected via Portmapper and sil24 card have excellent health-status with smartd.
2.) I get no(!!!!) errors at all if I use the disks as single drives or with lvm. I verified this by copying large amounts of data (100-200GB)
with rsync , cp-av and running bonnie++ single and simultaneously
to various combinations of drives.
3.) I get the errors as soon as I use raid. Same errors with raid0 (2 disks), 1 (2 disks), 5 (3 disks) in any combination of the drives
Quoted 9 lines
4.) The errors appear usually first during mkfs (same with ext3 and reiserfs) and than
after writing about 10-50 GB to the raid, and repeat then at 5 to
10 minute intervals according the disk activity.
5.) I used Kernel 2.6.23.1 with Your latest patch: same result 6.) I used kernel 2.6.24 patch rc-6 : same result
7.) during the tests I marked all files with md5-sums: No data corruption (!!!), so maybe I can live with it.

Quoted 6 lines

Please apply the attached patch on top of 2.6.24-rc6 and report whether anything changes.
Thanks.

Hi Tejun,
thanks very much again for your time and readiness to help.

I applied the patch as you proposed without problem.

I attach the kernel logs of the past few days. As far as I can see, with the patched kernel errors appear only under heavy load .

If you want, I could provide a (temporary) ssh-login to my machine. Maybe it would make it easier for you to debug the problem. If you want to use it, I need your public ssh-certificate and your login-name and I would send the login-details to you.

Thanks again,
yours Kasimir.

Tejun Heo

2008-01-10 04:08 UTC #4 lore raw

Hello,

Quoted 11 lines · Kasimir Müller

thanks very much again for your time and readiness to help.
I applied the patch as you proposed without problem.
I attach the kernel logs of the past few days. As far as I can see, with the patched kernel errors appear only under heavy load .
If you want, I could provide a (temporary) ssh-login to my machine. Maybe it would make it easier for you to debug the problem. If you want to use it, I need your public ssh-certificate and your login-name and I would send the login-details to you.

Your log is pretty interesting. Lots of timeouts are on FLUSH and previously other non-data commands like CHECK POWER or SMART, which is very unusual. I wonder what's going on. Yeap, I wanna log in and take a look. I'll send you my pubkey in a private mail.

Thanks.

Signature · 1 line

tejun