Skip to contentSkip to conversation

Scsi errors with Megaraid 300-8x

DISCUSSION Johan Groth 3 participants 11 messages Aug 22, 2006 – Aug 24, 2006

lore
Share & tools
lore
#0 lore raw

Hi,
ever since I upgraded my server from a dual Opteron 244 (mobo Tyan 2885) system to a dual dual-core Opteron 285 (mobo Tyan 2895) system, I'm getting read errors that freezes the system which leads to my disk based backup software stopped working (faubackup). I think it is faubackup that triggers the bug.

I get these errors in the log:

Aug 20 06:35:08 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:35:56 jaguar kernel: end_request: I/O error, dev sda, sector 616924530
Aug 20 06:36:03 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:03 jaguar kernel: end_request: I/O error, dev sda, sector 616924538
Aug 20 06:36:03 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:03 jaguar kernel: end_request: I/O error, dev sda, sector 616924546
Aug 20 06:36:03 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:03 jaguar kernel: end_request: I/O error, dev sda, sector 616924554
Aug 20 06:36:07 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:07 jaguar kernel: end_request: I/O error, dev sda, sector 616924562
Aug 20 06:36:07 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:07 jaguar kernel: end_request: I/O error, dev sda, sector 616924570
Aug 20 06:36:07 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:07 jaguar kernel: end_request: I/O error, dev sda, sector 616924578
Aug 20 06:36:07 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:07 jaguar kernel: end_request: I/O error, dev sda, sector 616924538

The last sector is repeated until I reboot the machine. The only difference I've made to the raid configuration is that sdc is now 2x250 MB instead of 4x120MB, but that array is the target not the source (sda). The raid HW is an LSI Megaraid 300-8x with the following configuration:

Host: scsi0 Channel: 01 Id: 00 Lun: 00
Vendor: MegaRAID Model: LD 0 RAID0 312G Rev: 814D
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 01 Id: 01 Lun: 00
Vendor: MegaRAID Model: LD 1 RAID0 312G Rev: 814D
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 01 Id: 02 Lun: 00
Vendor: MegaRAID Model: LD 2 RAID0 474G Rev: 814D
Type: Direct-Access ANSI SCSI revision: 02

I'm running debian sid stock kernel 2.6.17.

Other hw changes that may, may not affect the kernel:
CPU: dual core, hence the kernel sees 4 cpus, before 2. Removed a sound card, a CS46xx card and am using internal sound instead. Installed a DVB-T Freeview card (digital TV).
More RAM, 2GB instead 1.

Should mention that I used to use 2.6.16 on the old system, but I've checked in the kernel tree and no changes has been made to the megaraid drivers since April, I think.

Can anyone help me?
Please, CC me as I'm not subscribed.

Regards,
Johan

#1 lore raw
Quoted 8 lines · Johan Groth

Hi,
ever since I upgraded my server from a dual Opteron 244 (mobo Tyan 2885) system to a dual dual-core Opteron 285 (mobo Tyan 2895) system, I'm getting read errors that freezes the system which leads to my disk based backup software stopped working (faubackup). I think it is faubackup that triggers the bug.

I get these errors in the log:

Aug 20 06:35:08 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:35:56 jaguar kernel: end_request: I/O error, dev sda, sector 616924530
Aug 20 06:36:03 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:03 jaguar kernel: end_request: I/O error, dev sda, sector 616924538

..

Quoted 5 lines
Aug 20 06:36:07 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:07 jaguar kernel: end_request: I/O error, dev sda, sector 616924538

The last sector is repeated until I reboot the machine. The only difference I've made to the raid configuration is that sdc is now 2x250 MB instead of 4x120MB, but that array is the target not the source (sda). The raid HW is an LSI Megaraid 300-8x with the following configuration:

..

That looks like the classic SCSI bad-sectory non-recovery bug. The code in scsi_lib.c, scsi_error.c, and sd.c is currently a bit of a mess here.

Basically, given an I/O request for 200 sectors, with a bad sector in the middle at number 100, what SCSI will often do is fail sectors number 1 through 100, one at a time, retrying the entire remainder of the request after each attempt. This takes hours, and results in no data for the first 99 good sectors.

What it needs to do *instead*, is retry each sector individually, rather than the entire request. This would result in sectors 1..99 and 101..200 succeeding, and retries/failure only for sector 100.

A slight optimization would be to fail the bio size around sector 100, rather than just the one sector.

I've got patches that do exactly this, and they work quite well. But they're probably not "pretty enough" for inclusion.

Cheers

#2 lore raw
Quoted 1 line · Mark Lord

Johan Groth wrote:

[snip]

Quoted 5 lines

Basically, given an I/O request for 200 sectors, with a bad sector in the middle at number 100, what SCSI will often do is fail sectors number 1 through 100, one at a time, retrying the entire remainder of the request after each attempt. This takes hours, and results in no data for the first 99 good sectors.

So what you are saying is that after the move to a new box and a new mobo a sector has gone bad on that raid slice? Weird, as I was very careful this those drives when I moved them.

I mean, the raid controller is the same, the cpus are the same, just more of them, the pci-x bus the same so I didn't expect any problems at all.

I was also under the impression that SATA raid controllers work like SCSI raid controllers in the way that if a bad sector is encountered the controller moves what it can and the mark the sector as bad. I might be very wrong about that, though.

However, if I have a bad sector I would like to have that one marked as bad so the kernel never tries to read it again. Any suggestions how I do that. I assume I have to boot something like Knoppix as sda is my system disk.

Regards,
Johan

#3 lore raw
Quoted 19 lines · Johan Groth
Quoted 1 line · Mark Lord

Johan Groth wrote:

[snip]

Quoted 5 lines

Basically, given an I/O request for 200 sectors, with a bad sector in the middle at number 100, what SCSI will often do is fail sectors number 1 through 100, one at a time, retrying the entire remainder of the request after each attempt. This takes hours, and results in no data for the first 99 good sectors.

So what you are saying is that after the move to a new box and a new mobo a sector has gone bad on that raid slice? Weird, as I was very careful this those drives when I moved them.

I mean, the raid controller is the same, the cpus are the same, just more of them, the pci-x bus the same so I didn't expect any problems at all.

I was also under the impression that SATA raid controllers work like SCSI raid controllers in the way that if a bad sector is encountered the controller moves what it can and the mark the sector as bad. I might be very wrong about that, though.

However, if I have a bad sector I would like to have that one marked as bad so the kernel never tries to read it again. Any suggestions how I do that. I assume I have to boot something like Knoppix as sda is my system disk.

Quoted 9 lines

Regards,
Johan

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Run badblocks in r+w mode on the bad disk and it will force the disk to re-allocate the bad sector if it can.

Justin.

#4 lore raw
Quoted 3 lines · Justin Piszcz

Run badblocks in r+w mode on the bad disk and it will force the disk to re-allocate the bad sector if it can.

Justin.

Is that possible to do in a non-destructive way? I don't want to loose all data and apparently I can't back it up either :(.

Regards,
Johan

#5 lore raw
Quoted 10 lines · Johan Groth
Quoted 3 lines · Justin Piszcz

Run badblocks in r+w mode on the bad disk and it will force the disk to re-allocate the bad sector if it can.

Justin.

Is that possible to do in a non-destructive way? I don't want to loose all data and apparently I can't back it up either :(.

Regards,
Johan

Nope, r+w will write over everything on the disk, but I have found -the- most effective way to see if a disk is good or not. I'd rather have the disk die to that test rather than using it in production and finding it dies with my data on it.

Justin.

#6 lore raw
Quoted 3 lines · Justin Piszcz

Nope, r+w will write over everything on the disk, but I have found -the- most effective way to see if a disk is good or not. I'd rather have the disk die to that test rather than using it in production and finding it dies with my data on it.

Hmm, we both should read the man page of badblocks a bit better :). I found this:

-n Use non-destructive read-write mode. By default only a non-destructive read-only test is done. This option must not be combined with the -w option, as they are mutually exclusive.

Cheers,
Johan

#7 lore raw
Quoted 8 lines · Johan Groth
Quoted 3 lines · Justin Piszcz

Nope, r+w will write over everything on the disk, but I have found -the- most effective way to see if a disk is good or not. I'd rather have the disk die to that test rather than using it in production and finding it dies with my data on it.

Hmm, we both should read the man page of badblocks a bit better :). I found this:

Quoted 7 lines

-n Use non-destructive read-write mode. By default only a
non-destructive read-only test is done. This option must not be combined with the -w option, as they are mutually exclusive.

Cheers,
Johan

I have not tested that option. I wonder if it is as good as a real R+W mode. What does smartctl -a /dev/sda say on the disk that you are having problems with?

#8 lore raw
In reply to Johan Groth
Quoted 6 lines · Johan Groth
Quoted 3 lines · Justin Piszcz

Run badblocks in r+w mode on the bad disk and it will force the disk to re-allocate the bad sector if it can.

Justin.

Is that possible to do in a non-destructive way? I don't want to loose all data and apparently I can't back it up either :(.

Yes, it is perfectly doable, but I don't think anyone has yet bothered to release a utility that actually does it.

OPPORTUNITY FOR FAME AND FORTUNE! (okay, maybe just some fame):
=================================
Hack the existing smartctl code to read out the failed sector numbers, and then issue single-sector read-overwrite to each of those bad sectors.

Very simple code. I'll do it myself eventually, but please beat me to it!

Cheers

#9 lore raw
Quoted 11 lines · Mark Lord
Quoted 6 lines · Johan Groth
Quoted 3 lines · Justin Piszcz

Run badblocks in r+w mode on the bad disk and it will force the disk to re-allocate the bad sector if it can.

Justin.

Is that possible to do in a non-destructive way? I don't want to loose all data and apparently I can't back it up either :(.

Yes, it is perfectly doable, but I don't think anyone has yet bothered to release a utility that actually does it.

Quoted 6 lines

OPPORTUNITY FOR FAME AND FORTUNE! (okay, maybe just some fame):
=================================
Hack the existing smartctl code to read out the failed sector numbers, and then issue single-sector read-overwrite to each of those bad sectors.

Very simple code. I'll do it myself eventually, but please beat me to it!

Authors of badblocks already has with the -n option :). When I ran badblocks on the entire partition it wanted to check over 210 million blocks and when it finally came to the bad sector parts the controller lost the drive and the kernel started to spit out scsi errors! Buggy driver, hardware error? God knows. Unfortunately I don't have a log to show you as I was in single user mode.

I would like to run badblocks again but only around the damaged part. Thing is that I know which sector the kernel thinks is bad but badblocks wants to know which block to start and end at. How do I convert a sector number to block number. The partition is a standard XFS partition, ie I haven't made any changes to block sizes when I formatted it.

Got this from xfs_info:

meta-data=/dev/sda7              isize=256    agcount=16, agsize=3433014 blks
         =                       sectsz=512   attr=1
data     =                       bsize=4096   blocks=54928224, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=26820, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

There's another thing I would like know. How do I find out what file a sector belongs to?

Regards,
Johan

#10 lore raw
Quoted 10 lines · Johan Groth
Quoted 6 lines · Mark Lord

OPPORTUNITY FOR FAME AND FORTUNE! (okay, maybe just some fame):
=================================
Hack the existing smartctl code to read out the failed sector numbers, and then issue single-sector read-overwrite to each of those bad sectors.

Very simple code. I'll do it myself eventually, but please beat me to it!

Authors of badblocks already has with the -n option :)

Not quite. As you pointed out:

Quoted 1 line

I would like to run badblocks again but only around the damaged part.

The drive *knows* which sectors are bad -- it's mostly all in the S.M.A.R.T. logs and such. Which smartctl already knows how to read.

So now we just need a script-kiddie to do a nice little awk script for you, which extracts the bad sectors info from the output of smartctl, and then feeds this as input to badblocks.

Cheers