Skip to contentSkip to conversation

advice for repair after IO error on raid device

DISCUSSION Roel van Meer 4 participants 7 messages Jun 22, 2010 – Jun 24, 2010

lore
Share & tools
lore
#0 lore raw

Hi list,

I recently I had a failed disk in a raid6 setup, which resulted in an IO error, which in turn caused XFS to shut down with the messages below.

I've seen on this list that incorrect use of xfs_repair might damage the fs even more, so I would like to ask for some advice on the best way to proceed.

Currently I have unmounted the filesystem, replaced the failed disk and rebuilt the raid array. I am upgrading xfstools to their latest version (the current version is 2.9.8). Any hints on how to continue would be highly appreciated.

Background: This is a Fedora Core 3 machine, with a vanilla 2.6.31 kernel.
The raid setup consists of 24x2TB disks in a raid6 setup. We use it to store our backup snapshots and the entire volume is written to tape once a week.

Thanks in advance,

roel

Jun 21 23:23:59 backup2 kernel: arcmsr6: abort device command of scsi id = 0 lun = 0
Jun 21 23:24:10 backup2 kernel: arcmsr6: ccb ='0xffff8800cb88ad40'????????????????????????????? isr got aborted command
Jun 21 23:24:10 backup2 kernel: arcmsr6: isr get an illegal ccb command???????????????????????????????? done acb = '0xffff880231c90408'ccb = '0xffff8800cb88ad40' ccbacb = '0xffff880231c90408' startdone = 0x0 ccboutstandingcount = 1
Jun 21 23:24:10 backup2 kernel: sd 6:0:0:0: [sdb] Unhandled error code
Jun 21 23:24:10 backup2 kernel: sd 6:0:0:0: [sdb] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
Jun 21 23:24:10 backup2 kernel: end_request: I/O error, dev sdb, sector 12887056410
Jun 21 23:24:10 backup2 kernel: I/O error in filesystem ("sdb1") meta-data dev sdb1 block 0x30020dff8?????? ("xfs_trans_read_buf") error 5 buf count 4096
Jun 21 23:24:10 backup2 kernel: xfs_force_shutdown(sdb1,0x1) called from line 414 of file fs/xfs/xfs_trans_buf.c.? Return address = 0xffffffffa0168eaf
Jun 21 23:24:10 backup2 kernel: xfs_force_shutdown(sdb1,0x2) called from line 811 of file fs/xfs/xfs_log.c.? Return address = 0xffffffffa015c35f
Jun 21 23:24:10 backup2 kernel: Filesystem "sdb1": I/O Error Detected.? Shutting down filesystem: sdb1
Jun 21 23:24:10 backup2 kernel: Please umount the filesystem, and rectify the problem(s)
Jun 21 23:24:20 backup2 kernel: Filesystem "sdb1": xfs_log_force: error 5 returned.
Jun 21 23:24:50 backup2 kernel: Filesystem "sdb1": xfs_log_force: error 5 returned.
Jun 21 23:25:20 backup2 kernel: Filesystem "sdb1": xfs_log_force: error 5 returned.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 1 line · Roel van Meer

Currently I have unmounted the filesystem, replaced the failed disk and rebuilt the raid array. I am upgrading xfstools to their latest version (the current version is 2.9.8). Any hints on how to continue would be highly appreciated.

Trying to answer my own question. I _think_ this is the way to go:

1) Mount and unmount the fs, in order to replay the log.
2) Run xfs_repair -n
3) Run xfs_repair

If someone could confirm (or reject) that, that would be great.

(By the way, is it necessary to run xfs_repair with -n first? If not, are there advantages that would justify the extra time it takes?)

Thanks again,

roel

Quoted 6 lines
Jun 21 23:23:59 backup2 kernel: arcmsr6: abort device command of scsi id = 0 lun = 0
Jun 21 23:24:10 backup2 kernel: arcmsr6: ccb ='0xffff8800cb88ad40'????????????????????????????? isr got aborted command
Jun 21 23:24:10 backup2 kernel: arcmsr6: isr get an illegal ccb command???????????????????????????????? done acb = '0xffff880231c90408'ccb = '0xffff8800cb88ad40' ccbacb = '0xffff880231c90408' startdone = 0x0 ccboutstandingcount = 1
Jun 21 23:24:10 backup2 kernel: sd 6:0:0:0: [sdb] Unhandled error code
Jun 21 23:24:10 backup2 kernel: sd 6:0:0:0: [sdb] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
Jun 21 23:24:10 backup2 kernel: end_request: I/O error, dev sdb, sector 12887056410
Quoted 8 lines
Jun 21 23:24:10 backup2 kernel: I/O error in filesystem ("sdb1") meta-data dev sdb1 block 0x30020dff8?????? ("xfs_trans_read_buf") error 5 buf count 4096
Jun 21 23:24:10 backup2 kernel: xfs_force_shutdown(sdb1,0x1) called from line 414 of file fs/xfs/xfs_trans_buf.c.? Return address = 0xffffffffa0168eaf
Jun 21 23:24:10 backup2 kernel: xfs_force_shutdown(sdb1,0x2) called from line 811 of file fs/xfs/xfs_log.c.? Return address = 0xffffffffa015c35f
Jun 21 23:24:10 backup2 kernel: Filesystem "sdb1": I/O Error Detected.? Shutting down filesystem: sdb1
Jun 21 23:24:10 backup2 kernel: Please umount the filesystem, and rectify the problem(s)
Jun 21 23:24:20 backup2 kernel: Filesystem "sdb1": xfs_log_force: error 5 returned.
Jun 21 23:24:50 backup2 kernel: Filesystem "sdb1": xfs_log_force: error 5 returned.
Jun 21 23:25:20 backup2 kernel: Filesystem "sdb1": xfs_log_force: error 5 returned.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw

Hi Roel!

Quoted 7 lines

Trying to answer my own question. I _think_ this is the way to go:

1) Mount and unmount the fs, in order to replay the log.
2) Run xfs_repair -n
3) Run xfs_repair

If someone could confirm (or reject) that, that would be great.

This is a sound plan, i always do a "xfs_repair -n" first. If you're lucky it will find no problems and you can skip step 3).
If "xfs_repair -n" does find problems you can ask for more advice here.

good luck,
Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
In reply to Roel van Meer

Le Tue, 22 Jun 2010 16:53:32 +0200
Roel van Meer <[email protected]> écrivait:

Quoted 1 line

If someone could confirm (or reject) that, that would be great.

Confirmation granted.

Quoted 2 lines

(By the way, is it necessary to run xfs_repair with -n first? If not, are there advantages that would justify the extra time it takes?)

It will indicate what modifications he would have done, before actually doing them... Like "inode 038953095 corrupted, would remove it" then "directory not connected, would move content moved to lost+found"

I found that usually xfs_repair is quit quick and never takes much more than a couple of minutes (even on very big arrays) so you have no valid reason to skip the "xfs_repair -n" part.

It may be nice to dump the filesystem metadata with xfs_metadump prior to using xfs_repair too. In the case where the repair's really gone bad, you could at least revert to the prior less broken state using xfs_mdrestore...

Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#4 lore raw
In reply to Roel van Meer
Quoted 3 lines · On Dienstag, 22. Juni 2010 Roel van Meer wrote:
Jun 21 23:23:59 backup2 kernel: arcmsr6: abort device command of scsi
 id = 0 lun = 0 Jun 21 23:24:10 backup2 kernel: arcmsr6: ccb
 ='0xffff8800cb88ad40'????????????????????????????? isr got aborted

This does not sound like a simple failed disk. We also use Areca RAID controllers, they are great, and a failed disk does *never* influence the running system - that's what you have the RAID controller for, after all. You only receive an e-mail from the controller about the disabled disk.

The "isr got aborted" sounds like a driver problem, please report that log snipped to areca support so they can help you (I'd be interested in the results too - please tell me per PM once you know). The address is [email protected], and they tend to answer quickly and good.

Signature · 10 lines
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw

Le Tue, 22 Jun 2010 18:07:18 +0200
Michael Monnerie <[email protected]> écrivait:

Quoted 1 line

The "isr got aborted" sounds like a driver problem

You bet, isr stands for "interrupt system routine", apparently the driver (or the firmware) seriously shat its pants. Maybe a firmware and/or kernel upgrade would be good too.

Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs