Question about file system failure
Quoted 29 lines · Dave Kleikamp
Quoted 28 lines · Dave Kleikamp
Quoted 7 lines · Chris Penney
I had an NFS file server using JFS fail this weekend. A reboot, which made fsck do a full check, seems to have cleared everything up. The initial errors I got were:
Jun 25 09:27:04 nicfs2 kernel: Incorrect number of segments after building list Jun 25 09:27:04 nicfs2 kernel: counted 16, received 15 Jun 25 09:27:04 nicfs2 kernel: req nr_sec 320, cur_nr_sec 8These are coming from
scsi_init_io()indrivers/scsi/scsi_lib.c. I don't know what it means, but I'm inclined to think that it indicates a software bug rather than a hardware error.Quoted 7 lines
Jun 25 09:27:04 nicfs2 kernel: device-mapper: dm-multipath: Failing path 8:96. Jun 25 09:27:04 nicfs2 kernel: cfq: depth 4 reached, tagging now on Jun 25 09:27:04 nicfs2 kernel: end_request: I/O error, dev sdc, sector 1592060824 Jun 25 09:27:04 nicfs2 kernel: device-mapper: dm-multipath: Failing path 8:32. Jun 25 09:27:04 nicfs2 kernel: end_request: I/O error, dev sdc, sector1592062936
I'm not sure if dm-multipath may be responsible.
Quoted 2 lines
Following that was a flurry of JFS errors. I assume these messages have nothing at all to do with JFS, but I wanted to make certain.
I don't think that JFS is the cause.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I can't turn up much googling that error. If anyone has any idea what caused that I'd love to hear it.
I'm copying this to linux-kernel in the hopes that someone there will be able to help. It would be useful to know what kernel you are running.
Well, I meant to cc linux-kernel. :-)
I'm running:
* SMP Pentium 4 w/ hyperthreading enabled (IBM x345)
* Dual QLogic 2340 HBAs
* STK D280 Disk Array (nothing in logs)
* Kernel 2.6.11.5 (built around March 24th)
* 2.6.11-rc3-udm2 patch
* linux-2.6.11-NFS_ALL patch (this was critical for me)
I'm using DM to make four multipath devices (each device is a 1TB lun) and then one stripe made from the four multipaths. Because the array is unsupported by multipathd (at least in the code I have) it can only failover once. We then manually fail it back to primary if that happens (which hasn't happened outside of testing).
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
One last question, for an NFS server is it better to mount the volume with errors=panic? It seems like that would keep I/Os from failing due to it being a read-only file system on error. In this case it would seem like a panic + boot would have let a lot of processes (this is used in a batch environment) resume.
Seems reasonable, but I'll let others comment.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
Chris
Thanks,
Shaggy--
David Kleikamp
IBM Linux Technology Center
Thanks,
Chris
Quoted 26 lines · Dave Kleikamp
Quoted 7 lines · Chris Penney
I had an NFS file server using JFS fail this weekend. A reboot, which made fsck do a full check, seems to have cleared everything up. The initial errors I got were:
Jun 25 09:27:04 nicfs2 kernel: Incorrect number of segments after building list Jun 25 09:27:04 nicfs2 kernel: counted 16, received 15 Jun 25 09:27:04 nicfs2 kernel: req nr_sec 320, cur_nr_sec 8These are coming from
scsi_init_io()indrivers/scsi/scsi_lib.c. I don't know what it means, but I'm inclined to think that it indicates a software bug rather than a hardware error.Quoted 7 lines
Jun 25 09:27:04 nicfs2 kernel: device-mapper: dm-multipath: Failing path 8:96. Jun 25 09:27:04 nicfs2 kernel: cfq: depth 4 reached, tagging now on Jun 25 09:27:04 nicfs2 kernel: end_request: I/O error, dev sdc, sector 1592060824 Jun 25 09:27:04 nicfs2 kernel: device-mapper: dm-multipath: Failing path 8:32. Jun 25 09:27:04 nicfs2 kernel: end_request: I/O error, dev sdc, sector1592062936
I'm not sure if dm-multipath may be responsible.
Quoted 2 lines
Following that was a flurry of JFS errors. I assume these messages have nothing at all to do with JFS, but I wanted to make certain.
Quoted 7 lines
I don't think that JFS is the cause.
Quoted 2 lines
I can't turn up much googling that error. If anyone has any idea what caused that I'd love to hear it.
I'm copying this to linux-kernel in the hopes that someone there will be able to help. It would be useful to know what kernel you are running.
Well, I meant to cc linux-kernel. :-)
Quoted 12 lines
Quoted 5 lines
One last question, for an NFS server is it better to mount the volume with errors=panic? It seems like that would keep I/Os from failing due to it being a read-only file system on error. In this case it would seem like a panic + boot would have let a lot of processes (this is used in a batch environment) resume.
Seems reasonable, but I'll let others comment.
Quoted 1 line
Chris
Thanks,
Shaggy
Signature · 2 lines
David Kleikamp IBM Linux Technology Center