xfs corruption, data disaster!
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 28 lines · Yujian Peng
Quoted 17 lines · Emmanuel Florac
Le Mon, 4 May 2015 07:00:32 +0000 (UTC) Yujian Peng <pengyujian5201314 <at> 126.com> écrivait:
Quoted 4 lines
I'm encountering a data disaster. I have a ceph cluster with 145 osd. The data center had a power problem yesterday, and all of the ceph nodes were down. But now I find that 6 disks(xfs) in 4 nodes have data corruption. Some disks are unable to mount, and some disks have
Quoted 8 lines
IO errors in syslog. mount: Structure needs cleaning
xfs_log_forece: error 5 returned
I tried to repair one withxfs_repair-L/dev/sdx1, but the ceph-osd reported a leveldb error:
Error initializing leveldb: Corruption: checksum mismatch
I cannot start the 6 osds and 22 pgs is down.
This is really a tragedy for me. Can you give me some idea to recovery the xfs? Thanks very much!For XFS problems, ask the XFS ML: xfs <at> oss.sgi.com
Quoted 9 lines
You didn't give enough details, by far. What version of kernel and distro are you running? If there were errors, please post extensive logs. If you have IO errors on some disks, you probably MUST replace them before going any further.
Why did you run
xfs_repair-L ? Did you tryxfs_repairwithout options first? Were you running the very very latest version ofxfs_repair(3.2.2) ?
Quoted 12 lines
The OS is ubuntu 12.04.5 with kernel 3.13.0
uname -a
Linux ceph19 3.13.0-32-generic #57~precise1-Ubuntu SMP Tue Jul 15 03:51:20 UTC 2014x86_64x86_64x86_64GNU/Linux
cat/etc/issue
Ubuntu 12.04.5 LTS \n \lxfs_repair-Vxfs_repairversion 3.1.7
I've triedxfs_repairwithout options, but it showed me some errors, so I used the -L option.
Thanks for your reply!
Responding quickly to a couple of things:
* xfs_repair -L wipes out the XFS log, not normally a good thing to do
* replacing disks with IO errors is not a great idea if you still need that data. You might want to copy the data from that disk to a new disk (same or greater size) and then try to repair that new disk. A lot depends on the type of IO error you see - you might have cable issues, HBA issues, or fairly normal read issues (which are not worth replacing a disk for).
You should work with your vendor's support team if you have a support contract or post the the XFS devel list (copied above) for help.
Good luck!
Ric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 42 lines · Ric Wheeler
Quoted 28 lines · Yujian Peng
Quoted 17 lines · Emmanuel Florac
Le Mon, 4 May 2015 07:00:32 +0000 (UTC) Yujian Peng <pengyujian5201314 <at> 126.com> écrivait:
Quoted 4 lines
I'm encountering a data disaster. I have a ceph cluster with 145 osd. The data center had a power problem yesterday, and all of the ceph nodes were down. But now I find that 6 disks(xfs) in 4 nodes have data corruption. Some disks are unable to mount, and some disks have
Quoted 8 lines
IO errors in syslog. mount: Structure needs cleaning
xfs_log_forece: error 5 returned
I tried to repair one withxfs_repair-L/dev/sdx1, but the ceph-osd reported a leveldb error:
Error initializing leveldb: Corruption: checksum mismatch
I cannot start the 6 osds and 22 pgs is down.
This is really a tragedy for me. Can you give me some idea to recovery the xfs? Thanks very much!For XFS problems, ask the XFS ML: xfs <at> oss.sgi.com
Quoted 9 lines
You didn't give enough details, by far. What version of kernel and distro are you running? If there were errors, please post extensive logs. If you have IO errors on some disks, you probably MUST replace them before going any further.
Why did you run
xfs_repair-L ? Did you tryxfs_repairwithout options first? Were you running the very very latest version ofxfs_repair(3.2.2) ?Quoted 12 lines
The OS is ubuntu 12.04.5 with kernel 3.13.0
uname -a
Linux ceph19 3.13.0-32-generic #57~precise1-Ubuntu SMP Tue Jul 15 03:51:20 UTC 2014x86_64x86_64x86_64GNU/Linux
cat/etc/issue
Ubuntu 12.04.5 LTS \n \lxfs_repair-Vxfs_repairversion 3.1.7
I've triedxfs_repairwithout options, but it showed me some errors, so I used the -L option.
Thanks for your reply!
Quoted 3 lines
Responding quickly to a couple of things:
*
xfs_repair-L wipes out the XFS log, not normally a good thing to do
And if required due to an unreplayable log, often indicates some problem with the storage system. For example a volatile write cache not synced as needed, and lost along with a power loss, leading to a corrupted and unreplayable XFS log.
Quoted 6 lines
* replacing disks with IO errors is not a great idea if you still need that data. You might want to copy the data from that disk to a new disk (same or greater size) and then try to repair that new disk. A lot depends on the type of IO error you see - you might have cable issues, HBA issues, or fairly normal read issues (which are not worth replacing a disk for).
Just a note that XFS sometimes starts saying "IO error" when the filesystem has shut down; this isn't the same as a block-device-level IO error, but you haven't posted logs or anything, so I'm just guessing here.
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
-Eric
Quoted 5 lines
You should work with your vendor's support team if you have a support contract or post the the XFS devel list (copied above) for help.
Good luck!
Quoted 8 lines
Ric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs