mount: Structure needs cleaning
I have two 2TB xfs volumes and earlier today the /var/log/messages shows "xfs_force_shutdown" after many errors (attached)
http://old.nabble.com/file/p33393100/var-log-message.txt var-log-message.txt
. Are there any options to try before running "xfs_repair -L"? The
volumes contain several millions files so that I my last resort. I'm a novice at best in Linux.
Signature · 7 lines
View this message in context: http://old.nabble.com/mount%3A-Structure-needs-cleaning-tp33393100p33393100.html Sent from the Xfs - General mailing list archive at Nabble.com. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 7 lines · MikeJeezy
I have two 2TB xfs volumes and earlier today the
/var/log/messagesshows "xfs_force_shutdown" after many errors (attached)
http://old.nabble.com/file/p33393100/var-log-message.txt var-log-message.txt
. Are there any options to try before running "xfs_repair-L"? The
volumes contain several millions files so that I my last resort. I'm a novice at best in Linux.
Googling "XFS_WANT_CORRUPTED_RETURN at line 280" turns up a whole lot of information on this. This is an Oct 2008 response to another user with this problem, from XFS developer Dave Chinner, one of the resident experts on this list:
"The freespace btrees are getting out of sync for some reason.
That is, when we go to allocate an extent, we have to update two free space btrees. This shutdown:
Quoted 1 line
XFS internal error
XFS_WANT_CORRUPTED_RETURNat line 280 of file
fs/xfs/xfs_alloc.c. Caller 0xf88e0018
Indicates the extent being allocated was not found in one of the two trees.
This corruption could have happened a long time ago in the past, and it may simply be coincidental that you've tripped over this at roughly the same time you upgraded the kernel.
Can you run xfs_check on the filesystem to determine if a freespace tree is corrupted (post the output if it is), then run xfs_repair to rebuild them?"
So, run "xfs_check /dev/sde1" and post the output here. Then await further instructions. Don't flush the log as that's probably not the problem. Doing so will delete pending metadata changes and could cause you more headaches.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · Stan Hoeppner
Can you run
xfs_checkon the filesystem to determine if a freespace tree is corrupted (post the output if it is), then runxfs_repairto rebuild them?"
Thank you for responding. This is a 24/7 production server and I did not anticipate getting a response this late on a Saturday, so I panicked quite frankly, and went ahead and ran "xfs_repair -L" on both volumes. I can now mount the volumes and everything looks okay as far as I can tell. There were only 2 files in the "lost+found" directory after the repair. Does that mean only two files were lost? Is there any way to tell how many files were lost?
Quoted 3 lines
This corruption could have happened a long time ago in the past, and it may simply be coincidental that you've tripped over this at roughly the same time you upgraded the kernel.
It would be nice to find out why this happened. I suspect it is as you suggested, previous corruption and not a hardware issue, because I have other volumes mounted to other VM's that are attached to the same SAN controller / RAID6 Array... and they did not have any issues - only this one VM.
Quoted 2 lines
So, run "
xfs_check/dev/sde1" and post the output here. Then await further instructions.
Can I still do this (or anything) to help uncover any causes or is it too late? I have also run yum update on the server because it was out of date.
Signature · 7 lines
View this message in context: http://old.nabble.com/mount%3A-Structure-needs-cleaning-tp33393100p33393429.html Sent from the Xfs - General mailing list archive at Nabble.com. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · MikeJeezy
Quoted 3 lines · Stan Hoeppner
Can you run
xfs_checkon the filesystem to determine if a freespace tree is corrupted (post the output if it is), then runxfs_repairto rebuild them?"Thank you for responding. This is a 24/7 production server and I did not anticipate getting a response this late on a Saturday, so I panicked quite frankly, and went ahead and ran "
xfs_repair-L" on both volumes. I can now
I wasn't sure how big a pickle you were in so I jumped in and tried to help best I could.
Quoted 4 lines
mount the volumes and everything looks okay as far as I can tell. There were only 2 files in the "lost+found" directory after the repair. Does that mean only two files were lost? Is there any way to tell how many files were lost?
I'm not sure. If this is free space btree corruption then you shouldn't have lost any user files. Others might answer this better than me.
Quoted 3 lines
Quoted 3 lines
This corruption could have happened a long time ago in the past, and it may simply be coincidental that you've tripped over this at roughly the same time you upgraded the kernel.
Note the text above is something I quoted from Dave's 2008 response to another user with the same problem. In that case he had just upgraded his kernel and suspected that as the cause. It was not.
Quoted 5 lines
It would be nice to find out why this happened. I suspect it is as you suggested, previous corruption and not a hardware issue, because I have other volumes mounted to other VM's that are attached to the same SAN controller / RAID6 Array... and they did not have any issues - only this one VM.
Are those other VMs using XFS filesystems?
Found this in the list archive:
Quoted 4 lines · Christoph Hellwig
Given that before ~2.6.35 LVM/device mapper was not able to pass through cache flush requests that is your most likely culprit. A repair will rebuild the freespace btrees, and make sure to keep the write caches down the whole stack disabled.
What kernel version are you running? Are you using LVM under XFS? What fstab mount options? Does your SAN array have battery backed write cache? Are the individual drive caches in the underlying array disabled?
Quoted 4 lines
Quoted 2 lines
So, run "
xfs_check/dev/sde1" and post the output here. Then await further instructions.Can I still do this (or anything) to help uncover any causes or is it too
If you already ran a repair and it fixed the damage then the check won't show anything.
Quoted 1 line
late? I have also run yum update on the server because it was out of date.
Answering the questions above may lead us to a possible/plausible cause.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · MikeJeezy
Quoted 3 lines · Stan Hoeppner
Can you run
xfs_checkon the filesystem to determine if a freespace tree is corrupted (post the output if it is), then runxfs_repairto rebuild them?"Thank you for responding. This is a 24/7 production server and I did not anticipate getting a response this late on a Saturday, so I panicked quite frankly, and went ahead and ran "
xfs_repair-L" on both volumes.
The only reason for running xfs-repair -L is if you cannot mount the filesystem to replay the log. i.e. on a shutdown like this, the usual process is:
<shutdown>
umount <dev>
mount <dev>
umount <dev>xfs_repair <dev>
The only reason for needing to run "xfs-repair -L <dev>" is if the mount after the shutdown fails to run log recovery.
Quoted 5 lines
I can now
mount the volumes and everything looks okay as far as I can tell. There were only 2 files in the "lost+found" directory after the repair. Does that mean only two files were lost? Is there any way to tell how many files were lost?
YOu can only find out by looking at what the output of xfs_repair told you about trashing inodes/directories.
Quoted 10 lines
Quoted 3 lines
This corruption could have happened a long time ago in the past, and it may simply be coincidental that you've tripped over this at roughly the same time you upgraded the kernel.
It would be nice to find out why this happened. I suspect it is as you suggested, previous corruption and not a hardware issue, because I have other volumes mounted to other VM's that are attached to the same SAN controller / RAID6 Array... and they did not have any issues - only this one VM.
Quoted 5 lines
Quoted 2 lines
So, run "
xfs_check/dev/sde1" and post the output here. Then await further instructions.Can I still do this (or anything) to help uncover any causes or is it too late? I have also run yum update on the server because it was out of date.
Too late. As it is, xfs-check is deprecated. use "xfs_repair -n <dev>" to check a filesystem for errors without modifying/fixing anything.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines
The only reason for running xfs-repair -L is if you cannot mount the filesystem to replay the log. i.e. on a shutdown like this, the usual process is:
Right, when I tried to mount the volume it said "Structure needs cleaning"
Quoted 2 lines
You can only find out by looking at what the output of
xfs_repairtold you about trashing inodes/directories.
Is this only output to the screen or to a log somewhere?
Quoted 2 lines · Stan Hoeppner
Are those other VMs using XFS filesystems?
What kernel version are you running?
2.6.18-274.18.1.el5
Quoted 1 line
Are you using LVM under XFS?
No
Quoted 1 line
What fstab mount options?
/dev/sdd1 /mnt/ob1 xfs defaults 0 0 /dev/sde1 /mnt/ob2 xfs defaults 0 0
Quoted 1 line
Does your SAN array have battery backed write cache?
This one does not currently, but I have ordered BBWC for it.
Quoted 1 line
Are the individual drive caches in the underlying array disabled?
Write cache: enabled
Read ahead: enabled
Signature · 7 lines
View this message in context: http://old.nabble.com/mount%3A-Structure-needs-cleaning-tp33393100p33397518.html Sent from the Xfs - General mailing list archive at Nabble.com. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · MikeJeezy
Quoted 2 lines · Stan Hoeppner
Are those other VMs using XFS filesystems?
What kernel version are you running?
2.6.18-274.18.1.el5
I'm not familiar enough with Red Hat kernel revs to know if all relevant patches are included in this kernel. There are a few Red Hat devs here who should have more insight on this.
Quoted 4 lines
Quoted 1 line
Are you using LVM under XFS?
No
Quoted 8 lines
Quoted 1 line
What fstab mount options?
/dev/sdd1 /mnt/ob1 xfs defaults 0 0 /dev/sde1 /mnt/ob2 xfs defaults 0 0Quoted 1 line
Does your SAN array have battery backed write cache?
This one does not currently, but I have ordered BBWC for it.
Good. I suggest disabling the SAN controller's write caching until the BBWC is installed and verified to be functioning correctly.
Quoted 4 lines
Quoted 1 line
Are the individual drive caches in the underlying array disabled?
Write cache: enabled
Read ahead: enabled
In the case of a SAN array or PCIe RAID controller, this dmesg output is telling you about the state of the controller's cache, not the individual drive caches. Enable/disable of the drive caches should be an option in the controller firmware interface. You want the individual drive write caches disabled. Leaving their read caches enabled is fine.
The reason is that a power drop, kernel panic, or hardware lockup (thermal etc) clears the drive write caches before the blocks are written to the platters. It is suspected that many/most of these free space btree corruptions, such as yours here, are caused by data in caches not being flushed to the platters. SAN/RAID controllers with BBWC usually guarantee data in the write cache gets properly flushed to the platters when the system comes back up.
So, way back when, you may have had a system (VM) crash of one kind or another, or an improper shutdown (VM power-off), then rebooted, and everything seemed fine. Months later, you discover you have a corrupted free space btree, which was caused by the crash long ago, that everyone forgot about, never documented, etc.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines
You want the individual drive write caches disabled. Leaving their read
caches enabled is fine.
Quoted 8 lines
The reason is that a power drop, kernel panic, or hardware lockup (thermal etc) clears the drive write caches before the blocks are written to the platters. It is suspected that many/most of these free space btree corruptions, such as yours here, are caused by data in caches not being flushed to the platters. SAN/RAID controllers with BBWC usually guarantee data in the write cache gets properly flushed to the platters when the system comes back up.
Makes total sense. Thanks so much.
Signature · 7 lines
View this message in context: http://old.nabble.com/mount%3A-Structure-needs-cleaning-tp33393100p33401743.html Sent from the Xfs - General mailing list archive at Nabble.com. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · MikeJeezy
Quoted 6 lines
The only reason for running xfs-repair -L is if you cannot mount the filesystem to replay the log. i.e. on a shutdown like this, the usual process is:
Right, when I tried to mount the volume it said "Structure needs cleaning"
OK, so you've done the right thing. I always like to check first ;)
Quoted 4 lines
Quoted 2 lines
You can only find out by looking at what the output of
xfs_repairtold you about trashing inodes/directories.Is this only output to the screen or to a log somewhere?
Only to stdout.
Quoted 6 lines
Quoted 2 lines · Stan Hoeppner
Are those other VMs using XFS filesystems?
What kernel version are you running?
2.6.18-274.18.1.el5
If you are running RHEL, you should really report problems through the support channel.
Quoted 7 lines
Quoted 1 line
Does your SAN array have battery backed write cache?
This one does not currently, but I have ordered BBWC for it.
Quoted 1 line
Are the individual drive caches in the underlying array disabled?
Write cache: enabled
There's your problem. You should read all the relevant sections on write caches in the XFS FAQ. e.g:
http://xfs.org/index.php/XFS_FAQ#Q._Which_settings_does_my_RAID_controller_need_.3F
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · Dave Chinner
Quoted 1 line
Write cache: enabled
There's your problem. You should read all the relevant sections on write caches in the XFS FAQ. e.g:
http://xfs.org/index.php/XFS_FAQ#Q._Which_settings_does_my_RAID_controller_need_.3F
As a side question to this: if I am using Linux's md software RAID with an HBA, and XFS is using write barriers (as I believe it does by default), is it still necessary to disable write caching on the drives?
My reading of the FAQ suggests that the write barrier by itself should be sufficient - but this requires that the md RAID driver implements write barriers correctly. I seem to remember reading a long time ago that LVM and/or dmraid didn't, although that may be different now.
Incidentally I'm using 'mdadm ... -b internal' to ensure that the RAID array marks dirty areas, so that if a write part-completes (e.g. a block is written to only one half of a RAID1) then at next power-up this should be made consistent again.
Regards,
Brian.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Brian Candler
Quoted 6 lines · Dave Chinner
Quoted 1 line
Write cache: enabled
There's your problem. You should read all the relevant sections on write caches in the XFS FAQ. e.g:
http://xfs.org/index.php/XFS_FAQ#Q._Which_settings_does_my_RAID_controller_need_.3F
As a side question to this: if I am using Linux's md software RAID with an HBA, and XFS is using write barriers (as I believe it does by default), is it still necessary to disable write caching on the drives?
No. The MD layers pass the correct flushes to the drives now.
Quoted 4 lines
My reading of the FAQ suggests that the write barrier by itself should be sufficient - but this requires that the md RAID driver implements write barriers correctly. I seem to remember reading a long time ago that LVM and/or dmraid didn't, although that may be different now.
It has since about 2.6.35.
Quoted 4 lines
Incidentally I'm using 'mdadm ... -b internal' to ensure that the RAID array marks dirty areas, so that if a write part-completes (e.g. a block is written to only one half of a RAID1) then at next power-up this should be made consistent again.
Assuming that the dirty area tracking uses write cache flushes correctly.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 7 lines · Dave Chinner
Quoted 5 lines
As a side question to this: if I am using Linux's md software RAID with an HBA, and XFS is using write barriers (as I believe it does by default), is it still necessary to disable write caching on the drives?
No. The MD layers pass the correct flushes to the drives now.
Thank you: that's very clear and very helpful.
Regards,
Brian.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs