Skip to contentSkip to conversation

XFS, empty files after a crash

DISCUSSION kfx 6 participants 11 messages Feb 21, 2012 – Feb 28, 2012

lore
Share & tools
lore

kfx

author
#0 lore raw

Hi,

After a crash, a lot of files on a xfs file system report an empty size with "ls -a" but not with "du". xfs_check and xfs_repair don't report any problem to repair (although an internal log is used) :(
So if I understand correctly, those empty files actually use space on the disk but are incorrectly reported as empty by the system. Is there a way to copy back the used blocks to recover the files ?

Example for a previously not empty file:

# ls -l myfile
-rw-r--r-- 1 me me 0 2012-01-24 15:29 myfile
# du  myfile
27460    myfile
# du --apparent-size myfile
0    myfile
# xfs_bmap myfile
myfile: no extents

System: Ubuntu 11.04 2.6.38-8-server x86_64

xfs_info /dev/sdc1

meta-data=/dev/sdc1              isize=256    agcount=10, agsize=268435328 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=2441215488, imaxpct=5
         =                       sunit=128    swidth=640 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Thank you

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 2 lines

Hi, After a crash, a lot of files on a xfs file system report an empty size with "ls -a" but not with "du". [ ... ]

It is a FAQ:

http://xfs.org/index.php/XFS_FAQ#Q:_Why_do_I_see_binary_NULLS_in_some_files_after_recovery_when_I_unplugged_the_power.3F

Usually it is a good idea to do a web search before reporting a problem.

Quoted 2 lines

Is there a way to copy back the used blocks to recover the files ?

No, because your applications told XFS that the content of those blocks could be thrown away in case of crash.

It is your responsibility to ensure that your applications use 'fsync' or at least 'fsyncdata' when data should be stored permanently before a crash occurs.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 9 lines · Peter Grandi
Quoted 2 lines

Hi, After a crash, a lot of files on a xfs file system report an empty size with "ls -a" but not with "du". [ ... ]

It is a FAQ:

http://xfs.org/index.php/XFS_FAQ#Q:_Why_do_I_see_binary_NULLS_in_some_files_after_recovery_when_I_unplugged_the_power.3F

Usually it is a good idea to do a web search before reporting a problem.

He seems to be reporting a different problem.

In the FAQ case above, one would get a file but "it has is a size but no extents " - i.e. ls -l would show 1M, but du would show 0, and no extents are allocated.

In this case, he has no size, blocks are reported used, but no extents are allocated. That actually seems like a new/odd case.

Quoted 9 lines
Quoted 2 lines

Is there a way to copy back the used blocks to recover the files ?

No, because your applications told XFS that the content of those blocks could be thrown away in case of crash.

It is your responsibility to ensure that your applications use 'fsync' or at least 'fsyncdata' when data should be stored permanently before a crash occurs.

I'd agree that there is likely nothing to recover; there are no extents allocated. I'm not sure why du is reporting space used though.

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
Quoted 1 line · Eric Sandeen

I'd agree that there is likely nothing to recover; there are no extents allocated. I'm not sure why du is reporting space used though.

I have noticed that xfs_bmap (which uses the ioctl bmap interface) will not report extents after EOF. There probably *are* extents allocated to this file, but they are not being reported by xfs_bmap.

kfx, try getting the inode number of the file (via stat or ls -i) and then doing something like this:

xfs_db -r $DEV -c "inode $INO" -c "bmap"

This will dump the entire extent map, and I suspect you'll see at least one extent.

nate

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#4 lore raw
Quoted 4 lines

kfx, try getting the inode number of the file (via stat or ls -i) and then doing something like this:

xfs_db -r $DEV -c "inode $INO" -c "bmap"

# xfs_db -r /dev/sdc1 -c "inode 114748" -c "bmap"
data offset 0 startblock 1881705728 (7/2657536) count 6460 flag 0
# xfs_db -r /dev/sdc1 -c "inode 114754" -c "bmap"
data offset 0 startblock 1077794560 (4/4052736) count 6582 flag 0

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw
Quoted 11 lines · On 02/23/2012 03:07 PM, [email protected] wrote:
Quoted 4 lines

kfx, try getting the inode number of the file (via stat or ls -i) and then doing something like this:

xfs_db -r $DEV -c "inode $INO" -c "bmap"

# xfs_db -r /dev/sdc1 -c "inode 114748" -c "bmap"
data offset 0 startblock 1881705728 (7/2657536) count 6460 flag 0
# xfs_db -r /dev/sdc1 -c "inode 114754" -c "bmap"
data offset 0 startblock 1077794560 (4/4052736) count 6582 flag 0

If you want to see what's behind those data extents (which are probably partially written), you could do something along these lines:

# Determine the AG size
agblocks=$(xfs_db -r /dev/sdc1 -c sb -c p | grep ^agblocks | sed 's/.* = //')

# Copy the extent in the first file, which consists of 6460 blocks (~26MB)

# in AG 7 starting at AG-relative block 2657536:
dd if=/dev/sdc1 bs=4096 skip=$(($agblocks * 7 + 2657536)) count=6460 of=./blob
# examine ./blob
...

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#6 lore raw
Quoted 15 lines · Nathaniel W. Turner
Quoted 11 lines · On 02/23/2012 03:07 PM, [email protected] wrote:
Quoted 4 lines

kfx, try getting the inode number of the file (via stat or ls -i) and then doing something like this:

xfs_db -r $DEV -c "inode $INO" -c "bmap"

# xfs_db -r /dev/sdc1 -c "inode 114748" -c "bmap"
data offset 0 startblock 1881705728 (7/2657536) count 6460 flag 0
# xfs_db -r /dev/sdc1 -c "inode 114754" -c "bmap"
data offset 0 startblock 1077794560 (4/4052736) count 6582 flag 0

If you want to see what's behind those data extents (which are probably partially written), you could do something along these lines:

Quoted 7 lines

# Determine the AG size
agblocks=$(xfs_db -r /dev/sdc1 -c sb -c p | grep ^agblocks | sed 's/.* = //')

# Copy the extent in the first file, which consists of 6460 blocks (~26MB)

# in AG 7 starting at AG-relative block 2657536:
dd if=/dev/sdc1 bs=4096 skip=$(($agblocks * 7 + 2657536)) count=6460 of=./blob
# examine ./blob

Awesome, it's working for each file with non-null data extents.

I guess that for inode for which the command "xfs_db -r $DEV -c "inode $INO" -c "bmap"" returns a null result there is no hope ?

Anyway, thanks a lot Nathaniel !

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw
In reply to kfx
Quoted 7 lines · kfx

After a crash, a lot of files on a xfs file system report an empty size with "ls -a" but not with "du".
xfs_check and xfs_repair don't report any problem to repair (although an internal log is used) :(
So if I understand correctly, those empty files actually use space on the disk but are incorrectly reported as empty by the system. Is there a way to copy back the used blocks to recover the files ?

What was the nature of the crash? Kernel panic? Power failure? Other?

Do you have have backups of previously existing files that are now shown as zero size? If so delete and restore, maybe just overwrite.

For new files that were in the process of being created you're simply out of luck. The metadata may have hit the journal but the file data in buffer cache was lost.

In memory write buffering/ordering techniques are what give modern filesystems their write throughput performance. Unfortunately this comes at a cost, when systems crash. As Peter mentioned, one way to mitigate this is coding apps to sync their writes. This obviously has a performance impact. Another is possibly to mount filesystems with the sync option, which will obviously severely impact write performance. Both will prevent or at least lessen the carnage you've seen as a result of this crash. You may not be able to live the performance of the latter, though.

Signature · 6 lines
Stan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw
In reply to kfx
Quoted 14 lines · kfx

Hi,

After a crash, a lot of files on a xfs file system report an empty size with "ls -a" but not with "du".
xfs_check and xfs_repair don't report any problem to repair (although an internal log is used) :(
So if I understand correctly, those empty files actually use space on the disk but are incorrectly reported as empty by the system. Is there a way to copy back the used blocks to recover the files ?

Example for a previously not empty file:

# ls -l myfile
-rw-r--r-- 1 me me 0 2012-01-24 15:29 myfile
Quoted 8 lines
# du  myfile
27460    myfile
# du --apparent-size myfile
0    myfile
# xfs_bmap myfile
myfile: no extents

That doesn't seem right:

$ ls -l foobar
-rw------- 1 root root 0 Feb 27 11:54 foobar
$ du foobar 
1024    foobar
$ du --apparent-size foobar 
0       foobar
$ xfs_bmap foobar
foobar:
	0: [0..2047]: 255169872..255171919

So xfs_bmap can and does report extents beyond EOF.

du uses the fstat(2) syscall to get the block count from the inode, so it's seeing an inode with a block count but no extents. Can you dump the inode via xfs_db like so:

$ ls -i foobar
604424233 foobar
$ sudo xfs_db -r -c "inode 604424233" -c p /dev/md0
core.magic = 0x494e
core.mode = 0100600
core.version = 2
.....

And post the output?

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#9 lore raw
Quoted 35 lines · Dave Chinner
Quoted 8 lines · kfx
# du  myfile
27460    myfile
# du --apparent-size myfile
0    myfile
# xfs_bmap myfile
myfile: no extents

That doesn't seem right:

$ ls -l foobar
-rw------- 1 root root 0 Feb 27 11:54 foobar
$ du foobar
1024    foobar
$ du --apparent-size foobar
0       foobar
$ xfs_bmap foobar
foobar:
	0: [0..2047]: 255169872..255171919

So xfs_bmap can and does report extents beyond EOF.

du uses the fstat(2) syscall to get the block count from the inode, so it's seeing an inode with a block count but no extents. Can you dump the inode via xfs_db like so:

$ ls -i foobar
604424233 foobar
$ sudo xfs_db -r -c "inode 604424233" -c p /dev/md0
core.magic = 0x494e
core.mode = 0100600
core.version = 2
.....
Quoted 1 line

And post the output?

[root@server 1_out]# du foobar
25840 foobar
[root@server 1_out]# du --apparent-size foobar
0 foobar
[root@server 1_out]# xfs_bmap foobar
foobar: no extents
[root@server 1_out]# ls -i foobar
114748 foobar
[root@server 1_out]# xfs_db -r -c "inode 114748" -c p /dev/sdc1 core.magic = 0x494e
core.mode = 0100644
core.version = 2
core.format = 2 (extents)
core.nlinkv2 = 1
core.onlink = 0
core.projid = 0
core.uid = 12488
core.gid = 12488
core.flushiter = 0
core.atime.sec = Tue Jan 24 14:50:52 2012
core.atime.nsec = 609667096
core.mtime.sec = Tue Jan 24 14:50:53 2012
core.mtime.nsec = 579672930
core.ctime.sec = Tue Jan 24 14:50:53 2012
core.ctime.nsec = 579672930
core.size = 0
core.nblocks = 6460
core.extsize = 0
core.nextents = 1
core.naextents = 0
core.forkoff = 0
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 1924597489
next_unlinked = null
u.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,1881705728,6460,0]

After using the Nathaniel's advices and dd'ing the file:
[root@server recover]# ls -alh foobar -rw-r--r-- 1 root root 26M Feb 24 10:39 /recover/foobar

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#10 lore raw
Quoted 27 lines · On Mon, Feb 27, 2012 at 09:43:42AM +0100, [email protected] wrote:
Quoted 26 lines · Dave Chinner
Quoted 8 lines · kfx
# du  myfile
27460    myfile
# du --apparent-size myfile
0    myfile
# xfs_bmap myfile
myfile: no extents

That doesn't seem right:

$ ls -l foobar
-rw------- 1 root root 0 Feb 27 11:54 foobar
$ du foobar
1024    foobar
$ du --apparent-size foobar
0       foobar
$ xfs_bmap foobar
foobar:
	0: [0..2047]: 255169872..255171919

So xfs_bmap can and does report extents beyond EOF.

du uses the fstat(2) syscall to get the block count from the inode, so it's seeing an inode with a block count but no extents. Can you dump the inode via xfs_db like so:

.....

Quoted 5 lines

[root@server 1_out]# xfs_db -r -c "inode 114748" -c p /dev/sdc1 core.magic = 0x494e
core.mode = 0100644
core.version = 2
core.format = 2 (extents)

.....

Quoted 4 lines

core.size = 0
core.nblocks = 6460
core.extsize = 0
core.nextents = 1

.....

Quoted 1 line

core.prealloc = 0

....

Quoted 2 lines

core.extsz = 0
core.extszinherit = 0

....

Quoted 2 lines

u.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,1881705728,6460,0]

Ok, so there's an extent there, but xfs_bmap is not showing it because it isn't a preallocation but rather a normal extent. What this indicates is that the inode size was not updated on disk after the data write had occurred.

Given you are running 2.6.38 (the unbuntu kernel), this is likely caused by a known bug introduced in 2.6.37 by VFS changes outside XFS to prevent sync livelocks. That has been fixed in current 3.0.x stable kernels and more recent 3.2+ kernels. I'd suggest upgrading your kernel.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs