Skip to contentSkip to conversation

"This is a bug."

DISCUSSION Tapani Tarvainen 5 participants 21 messages Sep 10, 2015 – Sep 11, 2015

lore
Share & tools
lore
#0 lore raw

Hi,

After a rather spectacular crash we've got an xfs filesystem in unrepairable state: mount fails with "Structure needs cleaning", xfs_repair without options refuses to work (asks to mount first), and xfs_repair -L stopped with

corrupt dinode 2151195170, extent total = 1, nblocks = 0. This is a bug. Please capture the filesystem metadata with xfs_metadump and report it to [email protected].

I tried to dump the metadata and it failed, too:

# xfs_metadump /dev/sdata1/data1 /data2/tmp/data1_metadump
xfs_metadump: cannot init perag data (117)
*** glibc detected *** xfs_db: double free or corruption (!prev): 0x0000000003361000 ***
======= Backtrace: ========= 
[...]
Aborted

At this point I'm going to give up trying to recover the data (recreate filesystem and restore from backup), but if you want to analyze it to find the bug, I have enough spare disk space to keep a copy for a while (took lvm snapshot of it before recovery attempt, could dd it to another location).

The machine is running Debian Wheezy (7.8),
kernel 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u1 x86_64 GNU/Linux, xfsprogs version 3.1.7+b1.
And the filesystem is 6TB in size.

If you want to take a look, please let me know what I can do to help.

Thank you,

Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#1 lore raw

A perhaps interesting addition: xfs_metadump succeeds with the -o option (without it it fails consistently).

If you think the dump would be useful to you I can probably send it unobfuscated (would have to check but I don't think there's anything sensitive in the filenames or attributes).

Signature · 1 line
Tapani Tarvainen
Quoted 28 lines · Tapani Tarvainen

Hi,

After a rather spectacular crash we've got an xfs filesystem in unrepairable state: mount fails with "Structure needs cleaning", xfs_repair without options refuses to work (asks to mount first), and xfs_repair -L stopped with

corrupt dinode 2151195170, extent total = 1, nblocks = 0. This is a bug. Please capture the filesystem metadata with xfs_metadump and report it to [email protected].

I tried to dump the metadata and it failed, too:

# xfs_metadump /dev/sdata1/data1 /data2/tmp/data1_metadump
xfs_metadump: cannot init perag data (117)
*** glibc detected *** xfs_db: double free or corruption (!prev): 0x0000000003361000 ***
======= Backtrace: ========= 
[...]
Aborted

At this point I'm going to give up trying to recover the data (recreate filesystem and restore from backup), but if you want to analyze it to find the bug, I have enough spare disk space to keep a copy for a while (took lvm snapshot of it before recovery attempt, could dd it to another location).

Quoted 11 lines

The machine is running Debian Wheezy (7.8),
kernel 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u1 x86_64 GNU/Linux, xfsprogs version 3.1.7+b1.
And the filesystem is 6TB in size.

If you want to take a look, please let me know what I can do to help.

Thank you,

Signature · 1 line
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw

Le Thu, 10 Sep 2015 13:31:42 +0300
Tapani Tarvainen <[email protected]> écrivait:

Quoted 11 lines
Quoted 11 lines

Hi,

After a rather spectacular crash we've got an xfs filesystem in unrepairable state: mount fails with "Structure needs cleaning", xfs_repair without options refuses to work (asks to mount first), and xfs_repair -L stopped with

corrupt dinode 2151195170, extent total = 1, nblocks = 0. This is a bug. Please capture the filesystem metadata with xfs_metadump and report it to [email protected].

If you're not afraid of binaries from the web, I've just compiled xfs-repair 4.2.0 on a Wheezy machine:

http://update.intellique.com/pub/xfs_repair-4.2.0.gz

Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
Quoted 2 lines · Emmanuel Florac

If you're not afraid of binaries from the web, I've just compiled xfs-repair 4.2.0 on a Wheezy machine:

Thanks, but I'd just compiled it myself... waiting to see what it does.

Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#4 lore raw
In reply to Tapani Tarvainen

Le Thu, 10 Sep 2015 12:18:34 +0300
Tapani Tarvainen <[email protected]> écrivait:

Quoted 1 line

xfsprogs version 3.1.7+b1.

Don't use that, it's much too old. Use at least a 3.2.x version, or if possible the very latest version of xfs_repair. You can copy over xfs_repair from another machine as it's a static binary.

Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw
Quoted 5 lines · Emmanuel Florac
Quoted 1 line

xfsprogs version 3.1.7+b1.

Don't use that, it's much too old. Use at least a 3.2.x version, or if possible the very latest version of xfs_repair. You can copy over xfs_repair from another machine as it's a static binary.

Seems it isn't static in Debian (copied over from a Jessie box):

# /home/tt/xfs_repair -v /dev/sdata1/data1
/home/tt/xfs_repair: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /home/tt/xfs_repair)

But if a new version is really likely to help I can build it from source. Thank you for the suggestion.

Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#6 lore raw
Quoted 6 lines · Tapani Tarvainen
Quoted 4 lines · Emmanuel Florac
Quoted 1 line

xfsprogs version 3.1.7+b1.

Don't use that, it's much too old. Use at least a 3.2.x version, or if possible the very latest version of xfs_repair.

With (self-compiled) 4.2.0 version xfs_repair without -L no longer refuses to run but after a while failed with

[...]
correcting nextents for inode 2152363147
xfs_repair: dinode.c:1961: process_inode_data_fork: Assertion `err == 0' failed. Aborted

With -L ... same result.

Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#7 lore raw
Quoted 11 lines · Tapani Tarvainen
Quoted 6 lines · Tapani Tarvainen
Quoted 4 lines · Emmanuel Florac
Quoted 1 line

xfsprogs version 3.1.7+b1.

Don't use that, it's much too old. Use at least a 3.2.x version, or if possible the very latest version of xfs_repair.

With (self-compiled) 4.2.0 version xfs_repair without -L no longer refuses to run but after a while failed with

Quoted 7 lines

[...]
correcting nextents for inode 2152363147
xfs_repair: dinode.c:1961: process_inode_data_fork: Assertion `err == 0' failed. Aborted

With -L ... same result.

Care to post the metadump?

Brian

Quoted 7 lines
Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#10 lore raw
Quoted 3 lines · Brian Foster
Quoted 1 line

It is 2.5GB so not really nice to mail...

Can you compress it?

Ah. Of course, should've done it in the first place. Still 250MB though:

https://huom.it.jyu.fi/tmp/data1.metadump.gz

Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#11 lore raw
Quoted 11 lines · Tapani Tarvainen
Quoted 3 lines · Brian Foster
Quoted 1 line

It is 2.5GB so not really nice to mail...

Can you compress it?

Ah. Of course, should've done it in the first place. Still 250MB though:

https://huom.it.jyu.fi/tmp/data1.metadump.gz

First off, I see ~60MB of corruption output before I even get to the reported repair failure, so this appears to be an extremely severe corruption and I wouldn't be surprised if ultimately beyond repair (not that it matters for you, since you are restoring from backups).

The failure itself is an assert failure against an error return value that appears to have a fallback path, so I'm not really sure why it's there. I tried just removing it to see what happens. It ran to completion, but there was a ton of output, write verifier errors, etc., so I'm not totally sure how coherent the result is yet. I'll run another repair pass and do some directory traversals and whatnot and see if it explodes...

I suspect what's more interesting at this point is what happened to cause this level of corruption? What kind of event lead to this? Was it a pure filesystem crash or some kind of hardware/raid failure?

Also, do you happen to know the geometry (xfs_info) of the original fs? Repair was showing agno's up in the 20k's and now that I've mounted the repaired image, xfs_info shows the following:

meta-data=/dev/loop0             isize=256    agcount=24576, agsize=65536 blks
         =                       sectsz=4096  attr=2, projid32bit=0
         =                       crc=0        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=1610612736, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

So that's a 6TB fs with over 24000 allocation groups of size 256MB, as opposed to the mkfs default of 6 allocation groups of 1TB each. Is that intentional?

Brian

Quoted 7 lines
Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#12 lore raw
Quoted 18 lines · Brian Foster
Quoted 11 lines · Tapani Tarvainen
Quoted 3 lines · Brian Foster
Quoted 1 line

It is 2.5GB so not really nice to mail...

Can you compress it?

Ah. Of course, should've done it in the first place. Still 250MB though:

https://huom.it.jyu.fi/tmp/data1.metadump.gz

First off, I see ~60MB of corruption output before I even get to the reported repair failure, so this appears to be an extremely severe corruption and I wouldn't be surprised if ultimately beyond repair (not that it matters for you, since you are restoring from backups).

Quoted 8 lines

The failure itself is an assert failure against an error return value that appears to have a fallback path, so I'm not really sure why it's there. I tried just removing it to see what happens. It ran to completion, but there was a ton of output, write verifier errors, etc., so I'm not totally sure how coherent the result is yet. I'll run another repair pass and do some directory traversals and whatnot and see if it explodes...

FWIW, the follow up repair did come up clean so it appears (so far) to have put the fs back together from a metadata standpoint. That said,

Quoted 1 line

570k files end up in lost+found and who knows whether the files

themselves would have contained the expected data once all of the bmaps are fixed up and whatnot.

Brian

Quoted 32 lines

I suspect what's more interesting at this point is what happened to cause this level of corruption? What kind of event lead to this? Was it a pure filesystem crash or some kind of hardware/raid failure?

Also, do you happen to know the geometry (xfs_info) of the original fs? Repair was showing agno's up in the 20k's and now that I've mounted the repaired image, xfs_info shows the following:

meta-data=/dev/loop0             isize=256    agcount=24576, agsize=65536 blks
         =                       sectsz=4096  attr=2, projid32bit=0
         =                       crc=0        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=1610612736, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

So that's a 6TB fs with over 24000 allocation groups of size 256MB, as opposed to the mkfs default of 6 allocation groups of 1TB each. Is that intentional?

Brian

Quoted 7 lines
Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#13 lore raw
Quoted 2 lines · Brian Foster

FWIW, the follow up repair did come up clean so it appears (so far) to have put the fs back together from a metadata standpoint.

Indeed, now I can mount it. Not that I expect to find much useful there.

Also, it turns out I have another similar case: another filesystem (in the same RAID set) failed also, even though it'd initially mounted without problems. Now it shows the same symptoms, neither mount nor repair works (have't tried repair -L yet).

Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#14 lore raw
Quoted 4 lines · Tapani Tarvainen

Also, it turns out I have another similar case: another filesystem (in the same RAID set) failed also, even though it'd initially mounted without problems. Now it shows the same symptoms, neither mount nor repair works (have't tried repair -L yet).

Apparently it only failed after someone tried to write on it.

Potentially interesting stuff from dmesg:

[31604.130052] XFS (dm-4): xfs_da_do_buf: bno 8388608 dir: inode 964
[31604.130080] XFS (dm-4): [00] br_startoff 8388608 br_startblock -2 br_blockcount 1 br_state 0
[31604.130126] XFS (dm-4): Internal error xfs_da_do_buf(1) at line 2011 of file /build/linux-4wkEzn/
linux-3.2.68/fs/xfs/xfs_da_btree.c.  Caller 0xffffffffa0371b67
[31604.130127] 
[31604.130213] Pid: 18082, comm: du Not tainted 3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u1

[...]

Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#15 lore raw
In reply to Brian Foster
Quoted 3 lines · Brian Foster

First off, I see ~60MB of corruption output before I even get to the reported repair failure, so this appears to be an extremely severe corruption and I wouldn't be surprised if ultimately beyond repair

I assumed as much already.

Quoted 3 lines

I suspect what's more interesting at this point is what happened to cause this level of corruption? What kind of event lead to this? Was it a pure filesystem crash or some kind of hardware/raid failure?

Hardware failure. Details are still a bit unclear but apparently raid controller went haywire, offlining the array in the middle of heavy filesystem use.

Quoted 1 line

Also, do you happen to know the geometry (xfs_info) of the original fs?

No (and xfs_info doesn't work on the copy made after crash as it can't be mounted).

Quoted 2 lines

Repair was showing agno's up in the 20k's and now that I've mounted the repaired image, xfs_info shows the following:

[...]

Quoted 3 lines

So that's a 6TB fs with over 24000 allocation groups of size 256MB, as opposed to the mkfs default of 6 allocation groups of 1TB each. Is that intentional?

Not to my knowledge. Unless I'm mistaken, the filesystem was created while the machine was running Debian Squeeze, using whatever defaults were back then.

Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#16 lore raw
Quoted 23 lines · Tapani Tarvainen
Quoted 3 lines · Brian Foster

First off, I see ~60MB of corruption output before I even get to the reported repair failure, so this appears to be an extremely severe corruption and I wouldn't be surprised if ultimately beyond repair

I assumed as much already.

Quoted 3 lines

I suspect what's more interesting at this point is what happened to cause this level of corruption? What kind of event lead to this? Was it a pure filesystem crash or some kind of hardware/raid failure?

Hardware failure. Details are still a bit unclear but apparently raid controller went haywire, offlining the array in the middle of heavy filesystem use.

Quoted 1 line

Also, do you happen to know the geometry (xfs_info) of the original fs?

No (and xfs_info doesn't work on the copy made after crash as it can't be mounted).

Quoted 2 lines

Repair was showing agno's up in the 20k's and now that I've mounted the repaired image, xfs_info shows the following:

Quoted 9 lines

[...]

Quoted 3 lines

So that's a 6TB fs with over 24000 allocation groups of size 256MB, as opposed to the mkfs default of 6 allocation groups of 1TB each. Is that intentional?

Not to my knowledge. Unless I'm mistaken, the filesystem was created while the machine was running Debian Squeeze, using whatever defaults were back then.

Strange... was the filesystem created small and then grown to a much larger size via xfs_growfs? I just formatted a 1GB fs that started with 4 allocation groups and ends with 24576 (same as above) AGs when grown to 6TB.

Brian

Quoted 2 lines
Signature · 1 line
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#17 lore raw
Quoted 10 lines · Brian Foster
Quoted 7 lines
Quoted 3 lines

So that's a 6TB fs with over 24000 allocation groups of size 256MB, as opposed to the mkfs default of 6 allocation groups of 1TB each. Is that intentional?

Not to my knowledge. Unless I'm mistaken, the filesystem was created while the machine was running Debian Squeeze, using whatever defaults were back then.

Strange... was the filesystem created small and then grown to a much larger size via xfs_growfs?

Almost certainly yes, although how small it initially was I'm not sure.

Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#18 lore raw
Quoted 10 lines · Tapani Tarvainen
Quoted 7 lines · Brian Foster
Quoted 7 lines
Quoted 3 lines

So that's a 6TB fs with over 24000 allocation groups of size 256MB, as opposed to the mkfs default of 6 allocation groups of 1TB each. Is that intentional?

Not to my knowledge. Unless I'm mistaken, the filesystem was created while the machine was running Debian Squeeze, using whatever defaults were back then.

Quoted 6 lines
Quoted 2 lines

Strange... was the filesystem created small and then grown to a much larger size via xfs_growfs?

Almost certainly yes, although how small it initially was I'm not sure.

That probably explains that then. While growfs is obviously supported, it's not usually a great idea to grow from something really small to really large like this precisely because you end up with this kind of weird geometry. mkfs tries to format the fs to an ideal default geometry based on the current size of the device, but the allocation group size cannot be modified once the filesystem is created. Therefore, growfs can only add more AGs of the original size.

As a result, you end up with a 6TB filesystem with >24k allocation groups, whereas mkfs will format a 6TB device with 6 allocation groups by default (though I think specifying a stripe unit can tweak this). My understanding is that this could be increased sanely on large cpu count systems and such, but we're probably talking about going to something on the order of 32 or 64 allocation groups as opposed to thousands.

I'd expect such a large filesystem with such small allocation groups to probably introduce overhead in terms of metadata usage (24k agi's, agf's, 2x free space btrees and 1x inode btree per AG), spending more time in AG selection algorithms for allocations and whatnot, increased fragmentation due to capping the maximum contiguous extent size, creating more work for userspace tools such as repair, etc., and probably to have other weird or non-obvious side effects that I'm not familiar with.

Brian

Quoted 2 lines
Signature · 1 line
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#19 lore raw
Quoted 5 lines · Brian Foster
Quoted 5 lines
Quoted 2 lines

... was the filesystem created small and then grown to a much larger size via xfs_growfs?

Almost certainly yes, although how small it initially was I'm not sure.

It actually been grown several times over the years - the system is rather old. Indeed, all its disks have been replaced with bigger ones without reinstallation, so the filesystem could not have been initially created as big as it is now.

Quoted 3 lines

That probably explains that then. While growfs is obviously supported, it's not usually a great idea to grow from something really small to really large like this

That's good to know - but sometimes you just can't plan ahead far enough.

Quoted 8 lines

I'd expect such a large filesystem with such small allocation groups to probably introduce overhead in terms of metadata usage (24k agi's, agf's, 2x free space btrees and 1x inode btree per AG), spending more time in AG selection algorithms for allocations and whatnot, increased fragmentation due to capping the maximum contiguous extent size, creating more work for userspace tools such as repair, etc., and probably to have other weird or non-obvious side effects that I'm not familiar with.

So it's likely to also make it more fragile and harder to repair in case of a disaster like this.

So, my take from this is that

(1) The bug was real but it was just in the old version of xfs_repair in Debian Wheezy, and even when the machine is updated to Jessie (due soon) it's better to install latest (4.20) xfsprogs from sources rather Jessie's packaged 3.20; and

(2) When a filesystem grows a lot it is better to recreate it (at least every now and then if the growth is incremental) rather than keep growing it forever.

If there's anything you'd like to add, and especially if there is something you'd still like to debug where I could help, please let me know.

Thank you for your help,

Signature · 6 lines
Tapani Tarvainen

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#20 lore raw
In reply to Tapani Tarvainen
Quoted 10 lines · Tapani Tarvainen
Quoted 7 lines · Brian Foster
Quoted 7 lines
Quoted 3 lines

So that's a 6TB fs with over 24000 allocation groups of size 256MB, as opposed to the mkfs default of 6 allocation groups of 1TB each. Is that intentional?

Not to my knowledge. Unless I'm mistaken, the filesystem was created while the machine was running Debian Squeeze, using whatever defaults were back then.

Quoted 5 lines
Quoted 2 lines

Strange... was the filesystem created small and then grown to a much larger size via xfs_growfs?

Almost certainly yes, although how small it initially was I'm not sure.

Oof; with a default of 4 AGs that means that this filesystem was likely grown from 1G to 6T.

Like Brian says, that is definitely not recommended. ;)

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs