Failure growing xfs with linux 3.10.5

Hello!

I think I'm facing the same problem as already described here:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

xfs_info /mnt
meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=319815680, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=60160, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Doing xfs_growfs again gives:

xfs_growfs /mnt
meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=319815680, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=60160, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning data blocks changed from 319815680 to 346030080

Quoted 4 lines · Michael Maier

Hello!

I think I'm facing the same problem as already described here:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Maybe you can try the tracing Dave suggested in that thread?

Quoted 7 lines · Eric Sandeen

Quoted 4 lines · Michael Maier

Hello!

I think I'm facing the same problem as already described here:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Maybe you can try the tracing Dave suggested in that thread?

I sent you a trace.

Meanwhile, I faced another problem on another xfs-file system with linux 3.10.5 which I never saw before. During writing a few bytes to disc, I got "disc full" and the writing failed.

At the same time, df reported 69G of free space! I ran xfs_repair -n and got:

xfs_repair -n /dev/mapper/raid0-daten2
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
sb_ifree 591, counted 492
^^^^^^^^^^^^^^^^^^^^^^^^^
What does this mean? How can I get rid of it w/o loosing data? This file system was created a few days ago and never resized.

Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.

Quoted 8 lines · Michael Maier

Meanwhile, I faced another problem on another xfs-file system with linux 3.10.5 which I never saw before. During writing a few bytes to disc, I got "disc full" and the writing failed.

At the same time, df reported 69G of free space! I ran xfs_repair -n and got:

Quoted 8 lines

xfs_repair -n /dev/mapper/raid0-daten2
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
sb_ifree 591, counted 492
^^^^^^^^^^^^^^^^^^^^^^^^^
What does this mean? How can I get rid of it w/o loosing data? This file system was created a few days ago and never resized.

Superblock inode counting is lazy - it can get out of sync in after an unclean shutdown, but generally mounting a dirty filesystem will result in it being recalculated rather than trusted to be correct. So there's nothing to worry about here.

Quoted 18 lines · Dave Chinner

Quoted 16 lines · Michael Maier

Meanwhile, I faced another problem on another xfs-file system with linux 3.10.5 which I never saw before. During writing a few bytes to disc, I got "disc full" and the writing failed.

At the same time, df reported 69G of free space! I ran xfs_repair -n and got:

xfs_repair -n /dev/mapper/raid0-daten2
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
sb_ifree 591, counted 492
^^^^^^^^^^^^^^^^^^^^^^^^^
What does this mean? How can I get rid of it w/o loosing data? This file system was created a few days ago and never resized.

Quoted 4 lines

Superblock inode counting is lazy - it can get out of sync in after an unclean shutdown, but generally mounting a dirty filesystem will result in it being recalculated rather than trusted to be correct. So there's nothing to worry about here.

When will it be self healed?

I still can see it today after 4 remounts!

This is strange and I can't use the free space, which I need! How can it be forced to be repaired w/o data loss?

Quoted 24 lines · Michael Maier

Quoted 22 lines · Dave Chinner

Quoted 8 lines · Michael Maier

Meanwhile, I faced another problem on another xfs-file system with linux 3.10.5 which I never saw before. During writing a few bytes to disc, I got "disc full" and the writing failed.

At the same time, df reported 69G of free space! I ran xfs_repair -n and got:

Quoted 8 lines

xfs_repair -n /dev/mapper/raid0-daten2
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
sb_ifree 591, counted 492
^^^^^^^^^^^^^^^^^^^^^^^^^
What does this mean? How can I get rid of it w/o loosing data? This file system was created a few days ago and never resized.

Superblock inode counting is lazy - it can get out of sync in after an unclean shutdown, but generally mounting a dirty filesystem will result in it being recalculated rather than trusted to be correct. So there's nothing to worry about here.

Quoted 1 line

When will it be self healed?

that depends on whether there's actually a problem. Like I said in the part you snipped off - if you ran xfs_repair -n on filesystem that needs log recovery that accounting difference is expected.

Quoted 2 lines

I still can see it today after 4 remounts!

See what?

Quoted 2 lines

This is strange and I can't use the free space, which I need! How can it be forced to be repaired w/o data loss?

The above is complaining about a free inode count mismatch, not a problem about free space being wrong. What problem are you actually having?

Quoted 27 lines · Dave Chinner

Quoted 25 lines · Michael Maier

Quoted 18 lines · Dave Chinner

Quoted 8 lines · Michael Maier

Meanwhile, I faced another problem on another xfs-file system with linux 3.10.5 which I never saw before. During writing a few bytes to disc, I got "disc full" and the writing failed.

At the same time, df reported 69G of free space! I ran xfs_repair -n and got:

Quoted 8 lines

xfs_repair -n /dev/mapper/raid0-daten2
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
sb_ifree 591, counted 492
^^^^^^^^^^^^^^^^^^^^^^^^^
What does this mean? How can I get rid of it w/o loosing data? This file system was created a few days ago and never resized.

Quoted 4 lines

Superblock inode counting is lazy - it can get out of sync in after an unclean shutdown, but generally mounting a dirty filesystem will result in it being recalculated rather than trusted to be correct. So there's nothing to worry about here.

When will it be self healed?

Quoted 3 lines

that depends on whether there's actually a problem. Like I said in the part you snipped off - if you ran xfs_repair -n on filesystem that needs log recovery that accounting difference is expected.

I know, that option -n doesn't do anything. It was intended, because xfs_repair destroyed a lot of data when applied at the other problem I have _and_ it repaired nothing at the same time!

Quoted 6 lines

Quoted 2 lines

This is strange and I can't use the free space, which I need! How can it be forced to be repaired w/o data loss?

The above is complaining about a free inode count mismatch, not a problem about free space being wrong. What problem are you actually having?

The application, which wanted to write a few bytes gets a "disk full" error although df -h reports 69GB of free space.

Quoted 28 lines · Michael Maier

Quoted 27 lines · Dave Chinner

Quoted 25 lines · Michael Maier

Quoted 18 lines · Dave Chinner

Quoted 8 lines · Michael Maier

Meanwhile, I faced another problem on another xfs-file system with linux 3.10.5 which I never saw before. During writing a few bytes to disc, I got "disc full" and the writing failed.

At the same time, df reported 69G of free space! I ran xfs_repair -n and got:

Quoted 8 lines

xfs_repair -n /dev/mapper/raid0-daten2
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
sb_ifree 591, counted 492
^^^^^^^^^^^^^^^^^^^^^^^^^
What does this mean? How can I get rid of it w/o loosing data? This file system was created a few days ago and never resized.

Quoted 4 lines

Superblock inode counting is lazy - it can get out of sync in after an unclean shutdown, but generally mounting a dirty filesystem will result in it being recalculated rather than trusted to be correct. So there's nothing to worry about here.

When will it be self healed?

Quoted 7 lines

Quoted 3 lines

that depends on whether there's actually a problem. Like I said in the part you snipped off - if you ran xfs_repair -n on filesystem that needs log recovery that accounting difference is expected.

I know, that option -n doesn't do anything. It was intended, because xfs_repair destroyed a lot of data when applied at the other problem I have _and_ it repaired nothing at the same time!

xfs_repair will remove files it cannot repair because their metadata is are too corrupted to repair or cannot be repaired safely. That's always been the case for any filesystem repair tool - all they guarantee is that the filesystem will be consistent after they are run. Repairing a corrupted filesystem almost always results in some form of data loss occurring....

If there is nothing wrong with the filesystem except the accouting is wrong, then it will fix the accounting problem in phase 5 when run without the -n parameter.

Quoted 9 lines

Quoted 6 lines

Quoted 2 lines

This is strange and I can't use the free space, which I need! How can it be forced to be repaired w/o data loss?

The above is complaining about a free inode count mismatch, not a problem about free space being wrong. What problem are you actually having?

The application, which wanted to write a few bytes gets a "disk full" error although df -h reports 69GB of free space.

That's not necessarily a corruption, though, and most likely isn't related to the accounting issue xfs_repair is reporting. Indeed, this is typically a sign of being unable to allocate an inode because there is insufficient contiguous free space in the filesystem to allocate a new inode chunk. What does your free space histogram look like?

# xfs_db -r -c "freesp -s" <dev>

Quoted 8 lines · Eric Sandeen

Quoted 4 lines · Michael Maier

Hello!

I think I'm facing the same problem as already described here:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Maybe you can try the tracing Dave suggested in that thread? It certainly does look similar.

I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).

xfs_growfs /mnt
meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=319815680, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=60160, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning data blocks changed from 319815680 to 346030080

The entry in messages was:

Aug 12 18:09:50 dualc kernel: [  257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00  XFSB............
Aug 12 18:09:50 dualc kernel: [  257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Aug 12 18:09:50 dualc kernel: [  257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1  F.....M.......f.
Aug 12 18:09:50 dualc kernel: [  257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80  ................
Aug 12 18:09:50 dualc kernel: [  257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file

/daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd

Aug 12 18:09:50 dualc kernel: [  257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
Aug 12 18:09:50 dualc kernel: [  257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
Aug 12 18:09:50 dualc kernel: [  257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16

[ re-ccing the list, because finding this is in everyone's interest ]

Quoted 32 lines · Michael Maier

Quoted 8 lines · Eric Sandeen

Quoted 4 lines · Michael Maier

Hello!

I think I'm facing the same problem as already described here:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Maybe you can try the tracing Dave suggested in that thread? It certainly does look similar.

I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).

xfs_growfs /mnt
meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=319815680, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=60160, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning data blocks changed from 319815680 to 346030080

The entry in messages was:

Aug 12 18:09:50 dualc kernel: [  257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00  XFSB............
Aug 12 18:09:50 dualc kernel: [  257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Aug 12 18:09:50 dualc kernel: [  257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1  F.....M.......f.
Aug 12 18:09:50 dualc kernel: [  257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80  ................
Aug 12 18:09:50 dualc kernel: [  257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file

/daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd

.....

Quoted 3 lines

Aug 12 18:09:50 dualc kernel: [  257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
Aug 12 18:09:50 dualc kernel: [  257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
Aug 12 18:09:50 dualc kernel: [  257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16

Ok, so that's reading the secondary superblock for AG 16. You're growing the filesystem from 42 to 45 AGs, so this problem is not related to the actual grow operation - it's tripping over a problem that already exists on disk before the grow operation is started. i.e. this is likely to be a real corruption being seen, and it happened some time in the distant past and so we probably won't ever be able to pinpoint the cause of the problem.

That said, let's have a look at the broken superblock. Can you post the output of the commands:

# xfs_db -r -c "sb 16" -c p <dev>

and

# xfs_db -r -c "sb 16" -c "type data" -c p <dev>

so we can see the exact contents of that superblock?

FWIW, how many times has this filesystem ben grown?

Did it start
with only 32 AGs (i.e. 10TB in size)?

Quoted 39 lines · Dave Chinner

[ re-ccing the list, because finding this is in everyone's interest ]

Quoted 36 lines · Michael Maier

Quoted 8 lines · Eric Sandeen

Quoted 4 lines · Michael Maier

Hello!

I think I'm facing the same problem as already described here:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Maybe you can try the tracing Dave suggested in that thread? It certainly does look similar.

I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).

xfs_growfs /mnt

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=319815680, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=60160, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning data blocks changed from 319815680 to 346030080

The entry in messages was:

Aug 12 18:09:50 dualc kernel: [  257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00  XFSB............
Aug 12 18:09:50 dualc kernel: [  257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Aug 12 18:09:50 dualc kernel: [  257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1  F.....M.......f.
Aug 12 18:09:50 dualc kernel: [  257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80  ................
Aug 12 18:09:50 dualc kernel: [  257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file

/daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd

Quoted 7 lines

.....

Quoted 3 lines

Aug 12 18:09:50 dualc kernel: [  257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
Aug 12 18:09:50 dualc kernel: [  257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
Aug 12 18:09:50 dualc kernel: [  257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16

Ok, so that's reading the secondary superblock for AG 16. You're growing the filesystem from 42 to 45 AGs, so this problem is not

Quoted 10 lines

related to the actual grow operation - it's tripping over a problem that already exists on disk before the grow operation is started. i.e. this is likely to be a real corruption being seen, and it happened some time in the distant past and so we probably won't ever be able to pinpoint the cause of the problem.

That said, let's have a look at the broken superblock. Can you post the output of the commands:

# xfs_db -r -c "sb 16" -c p <dev>

done after the failed growfs mentioned above:

magicnum = 0x58465342
blocksize = 4096
dblocks = 319815680
rblocks = 0
rextents = 0
uuid = 4691c680-a9a9-4d8c-8fe2-18fde87f66e1 logstart = 67108868
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 7700480
agcount = 42
rbmblocks = 0
logblocks = 60160
versionnum = 0xb4a4
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000" blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 23
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 6464
ifree = 631
fdblocks = 1124026
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 1
features2 = 0xa
bad_features2 = 0xa

Quoted 4 lines

and

# xfs_db -r -c "sb 16" -c "type data" -c p <dev>

000: 58465342 00001000 00000000 13100000 00000000 00000000 00000000 00000000 020: 4691c680 a9a94d8c 8fe218fd e87f66e1 00000000 04000004 00000000 00000080 040: 00000000 00000081 00000000 00000082 00000001 00758000 0000002a 00000000 060: 0000eb00 b4a40200 01000010 00000000 00000000 00000000 0c090804 17000019 080: 00000000 00001940 00000000 00000277 00000000 001126ba 00000000 00000000 0a0: 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000 0c0: 00000000 00000001 0000000a 0000000a 8f980320 73987e9e db829704 ef73fe2e 0e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 100: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 120: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 140: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 160: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 180: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1a0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1c0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e

Quoted 4 lines

so we can see the exact contents of that superblock?

FWIW, how many times has this filesystem ben grown?

I can't say for sure, about 4 or 5 times?

Quoted 2 lines

Did it start
with only 32 AGs (i.e. 10TB in size)?

10TB? No. The device just has 3 TB. You most probably meant 10GB? I'm not sure, but it definitely started with > 100GB.

Quoted 40 lines · Michael Maier

Quoted 39 lines · Dave Chinner

[ re-ccing the list, because finding this is in everyone's interest ]

Quoted 36 lines · Michael Maier

Quoted 8 lines · Eric Sandeen

Quoted 4 lines · Michael Maier

Hello!

I think I'm facing the same problem as already described here:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Maybe you can try the tracing Dave suggested in that thread? It certainly does look similar.

I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).

xfs_growfs /mnt

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=319815680, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=60160, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning data blocks changed from 319815680 to 346030080

The entry in messages was:

Aug 12 18:09:50 dualc kernel: [  257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00  XFSB............
Aug 12 18:09:50 dualc kernel: [  257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Aug 12 18:09:50 dualc kernel: [  257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1  F.....M.......f.
Aug 12 18:09:50 dualc kernel: [  257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80  ................
Aug 12 18:09:50 dualc kernel: [  257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file

/daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd

Quoted 7 lines

Quoted 7 lines

.....

Quoted 3 lines

Aug 12 18:09:50 dualc kernel: [  257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
Aug 12 18:09:50 dualc kernel: [  257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
Aug 12 18:09:50 dualc kernel: [  257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16

Ok, so that's reading the secondary superblock for AG 16. You're growing the filesystem from 42 to 45 AGs, so this problem is not

The AGs are ~30GB. Going from 42 to 45 grows the XFS by 90GB. Michael, how much were you attempting to grow? Surely more than 90GB.

Quoted 86 lines

Quoted 10 lines

related to the actual grow operation - it's tripping over a problem that already exists on disk before the grow operation is started. i.e. this is likely to be a real corruption being seen, and it happened some time in the distant past and so we probably won't ever be able to pinpoint the cause of the problem.

That said, let's have a look at the broken superblock. Can you post the output of the commands:

# xfs_db -r -c "sb 16" -c p <dev>

done after the failed growfs mentioned above:

magicnum = 0x58465342
blocksize = 4096
dblocks = 319815680
rblocks = 0
rextents = 0
uuid = 4691c680-a9a9-4d8c-8fe2-18fde87f66e1 logstart = 67108868
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 7700480
agcount = 42
rbmblocks = 0
logblocks = 60160
versionnum = 0xb4a4
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000" blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 23
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 6464
ifree = 631
fdblocks = 1124026
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 1
features2 = 0xa
bad_features2 = 0xa

Quoted 4 lines

and

# xfs_db -r -c "sb 16" -c "type data" -c p <dev>

000: 58465342 00001000 00000000 13100000 00000000 00000000 00000000 00000000 020: 4691c680 a9a94d8c 8fe218fd e87f66e1 00000000 04000004 00000000 00000080 040: 00000000 00000081 00000000 00000082 00000001 00758000 0000002a 00000000 060: 0000eb00 b4a40200 01000010 00000000 00000000 00000000 0c090804 17000019 080: 00000000 00001940 00000000 00000277 00000000 001126ba 00000000 00000000 0a0: 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000 0c0: 00000000 00000001 0000000a 0000000a 8f980320 73987e9e db829704 ef73fe2e 0e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 100: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 120: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 140: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 160: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 180: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1a0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1c0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e

Quoted 4 lines

so we can see the exact contents of that superblock?

FWIW, how many times has this filesystem ben grown?

Quoted 7 lines

I can't say for sure, about 4 or 5 times?

Quoted 2 lines

Did it start
with only 32 AGs (i.e. 10TB in size)?

10TB? No. The device just has 3 TB. You most probably meant 10GB? I'm not sure, but it definitely started with > 100GB.

According to your xfs_info output that I highlighted above, and assuming my math here is correct,

(((7700480*4096)/1048576)*42)= 1,263,360 GB or ~1.23 TB

this filesystem was 1.23 TB w/42 AGs before the grow operation. Assuming defaults were used during mkfs.xfs it would appear the initial size of this filesystem was ~120GB. And it would appear it has been grown to ~10x its original size, and from 4 AGs to 42 AGs. That seems like a lot of growth, to me. And Dave states the latest grow operation was to 45 AGs, which would yield a ~1.32TB filesystem, not 3TB.

Quoted 3 lines · Stan Hoeppner

Quoted 2 lines · Michael Maier

Quoted 1 line · Dave Chinner

[ re-ccing the list, because finding this is in everyone's interest ]

Quoted 13 lines

Quoted 9 lines

Quoted 1 line

FWIW, how many times has this filesystem ben grown?

I can't say for sure, about 4 or 5 times?

Quoted 4 lines

Did it start
with only 32 AGs (i.e. 10TB in size)?

Quoted 1 line

Quoted 1 line

Quoted 1 line

10TB? No. The device just has 3 TB. You most probably meant 10GB?

I'm not sure, but it definitely started with > 100GB.

According to your xfs_info output that I highlighted above, and assuming my math here is correct,

Quoted 8 lines

(((7700480*4096)/1048576)*42)= 1,263,360 GB or ~1.23 TB

this filesystem was 1.23 TB w/42 AGs before the grow operation. Assuming defaults were used during mkfs.xfs it would appear the initial size of this filesystem was ~120GB. And it would appear it has been grown to ~10x its original size, and from 4 AGs to 42 AGs. That seems like a lot of growth, to me. And Dave states the latest grow operation was to 45 AGs, which would yield a ~1.32TB filesystem, not 3TB.

He wrote about 10TB (=Terra(!) Byte). The entire HD only has 3 TB. 10TB is impossible!

The last growfs was about 100GB. The FS was grown lots of times since the initial creation one or two years ago.

About the sizes before the last grow and after it: see
http://www.spinics.net/lists/xfs/msg21032.html

If you keep growing until you consume the disk, you'll have ~100 allocation groups. Typically you'd want to have no more than 4 AGs per spindle. You already have 42 (or 45) which will tend to seek the disk to death with many workloads, driving latency through the roof and decreasing throughput substantially. Do you notice any performance problems yet?

Or is this XFS strictly being used as a WORM like backup silo?

Quoted 6 lines · Stan Hoeppner

If you keep growing until you consume the disk, you'll have ~100 allocation groups. Typically you'd want to have no more than 4 AGs per spindle. You already have 42 (or 45) which will tend to seek the disk to death with many workloads, driving latency through the roof and decreasing throughput substantially. Do you notice any performance problems yet?

What are expected rates for copying e.g. a 10GB file? It's a Seagate Barracuda 3000GB Model ST3000DM001 SATA connected to SATA 6 Gb/s chip. The source and the destination FS is LUKS crypted. About 3 GB usable RAM (cache), AMD FX-8350 processor @ max. 3800MHz.

It's getting slower as more as the free space on the fs is reduced (beginning at about the last GB).

Resizing it makes the problem
disappear again.

Quoted 2 lines

Or is this XFS strictly being used as a WORM like backup silo?

yes

Quoted 8 lines · Michael Maier

Quoted 6 lines · Stan Hoeppner

If you keep growing until you consume the disk, you'll have ~100 allocation groups. Typically you'd want to have no more than 4 AGs per spindle. You already have 42 (or 45) which will tend to seek the disk to death with many workloads, driving latency through the roof and decreasing throughput substantially. Do you notice any performance problems yet?

Quoted 4 lines

What are expected rates for copying e.g. a 10GB file? It's a Seagate Barracuda 3000GB Model ST3000DM001 SATA connected to SATA 6 Gb/s chip. The source and the destination FS is LUKS crypted. About 3 GB usable RAM (cache), AMD FX-8350 processor @ max. 3800MHz.

Too many variables really to hazard a guess. If you put a gun to my head, I'd say strictly looking at the ingest rate of the Seagate, at a little less than half capacity, writing about 1/3rd of the way down the platters, optimum throughput should be 80-100 MB/s or so in the last 3 AGs, if free space isn't too heavily fragmented.

With so many smallish AGs, so many grow ops, and this backup workload, I'm curious as to what your free space map looks like. Would you mind posting the output of the following command, if you can?

$ xfs_db -r -c freesp <dev>

Quoted 48 lines · Dave Chinner

[ re-ccing the list, because finding this is in everyone's interest ]

Quoted 32 lines · Michael Maier

Quoted 8 lines · Eric Sandeen

Quoted 4 lines · Michael Maier

Hello!

I think I'm facing the same problem as already described here:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Maybe you can try the tracing Dave suggested in that thread? It certainly does look similar.

I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).

xfs_growfs /mnt
meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=319815680, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=60160, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning data blocks changed from 319815680 to 346030080

The entry in messages was:

Aug 12 18:09:50 dualc kernel: [  257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00  XFSB............
Aug 12 18:09:50 dualc kernel: [  257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Aug 12 18:09:50 dualc kernel: [  257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1  F.....M.......f.
Aug 12 18:09:50 dualc kernel: [  257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80  ................
Aug 12 18:09:50 dualc kernel: [  257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file

/daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd

.....

Quoted 3 lines

Aug 12 18:09:50 dualc kernel: [  257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
Aug 12 18:09:50 dualc kernel: [  257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
Aug 12 18:09:50 dualc kernel: [  257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16

Ok, so that's reading the secondary superblock for AG 16. You're growing the filesystem from 42 to 45 AGs, so this problem is not related to the actual grow operation - it's tripping over a problem that already exists on disk before the grow operation is started. i.e. this is likely to be a real corruption being seen, and it happened some time in the distant past and so we probably won't ever be able to pinpoint the cause of the problem.

Quoted 4 lines

That said, let's have a look at the broken superblock. Can you post the output of the commands:

# xfs_db -r -c "sb 16" -c p <dev>

done after the failed growfs mentioned above:

Quoted 3 lines

and

# xfs_db -r -c "sb 16" -c "type data" -c p <dev>

000: 58465342 00001000 00000000 13100000 00000000 00000000 00000000 00000000 020: 4691c680 a9a94d8c 8fe218fd e87f66e1 00000000 04000004 00000000 00000080 040: 00000000 00000081 00000000 00000082 00000001 00758000 0000002a 00000000 060: 0000eb00 b4a40200 01000010 00000000 00000000 00000000 0c090804 17000019 080: 00000000 00001940 00000000 00000277 00000000 001126ba 00000000 00000000 0a0: 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000 0c0: 00000000 00000001 0000000a 0000000a 8f980320 73987e9e db829704 ef73fe2e 0e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 100: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 120: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 140: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 160: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 180: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1a0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1c0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e

Quoted 3 lines

so we can see the exact contents of that superblock?

FWIW, how many times has this filesystem ben grown?

I can't say for sure, about 4 or 5 times?

Quoted 2 lines

Did it start
with only 32 AGs (i.e. 10TB in size)?

10TB? No. The device just has 3 TB. You most probably meant 10GB? I'm not sure, but it definitely started with > 100GB.

Quoted 49 lines · Michael Maier

Quoted 48 lines · Dave Chinner

[ re-ccing the list, because finding this is in everyone's interest ]

Quoted 32 lines · Michael Maier

Quoted 8 lines · Eric Sandeen

Quoted 4 lines · Michael Maier

Hello!

I think I'm facing the same problem as already described here:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Maybe you can try the tracing Dave suggested in that thread? It certainly does look similar.

I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).

xfs_growfs /mnt
meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=319815680, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=60160, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning data blocks changed from 319815680 to 346030080

The entry in messages was:

Aug 12 18:09:50 dualc kernel: [  257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00  XFSB............
Aug 12 18:09:50 dualc kernel: [  257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Aug 12 18:09:50 dualc kernel: [  257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1  F.....M.......f.
Aug 12 18:09:50 dualc kernel: [  257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80  ................
Aug 12 18:09:50 dualc kernel: [  257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file

/daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd

.....

Quoted 3 lines

Aug 12 18:09:50 dualc kernel: [  257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
Aug 12 18:09:50 dualc kernel: [  257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
Aug 12 18:09:50 dualc kernel: [  257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16

Ok, so that's reading the secondary superblock for AG 16. You're growing the filesystem from 42 to 45 AGs, so this problem is not related to the actual grow operation - it's tripping over a problem that already exists on disk before the grow operation is started. i.e. this is likely to be a real corruption being seen, and it happened some time in the distant past and so we probably won't ever be able to pinpoint the cause of the problem.

Quoted 6 lines

Quoted 4 lines

That said, let's have a look at the broken superblock. Can you post the output of the commands:

# xfs_db -r -c "sb 16" -c p <dev>

done after the failed growfs mentioned above:

Looks fine....

Quoted 4 lines

Quoted 3 lines

and

# xfs_db -r -c "sb 16" -c "type data" -c p <dev>

Quoted 16 lines

000: 58465342 00001000 00000000 13100000 00000000 00000000 00000000 00000000 020: 4691c680 a9a94d8c 8fe218fd e87f66e1 00000000 04000004 00000000 00000080 040: 00000000 00000081 00000000 00000082 00000001 00758000 0000002a 00000000 060: 0000eb00 b4a40200 01000010 00000000 00000000 00000000 0c090804 17000019 080: 00000000 00001940 00000000 00000277 00000000 001126ba 00000000 00000000 0a0: 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000 0c0: 00000000 00000001 0000000a 0000000a 8f980320 73987e9e db829704 ef73fe2e 0e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 100: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 120: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 140: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 160: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 180: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1a0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1c0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e

There's your problem - the empty space in the superblock is supposed to be zero. mkfs zeros it and we rely on it being zero for various reasons.

And one of those reasons is that we use the fact it shoul dbe zero to determine if we should be checking the CRC of the superblock. That is if there's a single bit error in the superblock and we are missing the correct bit in the version numbers that say CRCs are enabled, we use the fact that the superblock CRC field - which your filesystem knowns nothing about - should be zero to validate that the CRC feature bit is correctly set. The above superblock will indicate that there is a CRC set on the superblock, find the necessary version number is not correct, and so therefore we have a corruption in that superblock that the kernel code cannot handle without a user telling it what is correct.

So, the fact grwofs is failing is actually the correct behaviour for the filesystem to have in this case - the superblock is corrupt, just not obviously so.

Quoted 11 lines

Quoted 3 lines

so we can see the exact contents of that superblock?

FWIW, how many times has this filesystem ben grown?

I can't say for sure, about 4 or 5 times?

Quoted 2 lines

Did it start
with only 32 AGs (i.e. 10TB in size)?

10TB? No. The device just has 3 TB. You most probably meant 10GB? I'm not sure, but it definitely started with > 100GB.

I misplaced a digit A block size of 4096 bytes and:

    agcount=42, agsize=7700480 blks

So the filesystem size is 42 * 7700480 * 4096 = 1.26TB.

The question I'm asking is how many AGs did the filesystem start with, because this:

commit 1375cb65e87b327a8dd4f920c3e3d837fb40e9c2

Author: Dave Chinner <[email protected]>
Date:   Tue Oct 9 14:50:52 2012 +1100

xfs: growfs: don't read garbage for new secondary superblocks

When updating new secondary superblocks in a growfs operation, the superblock buffer is read from the newly grown region of the underlying device. This is not guaranteed to be zero, so violates the underlying assumption that the unused parts of superblocks are zero filled. Get a new buffer for these secondary superblocks to ensure that the unused regions are zero filled correctly.

Signed-off-by Dave Chinner <[email protected]>

Reviewed-by Carlos Maiolino <[email protected]>

Signed-off-by Ben Myers <[email protected]>

Is the only possible reason I can think of that would result in non-zero empty space in a secondary superblock. And that implies that the filesystem started with 16 AGs or less,

and was grown with
an older kernel with this bug in it.

If it makes you feel any better, the bug that caused this had been in the code for 15+ years and you are the first person I know of to have ever hit it....

xfs_repair doesn't appear to have any checks in it to detect this situation or repair it - there are some conditions for zeroing the unused parts of a superblock, but they are focussed around detecting and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15 years ago.

Quoted 134 lines · Dave Chinner

Quoted 49 lines · Michael Maier

Quoted 48 lines · Dave Chinner

[ re-ccing the list, because finding this is in everyone's interest ]

Quoted 32 lines · Michael Maier

Quoted 8 lines · Eric Sandeen

Quoted 4 lines · Michael Maier

Hello!

I think I'm facing the same problem as already described here:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Maybe you can try the tracing Dave suggested in that thread? It certainly does look similar.

I attached a trace report while executing xfs_growfs /mnt on linux 3.10.5 (does not happen with 3.9.8).

xfs_growfs /mnt
meta-data=/dev/mapper/backupMy-daten3 isize=256 agcount=42, agsize=7700480 blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=319815680, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=60160, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning data blocks changed from 319815680 to 346030080

The entry in messages was:

Aug 12 18:09:50 dualc kernel: [  257.368030] ffff8801e8dbd400: 58 46 53 42 00 00 10 00 00 00 00 00 13 10 00 00  XFSB............
Aug 12 18:09:50 dualc kernel: [  257.368037] ffff8801e8dbd410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Aug 12 18:09:50 dualc kernel: [  257.368042] ffff8801e8dbd420: 46 91 c6 80 a9 a9 4d 8c 8f e2 18 fd e8 7f 66 e1  F.....M.......f.
Aug 12 18:09:50 dualc kernel: [  257.368045] ffff8801e8dbd430: 00 00 00 00 04 00 00 04 00 00 00 00 00 00 00 80  ................
Aug 12 18:09:50 dualc kernel: [  257.368051] XFS (dm-33): Internal error xfs_sb_read_verify at line 730 of file

/daten2/tmp/rpm/BUILD/kernel-desktop-3.10.5/linux-3.10/fs/xfs/xfs_mount.c. Caller 0xffffffffa099a2fd

.....

Quoted 3 lines

Aug 12 18:09:50 dualc kernel: [  257.368533] XFS (dm-33): Corruption detected. Unmount and run xfs_repair
Aug 12 18:09:50 dualc kernel: [  257.368611] XFS (dm-33): metadata I/O error: block 0x3ac00000 ("xfs_trans_read_buf_map") error 117 numblks 1
Aug 12 18:09:50 dualc kernel: [  257.368623] XFS (dm-33): error 117 reading secondary superblock for ag 16

Ok, so that's reading the secondary superblock for AG 16. You're growing the filesystem from 42 to 45 AGs, so this problem is not related to the actual grow operation - it's tripping over a problem that already exists on disk before the grow operation is started. i.e. this is likely to be a real corruption being seen, and it happened some time in the distant past and so we probably won't ever be able to pinpoint the cause of the problem.

Quoted 6 lines

Quoted 4 lines

That said, let's have a look at the broken superblock. Can you post the output of the commands:

# xfs_db -r -c "sb 16" -c p <dev>

done after the failed growfs mentioned above:

Looks fine....

Quoted 4 lines

Quoted 3 lines

and

# xfs_db -r -c "sb 16" -c "type data" -c p <dev>

Quoted 16 lines

000: 58465342 00001000 00000000 13100000 00000000 00000000 00000000 00000000 020: 4691c680 a9a94d8c 8fe218fd e87f66e1 00000000 04000004 00000000 00000080 040: 00000000 00000081 00000000 00000082 00000001 00758000 0000002a 00000000 060: 0000eb00 b4a40200 01000010 00000000 00000000 00000000 0c090804 17000019 080: 00000000 00001940 00000000 00000277 00000000 001126ba 00000000 00000000 0a0: 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000 0c0: 00000000 00000001 0000000a 0000000a 8f980320 73987e9e db829704 ef73fe2e 0e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 100: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 120: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 140: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 160: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 180: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1a0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1c0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e 1e0: 8f980320 73987e9e db829704 ef73fe2e 8f980320 73987e9e db829704 ef73fe2e

There's your problem - the empty space in the superblock is supposed to be zero. mkfs zeros it and we rely on it being zero for various reasons.

And one of those reasons is that we use the fact it shoul dbe zero to determine if we should be checking the CRC of the superblock. That is if there's a single bit error in the superblock and we are missing the correct bit in the version numbers that say CRCs are enabled, we use the fact that the superblock CRC field - which your filesystem knowns nothing about - should be zero to validate that the CRC feature bit is correctly set. The above superblock will indicate that there is a CRC set on the superblock, find the necessary version number is not correct, and so therefore we have a corruption in that superblock that the kernel code cannot handle without a user telling it what is correct.

So, the fact grwofs is failing is actually the correct behaviour for the filesystem to have in this case - the superblock is corrupt, just not obviously so.

Quoted 11 lines

Quoted 3 lines

so we can see the exact contents of that superblock?

FWIW, how many times has this filesystem ben grown?

I can't say for sure, about 4 or 5 times?

Quoted 2 lines

Did it start
with only 32 AGs (i.e. 10TB in size)?

10TB? No. The device just has 3 TB. You most probably meant 10GB? I'm not sure, but it definitely started with > 100GB.

I misplaced a digit A block size of 4096 bytes and:

    agcount=42, agsize=7700480 blks

So the filesystem size is 42 * 7700480 * 4096 = 1.26TB.

The question I'm asking is how many AGs did the filesystem start with, because this:

commit 1375cb65e87b327a8dd4f920c3e3d837fb40e9c2

Author: Dave Chinner <[email protected]>
Date:   Tue Oct 9 14:50:52 2012 +1100

xfs: growfs: don't read garbage for new secondary superblocks

When updating new secondary superblocks in a growfs operation, the superblock buffer is read from the newly grown region of the underlying device. This is not guaranteed to be zero, so violates the underlying assumption that the unused parts of superblocks are zero filled. Get a new buffer for these secondary superblocks to ensure that the unused regions are zero filled correctly.

Quoted 7 lines

Signed-off-by Dave Chinner <[email protected]>

Reviewed-by Carlos Maiolino <[email protected]>

Signed-off-by Ben Myers <[email protected]>

Is the only possible reason I can think of that would result in non-zero empty space in a secondary superblock. And that implies that the filesystem started with 16 AGs or less,

yes

Quoted 2 lines

and was grown with
an older kernel with this bug in it.

yes.

Quoted 3 lines

If it makes you feel any better, the bug that caused this had been in the code for 15+ years and you are the first person I know of to have ever hit it....

Probably the second one :-) See
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Quoted 5 lines

xfs_repair doesn't appear to have any checks in it to detect this situation or repair it - there are some conditions for zeroing the unused parts of a superblock, but they are focussed around detecting and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15 years ago.

The _big problem_ is: xfs_repair not just doesn't repair it, but it _causes data loss_ in some situations!

Given the following situation I ran in:
- xfs_growfs started running linux 3.10.5.

- Saw the error message on the konsole:
XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning data blocks changed from 319815680 to 346030080

- Checked with df -> The growing seems to be done. Decision: Analyse the problem later when there is more time.

- Some days later, entry found in messages:
"Corruption detected. Unmount and run xfs_repair"

- I did it as suggested.
Result: FS has the original size again before growing the FS and
complete loss of all data written since this faulty growing. And: FS isn't repaired.
If it is not a problem at all (that's how I understood you here), why is there a error message and the suggest to run xfs_repair, which obviously isn't able at all to repair this problem but leads directly to data loss?

Thanks for your clarification. I hope other people read this thread before they are loosing data :-(.

What to do now?
- Don't use >= 3.10.x kernel. Or:

- Ignore it (how can I distinguish this case from other cases?) Or:
- Recreate the complete FS.

Thanks for clarification,
regards,
Michael.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Dave Chinner wrote:

Quoted 3 lines

If it makes you feel any better, the bug that caused this had been in the code for 15+ years and you are the first person I know of to have ever hit it....

Probably the second one :-) See
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Quoted 5 lines

xfs_repair doesn't appear to have any checks in it to detect this situation or repair it - there are some conditions for zeroing the unused parts of a superblock, but they are focussed around detecting and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15 years ago.

The _big problem_ is: xfs_repair not just doesn't repair it, but it _causes data loss_ in some situations!

Quoted 1 line · Michael Maier

Dave Chinner wrote:

...

Quoted 13 lines

Quoted 3 lines

If it makes you feel any better, the bug that caused this had been in the code for 15+ years and you are the first person I know of to have ever hit it....

Probably the second one :-) See
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Quoted 5 lines

xfs_repair doesn't appear to have any checks in it to detect this situation or repair it - there are some conditions for zeroing the unused parts of a superblock, but they are focussed around detecting and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15 years ago.

Quoted 3 lines

The _big problem_ is: xfs_repair not just doesn't repair it, but it _causes data loss_ in some situations!

So as far as I can tell at this point, a few things have happened to result in this unfortunate situation. Congratulations, you hit a perfect storm. :(

1) prior resize operations populated unused portions of backup sbs w/ junk
2) newer kernels fail to verify superblocks in this state
3) during your growfs under 3.10, that verification failure aborted backup superblock updates, leaving many unmodified
4a) xfs_repair doesn't find or fix the junk in the backup sbs, and 4b) when running, it looks for the superblocks which are "most matching"
other superblocks on the disk, and takes that version as correct.

So you had 16 superblocks (0-15) which were correct after the growfs. But 16 didn't verify and was aborted, so nothing was updated after that. This means that 16 onward have the wrong number of AGs and disk blocks; i.e. they are the pre-growfs size, and there are 26 of them.

Today, xfs_repair sees this 26-to-16 vote, and decides that the 26 matching superblocks "win," rewrites the first superblock with this geometry, and uses that to verify the rest of the filesytem. Hence anything post-growfs looks out of bounds, and gets nuked.

So right now, I'm thinking that the "proper geometry" heuristic should be adjusted, but how to do that in general, I'm not sure. Weighting sb 0 heavily, especially if it matches many subsequent superblocks, seems somewhat reasonable.

Quoted 18 lines · Eric Sandeen

Quoted 1 line · Michael Maier

Dave Chinner wrote:

...

Quoted 13 lines

Quoted 3 lines

If it makes you feel any better, the bug that caused this had been in the code for 15+ years and you are the first person I know of to have ever hit it....

Probably the second one :-) See
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Quoted 5 lines

xfs_repair doesn't appear to have any checks in it to detect this situation or repair it - there are some conditions for zeroing the unused parts of a superblock, but they are focussed around detecting and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15 years ago.

Quoted 7 lines

Quoted 3 lines

The _big problem_ is: xfs_repair not just doesn't repair it, but it _causes data loss_ in some situations!

So as far as I can tell at this point, a few things have happened to result in this unfortunate situation. Congratulations, you hit a perfect storm. :(

I can appease you - as it "only" hit my backup device and because I noticed the problem before I really needed it: I didn't hit any data loss over all, because the original data is ok and I repeated the backup w/ the fixed FS now!

Quoted 18 lines

1) prior resize operations populated unused portions of backup sbs w/ junk
2) newer kernels fail to verify superblocks in this state
3) during your growfs under 3.10, that verification failure aborted backup superblock updates, leaving many unmodified
4a) xfs_repair doesn't find or fix the junk in the backup sbs, and 4b) when running, it looks for the superblocks which are "most matching"
other superblocks on the disk, and takes that version as correct.

So you had 16 superblocks (0-15) which were correct after the growfs. But 16 didn't verify and was aborted, so nothing was updated after that. This means that 16 onward have the wrong number of AGs and disk blocks; i.e. they are the pre-growfs size, and there are 26 of them.

Today, xfs_repair sees this 26-to-16 vote, and decides that the 26 matching superblocks "win," rewrites the first superblock with this geometry, and uses that to verify the rest of the filesytem. Hence anything post-growfs looks out of bounds, and gets nuked.

Quoted 4 lines

So right now, I'm thinking that the "proper geometry" heuristic should be adjusted, but how to do that in general, I'm not sure. Weighting sb 0 heavily, especially if it matches many subsequent superblocks, seems somewhat reasonable.

This would have been my next question! I repaired it w/ the git xfs_repair on the already reduced to original size FS. I think, if I would have done the same w/ the grown FS, the FS most probably would be reduced to the size before the growing.

Wouldn't it be better to not grow at all if there are problems detected?
Means: Don't do the check after the growing, but before? Ok, I could have done it myself ... . From now on, I will do it like this!

Quoted 60 lines · Michael Maier

Quoted 18 lines · Eric Sandeen

Quoted 1 line · Michael Maier

Dave Chinner wrote:

...

Quoted 13 lines

Quoted 3 lines

If it makes you feel any better, the bug that caused this had been in the code for 15+ years and you are the first person I know of to have ever hit it....

Probably the second one :-) See
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428

Quoted 5 lines

xfs_repair doesn't appear to have any checks in it to detect this situation or repair it - there are some conditions for zeroing the unused parts of a superblock, but they are focussed around detecting and correcting damage caused by a buggy Irix 6.5-beta mkfs from 15 years ago.

Quoted 7 lines

Quoted 3 lines

The _big problem_ is: xfs_repair not just doesn't repair it, but it _causes data loss_ in some situations!

So as far as I can tell at this point, a few things have happened to result in this unfortunate situation. Congratulations, you hit a perfect storm. :(

I can appease you - as it "only" hit my backup device and because I noticed the problem before I really needed it: I didn't hit any data loss over all, because the original data is ok and I repeated the backup w/ the fixed FS now!

Quoted 18 lines

1) prior resize operations populated unused portions of backup sbs w/ junk
2) newer kernels fail to verify superblocks in this state
3) during your growfs under 3.10, that verification failure aborted backup superblock updates, leaving many unmodified
4a) xfs_repair doesn't find or fix the junk in the backup sbs, and 4b) when running, it looks for the superblocks which are "most matching"
other superblocks on the disk, and takes that version as correct.

So you had 16 superblocks (0-15) which were correct after the growfs. But 16 didn't verify and was aborted, so nothing was updated after that. This means that 16 onward have the wrong number of AGs and disk blocks; i.e. they are the pre-growfs size, and there are 26 of them.

Today, xfs_repair sees this 26-to-16 vote, and decides that the 26 matching superblocks "win," rewrites the first superblock with this geometry, and uses that to verify the rest of the filesytem. Hence anything post-growfs looks out of bounds, and gets nuked.

Quoted 4 lines

So right now, I'm thinking that the "proper geometry" heuristic should be adjusted, but how to do that in general, I'm not sure. Weighting sb 0 heavily, especially if it matches many subsequent superblocks, seems somewhat reasonable.

This would have been my next question! I repaired it w/ the git xfs_repair on the already reduced to original size FS. I think, if I would have done the same w/ the grown FS, the FS most probably would be reduced to the size before the growing.

Quoted 3 lines

Wouldn't it be better to not grow at all if there are problems detected?
Means: Don't do the check after the growing, but before? Ok, I could have done it myself ... . From now on, I will do it like this!

well, see the next couple patches I'm about to send to the list ... ;)

but a check prior wouldn't have helped you, because repair didn't detect the problem that growfs choked on.

Quoted 2 lines

but a check prior wouldn't have helped you, because repair didn't detect

Quoted 1 line

the problem that growfs choked on.

The old xfs_repair! Your patched one would have detected the problem if I got it right.

But globally speaking: you're right - it's impossible to get 100% security. But couldn't xfs_repair -n find other problems which therefore could be repaired before growing the FS?

Hence looks like we'll need some new xfs_repair functionality to fix this. It might take me a little while to get you a fix - perhaps someone else with a little bit of spare time could get it done sooner than I can. Anyone?