ext4 > 16T woes
I'll try to look into this but just wanted to put it out there.
After creating a 32T fs and filling it with 1T preallocated files, I get quite a lot of corruption; this wasn't really the test I wanted to do, I wanted to then remove a file with mostly high blocks, run fsstress, and test recovery :(
# uname -r 3.4.0-rc4+ # truncate --size 32t imagefile2 # mkfs.ext4 -F imagefile2 mke2fs 1.42.3 (14-May-2012) ... # mount -o loop imagefile2 /mnt/scratch # for I in `seq 1 32`; do fallocate -l 1t /mnt/scratch/1tfile$I; done # umount /mnt/scratch # time e2fsck -fn imagefile2 e2fsck 1.42.3 (14-May-2012) <many many many lines snipped> Free blocks count wrong for group #131072 (0, counted=32768). Fix? no
Free blocks count wrong for group #131073 (0, counted=32768). Fix? no
Free blocks count wrong for group #131074 (0, counted=4164). Fix? no
Free blocks count wrong (3, counted=4294967299).
Fix? no
imagefile2: ********** WARNING: Filesystem still has errors **********
imagefile2: 43/536870912 files (0.0% non-contiguous), 8589934589/8589934592 blocks
real 27m40.133s
user 27m12.886s
sys 0m6.238s
Quoted 6 lines · Eric Sandeen
I'll try to look into this but just wanted to put it out there.
After creating a 32T fs and filling it with 1T preallocated files, I get quite a lot of corruption; this wasn't really the test I wanted to do, I wanted to then remove a file with mostly high blocks, run fsstress, and test recovery :(
Hmm, we've done full-filesystem write + read +verify + e2fsck up to 128TB (took several days to write and read so much data) with RHEL6 without similar problems (up to 32TB on RHEL5) using real disks.
We don't use fallocate() or loop devices, so it may be that some problems are therein, or possibly with newer kernels.
Presumably this is using XFS for the backing loop file? I don't anticipate problems with that, but there may be strange interactions between fallocate(), sparse files, and TRIM/DISCARD?
Quoted 9 lines
# uname -r 3.4.0-rc4+ # truncate --size 32t imagefile2 # mkfs.ext4 -F imagefile2 mke2fs 1.42.3 (14-May-2012) ... # mount -o loop imagefile2 /mnt/scratch # for I in `seq 1 32`; do fallocate -l 1t /mnt/scratch/1tfile$I; done # umount /mnt/scratch
Did you try sync here, or running e2fsck on the loop device instead of the backing file? It looks like nothing was written to disk...
Quoted 20 lines
# time e2fsck -fn imagefile2 e2fsck 1.42.3 (14-May-2012) <many many many lines snipped> Free blocks count wrong for group #131072 (0, counted=32768). Fix? noFree blocks count wrong for group #131073 (0, counted=32768). Fix? no
Free blocks count wrong for group #131074 (0, counted=4164). Fix? no
Free blocks count wrong (3, counted=4294967299).
Fix? noimagefile2: ********** WARNING: Filesystem still has errors **********
imagefile2: 43/536870912 files (0.0% non-contiguous), 8589934589/8589934592 blocks
Quoted 8 lines
real 27m40.133s
user 27m12.886s
sys 0m6.238s--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Cheers, Andreas
Quoted 12 lines · Andreas Dilger
Quoted 6 lines · Eric Sandeen
I'll try to look into this but just wanted to put it out there.
After creating a 32T fs and filling it with 1T preallocated files, I get quite a lot of corruption; this wasn't really the test I wanted to do, I wanted to then remove a file with mostly high blocks, run fsstress, and test recovery :(
Hmm, we've done full-filesystem write + read +verify + e2fsck up to 128TB (took several days to write and read so much data) with RHEL6 without similar problems (up to 32TB on RHEL5) using real disks.
Quoted 6 lines
We don't use
fallocate()or loop devices, so it may be that some problems are therein, or possibly with newer kernels.Presumably this is using XFS for the backing loop file? I don't anticipate problems with that, but there may be strange interactions between
fallocate(), sparse files, and TRIM/DISCARD?
It is backed by XFS, yes. I don't think I'm freeing any blocks during the test, so I wouldn't expect discard to come into it. Really, all it should be doing is writing fairly small amounts of metadata, right?
I'll keep investigating.
-Eric
Quoted 49 lines
Quoted 9 lines
# uname -r 3.4.0-rc4+ # truncate --size 32t imagefile2 # mkfs.ext4 -F imagefile2 mke2fs 1.42.3 (14-May-2012) ... # mount -o loop imagefile2 /mnt/scratch # for I in `seq 1 32`; do fallocate -l 1t /mnt/scratch/1tfile$I; done # umount /mnt/scratchDid you try sync here, or running e2fsck on the loop device instead of the backing file? It looks like nothing was written to disk...
Quoted 20 lines
# time e2fsck -fn imagefile2 e2fsck 1.42.3 (14-May-2012) <many many many lines snipped> Free blocks count wrong for group #131072 (0, counted=32768). Fix? noFree blocks count wrong for group #131073 (0, counted=32768). Fix? no
Free blocks count wrong for group #131074 (0, counted=4164). Fix? no
Free blocks count wrong (3, counted=4294967299).
Fix? noimagefile2: ********** WARNING: Filesystem still has errors **********
imagefile2: 43/536870912 files (0.0% non-contiguous), 8589934589/8589934592 blocks
Quoted 8 lines
real 27m40.133s
user 27m12.886s
sys 0m6.238s--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlCheers, Andreas
Quoted 6 lines · Eric Sandeen
I'll try to look into this but just wanted to put it out there.
After creating a 32T fs and filling it with 1T preallocated files, I get quite a lot of corruption; this wasn't really the test I wanted to do, I wanted to then remove a file with mostly high blocks, run fsstress, and test recovery :(
FWIW I tracked this down to the rbtree bitmap backend and sent a patch,
[PATCH] libext2fs: Fix rbtree backend for extent lengths greater than 2^32
-Eric
Quoted 37 lines
# uname -r 3.4.0-rc4+ # truncate --size 32t imagefile2 # mkfs.ext4 -F imagefile2 mke2fs 1.42.3 (14-May-2012) ... # mount -o loop imagefile2 /mnt/scratch # for I in `seq 1 32`; do fallocate -l 1t /mnt/scratch/1tfile$I; done # umount /mnt/scratch # time e2fsck -fn imagefile2 e2fsck 1.42.3 (14-May-2012) <many many many lines snipped> Free blocks count wrong for group #131072 (0, counted=32768). Fix? noFree blocks count wrong for group #131073 (0, counted=32768). Fix? no
Free blocks count wrong for group #131074 (0, counted=4164). Fix? no
Free blocks count wrong (3, counted=4294967299).
Fix? noimagefile2: ********** WARNING: Filesystem still has errors **********
imagefile2: 43/536870912 files (0.0% non-contiguous), 8589934589/8589934592 blocks
real 27m40.133s
user 27m12.886s
sys 0m6.238s--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html