Where all does preallocated/extra space hide?
I was running some of the xfstests enospc tests on ext4, and they were failing; in one case, manymanymany small files are made to fill up a 100M filesystem. ext4 stops quite early with -ENOSPC, but after a bit, (or after a "sync") we get 40MB free again. So 40% of the fs space is hidden somewhere in preallocation...
I tried calling out to discard group prealloc but that's only a few blocks. I'll go trace through the sync paths to see what all gets released, but if anyone knows offhand where the rest of that space is hiding, please give me a shout. :)
Thanks,
-Eric
Quoted 6 lines · Eric Sandeen
I was running some of the xfstests enospc tests on ext4, and they were failing; in one case, manymanymany small files are made to fill up a 100M filesystem. ext4 stops quite early with
-ENOSPC, but after a bit, (or after a "sync") we get 40MB free again. So 40% of the fs space is hidden somewhere in preallocation...
Quoted 7 lines
I tried calling out to discard group prealloc but that's only a few blocks. I'll go trace through the sync paths to see what all gets released, but if anyone knows offhand where the rest of that space is hiding, please give me a shout. :)
Thanks,
-Eric
Possibly related; on a 1G filesystem, doing this:
#!/bin/bash
xfs_io -F -f -d -c 'pwrite -b 64k 0 512m' /mnt/test/io_test rm /mnt/test/io_test
in a "while true" loop spews ENOSPC. (it's a direct IO 512m write in 64k chunks). Buffered IO seems fine ...
-Eric
Quoted 11 lines · Eric Sandeen
I was running some of the xfstests enospc tests on ext4, and they were failing; in one case, manymanymany small files are made to fill up a 100M filesystem. ext4 stops quite early with
-ENOSPC, but after a bit, (or after a "sync") we get 40MB free again. So 40% of the fs space is hidden somewhere in preallocation...I tried calling out to discard group prealloc but that's only a few blocks. I'll go trace through the sync paths to see what all gets released, but if anyone knows offhand where the rest of that space is hiding, please give me a shout. :)
preallocation space is discarded by default if we fail a block allocation ext4_mb_discard_preallocations does that. What might be happening is the extra meta data blocks that we reserve for making sure we will be able to properly insert the new extent on block allocation. I guess we should force a data allocation when we fail with ENOSPC in ext4_da_writepages We currently force a journal commit so that the we claim back the blocks from deleted files. But we can also force block allocation for delayed allocated inodes so that we free some of the extra meta data we reserved
-aneesh
Quoted 23 lines · Aneesh Kumar K.V
Quoted 11 lines · Eric Sandeen
I was running some of the xfstests enospc tests on ext4, and they were failing; in one case, manymanymany small files are made to fill up a 100M filesystem. ext4 stops quite early with
-ENOSPC, but after a bit, (or after a "sync") we get 40MB free again. So 40% of the fs space is hidden somewhere in preallocation...I tried calling out to discard group prealloc but that's only a few blocks. I'll go trace through the sync paths to see what all gets released, but if anyone knows offhand where the rest of that space is hiding, please give me a shout. :)
preallocation space is discarded by default if we fail a block allocation
ext4_mb_discard_preallocationsdoes that. What might be happening is the extra meta data blocks that we reserve for making sure we will be able to properly insert the new extent on block allocation. I guess we should force a data allocation when we fail with ENOSPC inext4_da_writepagesWe currently force a journal commit so that the we claim back the blocks from deleted files. But we can also force block allocation for delayed allocated inodes so that we free some of the extra meta data we reserved
Quoted 1 line
-aneesh
Yep, I should have followed up, I narrowed it down to just that - the worst-case metadata blocks - 2 metadata blocks for a 20-byte write into an empty file. :)
I'm working on an inode walker to push out delalloc files on enospc.
Thanks,
-Eric
Quoted 24 lines · Eric Sandeen
Quoted 23 lines · Aneesh Kumar K.V
Quoted 11 lines · Eric Sandeen
I was running some of the xfstests enospc tests on ext4, and they were failing; in one case, manymanymany small files are made to fill up a 100M filesystem. ext4 stops quite early with
-ENOSPC, but after a bit, (or after a "sync") we get 40MB free again. So 40% of the fs space is hidden somewhere in preallocation...I tried calling out to discard group prealloc but that's only a few blocks. I'll go trace through the sync paths to see what all gets released, but if anyone knows offhand where the rest of that space is hiding, please give me a shout. :)
preallocation space is discarded by default if we fail a block allocation
ext4_mb_discard_preallocationsdoes that. What might be happening is the extra meta data blocks that we reserve for making sure we will be able to properly insert the new extent on block allocation. I guess we should force a data allocation when we fail with ENOSPC inext4_da_writepagesWe currently force a journal commit so that the we claim back the blocks from deleted files. But we can also force block allocation for delayed allocated inodes so that we free some of the extra meta data we reserved
Quoted 7 lines
Quoted 1 line
-aneesh
Yep, I should have followed up, I narrowed it down to just that - the worst-case metadata blocks - 2 metadata blocks for a 20-byte write into an empty file. :)
I'm working on an inode walker to push out delalloc files on enospc.
Should we do an inode walker ? I guess we should be doing something similar to balance_dirty_pages. That will kick in the flusher threads which inturn will force the block allocation of dirty inodes.
-aneesh
Quoted 29 lines · Aneesh Kumar K.V
Quoted 28 lines · Eric Sandeen
Quoted 22 lines · Aneesh Kumar K.V
Quoted 11 lines · Eric Sandeen
I was running some of the xfstests enospc tests on ext4, and they were failing; in one case, manymanymany small files are made to fill up a 100M filesystem. ext4 stops quite early with
-ENOSPC, but after a bit, (or after a "sync") we get 40MB free again. So 40% of the fs space is hidden somewhere in preallocation...I tried calling out to discard group prealloc but that's only a few blocks. I'll go trace through the sync paths to see what all gets released, but if anyone knows offhand where the rest of that space is hiding, please give me a shout. :)
preallocation space is discarded by default if we fail a block allocation
ext4_mb_discard_preallocationsdoes that. What might be happening is the extra meta data blocks that we reserve for making sure we will be able to properly insert the new extent on block allocation. I guess we should force a data allocation when we fail with ENOSPC inext4_da_writepagesWe currently force a journal commit so that the we claim back the blocks from deleted files. But we can also force block allocation for delayed allocated inodes so that we free some of the extra meta data we reservedQuoted 1 line
-aneesh
Yep, I should have followed up, I narrowed it down to just that - the worst-case metadata blocks - 2 metadata blocks for a 20-byte write into an empty file. :)
Quoted 8 lines
Quoted 1 line
I'm working on an inode walker to push out delalloc files on enospc.
Should we do an inode walker ? I guess we should be doing something similar to
balance_dirty_pages. That will kick in the flusher threads which inturn will force the block allocation of dirty inodes.-aneesh
it'll need to be synchronous to avoid a spurious enospc I think ... plus, we only want to flush delalloc inodes; flushing everythign would be needlessly expensive I think... need to think about the right way to do this.
-Eric