Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)
Hi everybody,
I'm trying to create a multiboot medium using Easy2Boot on a 64 GB USB key. Easy2Boot [1] is a "software" (well a collection of files really) that relies on grub4dos [2] to work and thus requires payload files to be strictly contiguous to do its trick. It's supposed to be set up once and then one just has to add files to boot from to a folder to modify the multiboot setup.
The ISO files I have are between 230 MB and 2 GB each. I divided my USB drive into two partitions : first one, 16 GB ntfs formatted so that grub4dos actually works (for some reasons it doesn't with ext4, plus I want to be able to use it with other OS); second one 48 GB formatted as ext4 with e2fsprogs version 1.42.9 from Archlinux. However when I try to boot from those files I systematically get the "file not contiguous" error in Easy2Boot.
Question is: is it possible to make huge files strictly contiguous on ex4, and if yes how?
*******
Details
*******
I tried different mkfs.ext4 options so as to cater to my use case, but to no avail. I tried pushing the inode_ratio to its maximum value (64 MB) and using fallocate besides regular copy:
# mkfs.ext4 -m 0 -L iso -i 67108864 -E root_owner=1000:100 /dev/sdc2 # fallocate -l 1235451904 /path/to/mounted/usb/folder/debian-live-7.2-amd64-gnome-desktop_fal.iso # cp debian-live-7.2-amd64-gnome-desktop.iso /path/to/folder/debian-live-7.2-amd64-gnome-desktop_fal.iso
But files aren't contiguous. e4defrag seems to confirm it:
# e4defrag -c '/run/media/neitsab/iso/_ISO'
<Fragmented files> now/best size/ext
1. /run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso
13/1 119378 KB
2. /run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64_fal.iso
13/1 119378 KBTotal/best extents 26/2
Average size per extent 119378 KB
Fragmentation score 0
[0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag] This directory (/run/media/neitsab/iso/_ISO) does not need defragmentation. Done.
Then I tried to use bigalloc and clusters:
# mkfs.ext4 -m 0 -L iso -i 67108864 -O bigalloc -C 256M -E root_owner=1000:100 /dev/sdc2 mke2fs 1.42.9 (28-Dec-2013)
Warning: the bigalloc feature is still under development
See https://ext4.wiki.kernel.org/index.php/Bigalloc for more information
warning: 53248 blocks unused.
Filesystem label=iso
OS type: Linux
Block size=4096 (log=2)
Cluster size=268435456 (log=18)
Stride=0 blocks, Stripe width=0 blocks
720 inodes, 11665408 blocks
0 blocks (0.00%) reserved for the super user First data block=0
Root directory owner=1000:100
1 block group
2146959360 blocks per group, 32760 clusters per group 720 inodes per group
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
It took it some lengthy time to allocate the group tables, then regular copy worked but fallocate hanged (process uninterruptible) and I had to reboot the machine, which led me to a kernel panic mentionning fallocate. The file copied via Nautilus still wasn't contiguous enough for Easy2Boot.
So what's left? Is it at all possible to make completely contiguous files on ext4? Not even on a fresh FS with plenty of free space?
Thanks for your comments!
[1] http://www.easy2boot.com/introduction/
[2] https://code.google.com/p/grub4dos-chenall/
Quoted 31 lines · Contact
Date: Thu, 17 Apr 2014 04:18:30 +0200
From: Contact <[email protected]>
To: "[email protected]" <[email protected]>
Subject: Make huge files strictly contiguous (fallocate, bigalloc,e4defrag...)Hi everybody,
I'm trying to create a multiboot medium using Easy2Boot on a 64 GB USB key. Easy2Boot [1] is a "software" (well a collection of files really) that relies on grub4dos [2] to work and thus requires payload files to be strictly contiguous to do its trick. It's supposed to be set up once and then one just has to add files to boot from to a folder to modify the multiboot setup.
The ISO files I have are between 230 MB and 2 GB each. I divided my USB drive into two partitions : first one, 16 GB ntfs formatted so that grub4dos actually works (for some reasons it doesn't with ext4, plus I want to be able to use it with other OS); second one 48 GB formatted as ext4 with e2fsprogs version 1.42.9 from Archlinux. However when I try to boot from those files I systematically get the "file not contiguous" error in Easy2Boot.
Question is: is it possible to make huge files strictly contiguous on ex4, and if yes how?
*******
Details
*******
Quoted 8 lines
I tried different mkfs.ext4 options so as to cater to my use case, but to no avail. I tried pushing the
inode_ratioto its maximum value (64 MB) and using fallocate besides regular copy:# mkfs.ext4 -m 0 -L iso -i 67108864 -E root_owner=1000:100 /dev/sdc2 # fallocate -l 1235451904 /path/to/mounted/usb/folder/debian-live-7.2-amd64-gnome-desktop_fal.iso # cp debian-live-7.2-amd64-gnome-desktop.iso /path/to/folder/debian-live-7.2-amd64-gnome-desktop_fal.iso
This is not how it it supposed to be used. Yes fallocate preallocates the file, but cp will truncate it so fallocate will certainly not help you in any way. In order for fallocate to be useful you'll have to write into the file without actually truncating it (dd can do this if you do not want to write your own program)
Also the file is probably as contiguous as it could be. Here is what I get on the file system with default mkfs options.
# e4defrag -c /mnt/test/file1 <File> now/best size/ext /mnt/test/file1 10/1 120649 KB
But that does not tell the whole story. See
xfs_io -f -c "fiemap -v" /mnt/test/file1/mnt/test/file1:
EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS 0: [0..262143]: 2768896..3031039 262144 0x0 1: [262144..524287]: 3031040..3293183 262144 0x0 2: [524288..786431]: 3293184..3555327 262144 0x0 3: [786432..1048575]: 3555328..3817471 262144 0x0 4: [1048576..1310719]: 3817472..4079615 262144 0x0 5: [1310720..1425407]: 4079616..4194303 114688 0x0 6: [1425408..1687551]: 4456448..4718591 262144 0x0 7: [1687552..1949695]: 4718592..4980735 262144 0x0 8: [1949696..2211839]: 4980736..5242879 262144 0x0 9: [2211840..2412991]: 5242880..5444031 201152 0x1
(Note that the output is in 512B blocks)
As you can see the file is mostly contiguous, but it is divided into several extents because of two reasons.
1. The extent in ext4 has a limited size of 32768 blocks for initialized extent and 32767 block for unwritten extent. So when we exceed that size we need another extent which might be physically contiguous on disk with the previous one.
2. Ext4 divides disk space into allocation groups of certain size (cluster size * 8)blocks. Now with flex_bg medatada such as inode tables, block bitmaps and so one are packed closely together so the do not have to be stored with each block group and you'll get more contiguous data space.
However we're still storing backup superblock and Groups descriptors in certain groups and those are the gaps you're seeing in the fiemap list.
For detailed overview you can use dumpe2fs to see what is allocated where on the file system.
Bigalloc file system should help since you'll get much larger contiguous data spaces since the cluster size would be much bigger hence you'll get much bigger block groups. But you'll still get multiple extents (I do not really know how grub4dos recognizes contiguous files) even though there will be mostly contiguous.
But then again you'll have Group descriptors and backup superblocks in some groups so potentially some files might end up not contiguous. And even though you can turn off backup superblock you can not turn off writing group descriptors.
So no, at the moment there is not way to make really *really* sure that the file you're creating even with fallocate will allways be strictly contiguous. It'll mostly work, but it really depends on where the file will be put on the file system and how big is the file.
Also I've been talking about the case where nothing else is using the file system, when that's not the case. It might happen that the allocation will interfere (even though we're trying to allocate as contiguous file as we can). And also the file system free space will become more fragmented over time as it is used, but it really depends on the workload.
Now for the problem with bigalloc, I am not sure what kernel version are you using, but it's probably old. fallocate on bigalloc should work just fine.
Hope it helps.
Thanks!
-Lukas
Quoted 63 lines
But files aren't contiguous. e4defrag seems to confirm it:
# e4defrag -c '/run/media/neitsab/iso/_ISO' <Fragmented files> now/best size/ext 1. /run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso 13/1 119378 KB 2. /run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64_fal.iso 13/1 119378 KBTotal/best extents 26/2
Average size per extent 119378 KB
Fragmentation score 0
[0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag] This directory (/run/media/neitsab/iso/_ISO) does not need defragmentation. Done.Then I tried to use bigalloc and clusters:
# mkfs.ext4 -m 0 -L iso -i 67108864 -O bigalloc -C 256M -E root_owner=1000:100 /dev/sdc2 mke2fs 1.42.9 (28-Dec-2013)Warning: the bigalloc feature is still under development
See https://ext4.wiki.kernel.org/index.php/Bigalloc for more informationwarning: 53248 blocks unused.
Filesystem label=iso
OS type: Linux
Block size=4096 (log=2)
Cluster size=268435456 (log=18)
Stride=0 blocks, Stripe width=0 blocks
720 inodes, 11665408 blocks
0 blocks (0.00%) reserved for the super user First data block=0
Root directory owner=1000:100
1 block group
2146959360 blocks per group, 32760 clusters per group 720 inodes per groupAllocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: doneIt took it some lengthy time to allocate the group tables, then regular copy worked but fallocate hanged (process uninterruptible) and I had to reboot the machine, which led me to a kernel panic mentionning fallocate. The file copied via Nautilus still wasn't contiguous enough for Easy2Boot.
So what's left? Is it at all possible to make completely contiguous files on ext4? Not even on a fresh FS with plenty of free space?
Thanks for your comments!
[1] http://www.easy2boot.com/introduction/
[2] https://code.google.com/p/grub4dos-chenall/
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines · Contact
I'm trying to create a multiboot medium using Easy2Boot on a 64 GB USB key. Easy2Boot [1] is a "software" (well a collection of files really) that relies on grub4dos [2] to work and thus requires payload files to be strictly contiguous to do its trick. It's supposed to be set up once and then one just has to add files to boot from to a folder to modify the multiboot setup.
The ISO files I have are between 230 MB and 2 GB each. I divided my USB drive into two partitions : first one, 16 GB ntfs formatted so that grub4dos actually works (for some reasons it doesn't with ext4, plus I want to be able to use it with other OS); second one 48 GB formatted as ext4 with e2fsprogs version 1.42.9 from Archlinux. However when I try to boot from those files I systematically get the "file not contiguous" error in Easy2Boot.
Quoted 2 lines
Question is: is it possible to make huge files strictly contiguous on ex4, and if yes how?
Most of the time, files don't have to be "strictly contiguous", so that's not something that we've spent a lot of time trying to achieve.
There is a way to do this if you are willing to use the tip of the e2fsprogs "maint" branch, and then you put something like this into your mke2fs.conf file:
easy2boot = {
features = extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,^resize_inode,sparse_super2 hash_alg = half_md4
num_backup_sb = 0
packed_meta_blocks = 1
make_hugefiles = 1
inode_ratio = 4194304
hugefiles_dir = /
hugefiles_name = my-big-file
hugefiles_digits = 0
hugefiles_size = 16G
num_hugefiles = 1
zero_hugefiles = false
}
Then "mke2fs -T easy2boot /dev/sdc1" will create an ext4 filesystem with a file called /my-big-file which is guaranteed to be contiguous.
For your particular use case, where you want to create a new filesystem when you want to create your strictly contiguous file, this might be the best way to go.
Cheers,
- Ted
Quoted 47 lines · Lukáš Czerner
This is not how it it supposed to be used. Yes fallocate preallocates the file, but cp will truncate it so fallocate will certainly not help you in any way. In order for fallocate to be useful you'll have to write into the file without actually truncating it (dd can do this if you do not want to write your own program)
Also the file is probably as contiguous as it could be. Here is what I get on the file system with default mkfs options.
# e4defrag -c /mnt/test/file1 <File> now/best size/ext /mnt/test/file1 10/1 120649 KBBut that does not tell the whole story. See
xfs_io-f -c "fiemap -v"/mnt/test/file1/mnt/test/file1:EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS 0: [0..262143]: 2768896..3031039 262144 0x0 1: [262144..524287]: 3031040..3293183 262144 0x0 2: [524288..786431]: 3293184..3555327 262144 0x0 3: [786432..1048575]: 3555328..3817471 262144 0x0 4: [1048576..1310719]: 3817472..4079615 262144 0x0 5: [1310720..1425407]: 4079616..4194303 114688 0x0 6: [1425408..1687551]: 4456448..4718591 262144 0x0 7: [1687552..1949695]: 4718592..4980735 262144 0x0 8: [1949696..2211839]: 4980736..5242879 262144 0x0 9: [2211840..2412991]: 5242880..5444031 201152 0x1(Note that the output is in 512B blocks)
As you can see the file is mostly contiguous, but it is divided into several extents because of two reasons.
1. The extent in ext4 has a limited size of 32768 blocks for initialized extent and 32767 block for unwritten extent. So when we exceed that size we need another extent which might be physically contiguous on disk with the previous one.
2. Ext4 divides disk space into allocation groups of certain size (cluster size * 8)blocks. Now with
flex_bgmedatada such as inode tables, block bitmaps and so one are packed closely together so the do not have to be stored with each block group and you'll get more contiguous data space.
Quoted 7 lines
However we're still storing backup superblock and Groups descriptors in certain groups and those are the gaps you're seeing in the fiemap list.
For detailed overview you can use dumpe2fs to see what is allocated where on the file system.
Thanks, that was much interesting. I had delved a bit into ext4 data structure before posting but was never able to get a clear grasp on the limitations concerning contiguity.
So, I tried the correct way you recommended about how to use fallocate:
# mkfs.ext4 -m 0 -L iso -i 67108864 -E root_owner=1000:100 /dev/sdc2 $ fallocate -l 1589166080 '/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso' $ dd if='/home/neitsab/iso/Manjaro/manjaro-gnome-0.8.9-x86_64.iso' of='/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso'
However grub4dos displayed the error message about file being non-contiguous. Output of xfs_io:
$ xfs_io -f -c "fiemap -v" '/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso' /run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso: EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS 0: [0..262143]: 3457024..3719167 262144 0x0 1: [262144..524287]: 3719168..3981311 262144 0x0 2: [524288..737279]: 3981312..4194303 212992 0x0 3: [737280..999423]: 4456448..4718591 262144 0x0 4: [999424..1261567]: 4718592..4980735 262144 0x0 5: [1261568..1523711]: 4980736..5242879 262144 0x0 6: [1523712..1785855]: 5242880..5505023 262144 0x0 7: [1785856..2047999]: 5505024..5767167 262144 0x0 8: [2048000..2310143]: 5767168..6029311 262144 0x0 9: [2310144..2572287]: 6029312..6291455 262144 0x0 10: [2572288..2834431]: 6291456..6553599 262144 0x0 11: [2834432..3096575]: 6569984..6832127 262144 0x0 12: [3096576..3103839]: 6832128..6839391 7264 0x1
Second and third extents aren't contiguous, neither are tenth and eleventh. But it was close!
Anyway, thanks to your explanations I understand why perfect contiguity isn't so likely. However I remember using with success a Perl script from 2007 called defragfs [1] recommended by Easy2Boot's author. After running this script a couple of times on the key's ISO folder I was able to boot on most of the files. Problem with this approach, is that AFAIK defragmentation is pretty harmful for flash drives, so I wasn't willing to do it every time I add an ISO.
Quoted 25 lines
Bigalloc file system should help since you'll get much larger contiguous data spaces since the cluster size would be much bigger hence you'll get much bigger block groups. But you'll still get multiple extents (I do not really know how grub4dos recognizes contiguous files) even though there will be mostly contiguous.
But then again you'll have Group descriptors and backup superblocks in some groups so potentially some files might end up not contiguous. And even though you can turn off backup superblock you can not turn off writing group descriptors.
So no, at the moment there is not way to make really *really* sure that the file you're creating even with fallocate will allways be strictly contiguous. It'll mostly work, but it really depends on where the file will be put on the file system and how big is the file.
Also I've been talking about the case where nothing else is using the file system, when that's not the case. It might happen that the allocation will interfere (even though we're trying to allocate as contiguous file as we can). And also the file system free space will become more fragmented over time as it is used, but it really depends on the workload.
Quoted 8 lines
Now for the problem with bigalloc, I am not sure what kernel version are you using, but it's probably old. fallocate on bigalloc should work just fine.
Hope it helps.
Thanks!
-Lukas
Concerning fallocate with bigalloc, I don't think it is related to my kernel which is rather recent:
$ uname -a Linux arch-clevo 3.14.1-1-ARCH #1 SMP PREEMPT Mon Apr 14 20:40:47 CEST 2014 x86_64 GNU/Linux
So, I tried again to use fallocate on the bigalloc'ed fs, it hanged again (uninterruptible sleep state, D in ps aux:
$ ps aux | grep fallocate neitsab 3673 0.0 0.0 0 0 ? D 19:36 0:00 [fallocate] kill/kill -9 don't change anything. gnome-system-monitor gives in the process properties for fallocate: "Wait channel | call_rwsem_down_write_failed".
So after searching a bit more, it seems to be related to a driver issue (see [2] and [3] for a discussion and more info), apparently fallocate is waiting on some I/O that never come... So that's on my USB key (for info it is a Sandisk Extreme 64 GB USB 3.0, model SDCZ80-0654G, using driver xhci_hcd).
As I've been advised in another discussion to be cautious with bigalloc because it is still in development, I'll leave it aside for now.
Quoted 24 lines · Theodore Ts'o
Most of the time, files don't have to be "strictly contiguous", so that's not something that we've spent a lot of time trying to achieve.
There is a way to do this if you are willing to use the tip of the e2fsprogs "maint" branch, and then you put something like this into your mke2fs.conf file:
easy2boot = {
features = extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,^resize_inode,sparse_super2hash_alg=half_md4
num_backup_sb= 0
packed_meta_blocks= 1
make_hugefiles= 1
inode_ratio= 4194304
hugefiles_dir= /
hugefiles_name= my-big-file
hugefiles_digits= 0
hugefiles_size= 16G
num_hugefiles= 1
zero_hugefiles= false
}
Quoted 7 lines
Then "mke2fs -T easy2boot
/dev/sdc1" will create an ext4 filesystem with a file called /my-big-file which is guaranteed to be contiguous.For your particular use case, where you want to create a new filesystem when you want to create your strictly contiguous file, this might be the best way to go.
Thanks a lot for the detailed solution! If I want mke2fs to create multiple contiguous files (this is for a multiboot medium so I'm gonna have quite a few of them, e.g. 20 files of 2 GB so as to account for the maximum), which variables should I modify in the mke2fs.conf entry? hugefiles_size and num_hugefiles?
Although I'd like to test this solution, I still have to get into git and compiling, moreover I'm looking for a less involving solution so for now I think I'll just stick to regular e2fsprogs and try to find another multiboot utility.
But thanks for your quick and detailed answers!
-- Bastien
[1] http://defragfs.sourceforge.net/
[2] http://linuxgazette.net/issue83/tag/6.html
[3] http://www.novell.com/support/kb/doc.php?id=7002725
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Just a little update:
Nonetheless my issues with fallocate and the bigalloc'ed fs, I tried to dd a couple of ISOs on the key and check whether I could boot from them. And to my great surprise... The answer was yes.
$ ls -l /run/media/neitsab/iso/_ISO/ total 4980736 drwxr-xr-x 2 neitsab users 4096 17 avril 21:38 . drwxr-xr-x 4 neitsab users 4096 17 avril 19:36 .. -rw------- 1 neitsab users 964689920 17 avril 21:37 gnome-3.12.iso -rw------- 1 neitsab users 1589166080 17 avril 20:45 manjaro-gnome-0.8.9-x86_64.iso -rw------- 1 neitsab users 993001472 17 avril 21:38 openSUSE-13.1-GNOME-Live-x86_64.iso
I can boot from the two others but not from manjaro's one, which happens to be the biggest too. A comparison of xfs_io output :
$ xfs_io -f -c "fiemap -v" '/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso' /run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso: EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS 0: [0..262143]: 3145728..3407871 262144 0x0 1: [262144..524287]: 3407872..3670015 262144 0x0 2: [524288..786431]: 3670016..3932159 262144 0x0 3: [786432..1048575]: 3932160..4194303 262144 0x0 4: [1048576..1310719]: 4194304..4456447 262144 0x0 5: [1310720..1572863]: 4456448..4718591 262144 0x0 6: [1572864..1835007]: 5242880..5505023 262144 0x0 7: [1835008..2097151]: 5505024..5767167 262144 0x0 8: [2097152..2359295]: 5767168..6029311 262144 0x0 9: [2359296..2621439]: 6029312..6291455 262144 0x0 10: [2621440..2883583]: 6291456..6553599 262144 0x0 11: [2883584..3103839]: 6553600..6773855 220256 0x1
$ xfs_io -f -c "fiemap -v" '/run/media/neitsab/iso/_ISO/openSUSE-13.1-GNOME-Live-x86_64.iso' /run/media/neitsab/iso/_ISO/openSUSE-13.1-GNOME-Live-x86_64.iso: EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS 0: [0..262143]: 8912896..9175039 262144 0x0 1: [262144..524287]: 9175040..9437183 262144 0x0 2: [524288..786431]: 9437184..9699327 262144 0x0 3: [786432..1048575]: 9699328..9961471 262144 0x0 4: [1048576..1310719]: 9961472..10223615 262144 0x0 5: [1310720..1572863]: 10223616..10485759 262144 0x0 6: [1572864..1835007]: 10485760..10747903 262144 0x0 7: [1835008..1939455]: 10747904..10852351 104448 0x1
$ xfs_io -f -c "fiemap -v" '/run/media/neitsab/iso/_ISO/gnome-3.12.iso' /run/media/neitsab/iso/_ISO/gnome-3.12.iso: EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS 0: [0..262143]: 6815744..7077887 262144 0x0 1: [262144..524287]: 7077888..7340031 262144 0x0 2: [524288..786431]: 7340032..7602175 262144 0x0 3: [786432..1048575]: 7602176..7864319 262144 0x0 4: [1048576..1310719]: 7864320..8126463 262144 0x0 5: [1310720..1572863]: 8126464..8388607 262144 0x0 6: [1572864..1835007]: 8388608..8650751 262144 0x0 7: [1835008..1884159]: 8650752..8699903 49152 0x1
So they're all contiguous but Manjaro. I guess that brings us back to the randomness of block allocation as explained by Lukáš and to an effect of file size on the probability to be non-contiguous (which is rather logical)!
Thanks all for your explanations,
Cheers
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 67 lines · Contact
Date: Thu, 17 Apr 2014 20:41:34 +0200
From: Contact <[email protected]>
To: Theodore Ts'o <[email protected]>, [email protected]
Cc: "[email protected]" <[email protected]>
Subject: Re: Make huge files strictly contiguous (fallocate, bigalloc,e4defrag...)Quoted 47 lines · Lukáš Czerner
This is not how it it supposed to be used. Yes fallocate preallocates the file, but cp will truncate it so fallocate will certainly not help you in any way. In order for fallocate to be useful you'll have to write into the file without actually truncating it (dd can do this if you do not want to write your own program)
Also the file is probably as contiguous as it could be. Here is what I get on the file system with default mkfs options.
# e4defrag -c /mnt/test/file1 <File> now/best size/ext /mnt/test/file1 10/1 120649 KBBut that does not tell the whole story. See
xfs_io-f -c "fiemap -v"/mnt/test/file1/mnt/test/file1:EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS 0: [0..262143]: 2768896..3031039 262144 0x0 1: [262144..524287]: 3031040..3293183 262144 0x0 2: [524288..786431]: 3293184..3555327 262144 0x0 3: [786432..1048575]: 3555328..3817471 262144 0x0 4: [1048576..1310719]: 3817472..4079615 262144 0x0 5: [1310720..1425407]: 4079616..4194303 114688 0x0 6: [1425408..1687551]: 4456448..4718591 262144 0x0 7: [1687552..1949695]: 4718592..4980735 262144 0x0 8: [1949696..2211839]: 4980736..5242879 262144 0x0 9: [2211840..2412991]: 5242880..5444031 201152 0x1(Note that the output is in 512B blocks)
As you can see the file is mostly contiguous, but it is divided into several extents because of two reasons.
1. The extent in ext4 has a limited size of 32768 blocks for initialized extent and 32767 block for unwritten extent. So when we exceed that size we need another extent which might be physically contiguous on disk with the previous one.
2. Ext4 divides disk space into allocation groups of certain size (cluster size * 8)blocks. Now with
flex_bgmedatada such as inode tables, block bitmaps and so one are packed closely together so the do not have to be stored with each block group and you'll get more contiguous data space.Quoted 7 lines
However we're still storing backup superblock and Groups descriptors in certain groups and those are the gaps you're seeing in the fiemap list.
For detailed overview you can use dumpe2fs to see what is allocated where on the file system.
Thanks, that was much interesting. I had delved a bit into ext4 data structure before posting but was never able to get a clear grasp on the limitations concerning contiguity.
Quoted 7 lines
So, I tried the correct way you recommended about how to use fallocate:
# mkfs.ext4 -m 0 -L iso -i 67108864 -E root_owner=1000:100 /dev/sdc2 $ fallocate -l 1589166080 '/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso' $ dd if='/home/neitsab/iso/Manjaro/manjaro-gnome-0.8.9-x86_64.iso' of='/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso'
The way you're using dd for this purpose is wrong. I said that dd can write to the file without truncating it first, but it's not doing it by default. You have to use 'notrunc' see man page. But I think that it would not have made a difference anyway.
-Lukas
Quoted 6 lines · Lukáš Czerner
The way you're using dd for this purpose is wrong. I said that dd can write to the file without truncating it first, but it's not doing it by default. You have to use 'notrunc' see man page. But I think that it would not have made a difference anyway.
Thansk, I stand corrected. I'll try it to see whether I can get better results with it.
Quoted 3 lines
And even though you can turn off backup superblock you can not turn off writing group descriptors.
While we're at it, how can I do this? I searched the mkfs.ext4/mke2fs.conf/ext4/tune2fs man pages for the option to turn off writing backup superblock you mentioned, however the closest I found is tune2fs -O ^sparse_super which is part of the defaults mke2fs.conf options... Could you point me at it so that I can try to get this source of non-contiguity away?
Thanks,
-Bastien
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Contact
While we're at it, how can I do this? I searched the mkfs.ext4/mke2fs.conf/ext4/tune2fs man pages for the option to turn off writing backup superblock you mentioned, however the closest I found is tune2fs -O ^sparse_super which is part of the defaults mke2fs.conf options... Could you point me at it so that I can try to get this source of non-contiguity away?
This requires recompiling e2fsprogs using the tip of the maint branch, as I had suggested earlier (and which you had ruled out because you didn't want to build your own e2fsprogs from source).
Once you do this, and you put the following in
hugefiles = {
features = extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,^resize_inode,sparse_super2 hash_alg = half_md4
num_backup_sb = 0
packed_meta_blocks = 1
flex_bg_size = 262144
inode_ratio = 4194304
make_hugefiles = 1
hugefiles_dir = /boot
hugefiles_name = contig-files
hugefiles_digits = 1
hugefiles_size = 256M
num_hugefiles = 3
}
Using the sparse_super2 feature and setting num_backup_sb=0 disables the backup superblocks. Then setting flex_bg_size to a very large value (262144 in this case), will force all of the allocation bitmaps and inodetables to the very beginning of the file system.
The inode_ratio setting simply radically reduces the number of inodes in the file system. That was there because originally I was using this this command set with the hugefiles_size set to 1 or 2 gigabytes, and filling the disk with completely with large files (num_hugefiles = 0 means "create as many huge files as necessary to fill the disk).
The rest of the "hugefiles" related settings are just simply allowing mke2fs to create the contiguous files for you.
Note that because this uses a new compat feature, "sparse_super2", if you use the new mke2fs to create such a file system, you will also need to make sure the file system is checked using a version of e2fsck that supports this new feature as well. No kernel changes are required to support this, though.
Cheers,
- Ted
Quoted 5 lines · Theodore Ts'o
This requires recompiling e2fsprogs using the tip of the maint branch, as I had suggested earlier (and which you had ruled out because you didn't want to build your own e2fsprogs from source).
Thanks, that explains why I wasn't able to find this option in regular e2fsprogs.
Quoted 17 lines
Once you do this, and you put the following in
hugefiles = {
features = extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize,^resize_inode,sparse_super2hash_alg=half_md4
num_backup_sb= 0
packed_meta_blocks= 1
flex_bg_size= 262144
inode_ratio= 4194304
make_hugefiles= 1
hugefiles_dir= /boot
hugefiles_name= contig-files
hugefiles_digits= 1
hugefiles_size= 256M
num_hugefiles= 3
}
Quoted 5 lines
Using the
sparse_super2feature and setting num_backup_sb=0 disables the backup superblocks. Then settingflex_bg_sizeto a very large value (262144 in this case), will force all of the allocation bitmaps and inodetables to the very beginning of the file system.
The tip and stanza look very useful though, I'll reconsider retrieving and compiling maint branch to try, thanks.
Quoted 6 lines
Note that because this uses a new compat feature, "
sparse_super2", if you use the new mke2fs to create such a file system, you will also need to make sure the file system is checked using a version of e2fsck that supports this new feature as well. No kernel changes are required to support this, though.
Any plans to get this feature into master at some point in the development?
--Bastien
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Contact
Quoted 6 lines
Note that because this uses a new compat feature, "
sparse_super2", if you use the new mke2fs to create such a file system, you will also need to make sure the file system is checked using a version of e2fsck that supports this new feature as well. No kernel changes are required to support this, though.Any plans to get this feature into master at some point in the development?
The maint branch is regularly merged into the master and next branches, so it's there today. The maint branch will very shortly be the source of a 1.42.10 maintenance release.
The master branch will eventualy be a 1.43 release; it has some much more disruptive changes, and adds support for metadata checksums, inline data, etc. It's likely that 1.42.10 will be the last maintenance release, and the focus will be on stablizing 1.43 and making sure we're really ready to support the ABI's for the new library functions in libext2fs to support these new features _forever_.
Cheers,
- Ted