counting fragments takes more time than defragmenting
Hi Chris,
After our quick chat, I gave it a shot on 3.19.6, and things are better than last time I tried.
legolas:/var/local/nobck/VirtualBox VMs# lsattr Win7/ ---------------C Win7/Logs
---------------C Win7/Snapshots ---------------C Win7/Win7.vdi ---------------C Win7/Win7.png ---------------C Win7/autotune1.png ---------------C Win7/new_autotune2.png ---------------C Win7/Win7.vbox-prev ---------------C Win7/Win7.vbox
But I have snapshots of that subvolume, so obviously that gets in the way of disabling COW.
I had a look, and I have 100K fragments. That took 10mn to figure out:
legolas:/var/local/nobck/VirtualBox VMs/Win7# filefrag Win7.vdi Win7.vdi: 104306 extents found
This first filefrag run took about 10mn to count all the fragments on my SSD. That feels a bit slow, but maybe the userland tool is doing things in suboptimal ways.
Defrag actually worked (mostly) and wasn't too slow. It used to take hours not to finish, and now it worked in 3mn:
legolas:/var/local/nobck/VirtualBox VMs/Win7# time btrfs fi defrag Win7.vdi
real 3m43.807s
user 0m0.000s
sys 0m44.044s
This is defintely better than before.
Note that it's not fully defragged, but close enough. Each subsequent run, filefrag is faster, and defrag is still faster than filefrag:
legolas:/var/local/nobck/VirtualBox VMs/Win7# time filefrag Win7.vdi Win7.vdi: 11428 extents found
real 2m42.090s
user 0m0.000s
sys 2m37.308s
legolas:/var/local/nobck/VirtualBox VMs/Win7# time btrfs fi defrag Win7.vdi
real 0m7.483s
user 0m0.000s
sys 0m2.672s
legolas:/var/local/nobck/VirtualBox VMs/Win7# time filefrag Win7.vdi Win7.vdi: 11132 extents found
real 0m22.525s
user 0m0.000s
sys 0m22.264s
It's a bit unexpected that I still have 10k fragments after 2 defrag runs, but it's better than 100k :)
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901Hello again,
Just curious, is anyone seeing similar things with big VM images or other DBs?
I forgot to mention that my vdi file is 88GB.
It's surprising that it took longer to count the fragments than to actually defragment the file.
Or that it took 3 defrag runs to get down to 11K extents from 104K.
Are others seeing similar things?
Marc
Quoted 69 lines · Marc MERLIN
Hi Chris,
After our quick chat, I gave it a shot on 3.19.6, and things are better than last time I tried.
legolas:/var/local/nobck/VirtualBox VMs# lsattr Win7/ ---------------C Win7/Logs
---------------C Win7/Snapshots ---------------C Win7/Win7.vdi ---------------C Win7/Win7.png ---------------C Win7/autotune1.png ---------------C Win7/new_autotune2.png ---------------C Win7/Win7.vbox-prev ---------------C Win7/Win7.vboxBut I have snapshots of that subvolume, so obviously that gets in the way of disabling COW.
I had a look, and I have 100K fragments. That took 10mn to figure out:
legolas:/var/local/nobck/VirtualBox VMs/Win7# filefrag Win7.vdi Win7.vdi: 104306 extents found
This first filefrag run took about 10mn to count all the fragments on my SSD. That feels a bit slow, but maybe the userland tool is doing things in suboptimal ways.
Defrag actually worked (mostly) and wasn't too slow. It used to take hours not to finish, and now it worked in 3mn:
legolas:/var/local/nobck/VirtualBox VMs/Win7# time btrfs fi defrag Win7.vdi
real 3m43.807s
user 0m0.000s
sys 0m44.044sThis is defintely better than before.
Note that it's not fully defragged, but close enough. Each subsequent run, filefrag is faster, and defrag is still faster than filefrag:legolas:/var/local/nobck/VirtualBox VMs/Win7# time filefrag Win7.vdi Win7.vdi: 11428 extents found
real 2m42.090s
user 0m0.000s
sys 2m37.308slegolas:/var/local/nobck/VirtualBox VMs/Win7# time btrfs fi defrag Win7.vdi
real 0m7.483s
user 0m0.000s
sys 0m2.672slegolas:/var/local/nobck/VirtualBox VMs/Win7# time filefrag Win7.vdi Win7.vdi: 11132 extents found
real 0m22.525s
user 0m0.000s
sys 0m22.264sIt's a bit unexpected that I still have 10k fragments after 2 defrag runs, but it's better than 100k :)
Marc
Signature · 8 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 7 lines · Marc MERLIN
Hello again,
Just curious, is anyone seeing similar things with big VM images or other DBs?
I forgot to mention that my vdi file is 88GB.
Quoted 5 lines
It's surprising that it took longer to count the fragments than to actually defragment the file.
Or that it took 3 defrag runs to get down to 11K extents from 104K.Are others seeing similar things?
Filefrag is pretty much instant for my 30GB (150 extents) virtual disk, no CoW on file, no snapshots on volume.
But what doesn't make sense to me is btrfs fi defrag; the -t option says
-t <size>
defragment only files at least <size> bytes big
The -t value goes into struct btrfs_ioctl_defrag_range_args.extent_thresh which is documented as
/*
* any extent bigger than this will be considered
* already defragged. Use 0 to take the kernel default
* Use 1 to say every single extent must be rewritten */
Default extent_thresh is 256K. I can't see how 1 would say every single extent must be rewritten. On the contrary; 1 skips every extent. The compress flag even sets extent_thresh=(u32)-1 to force a rewrite.
Marc, try btrfs fi defrag -t 4294967295 Win7.vdi for maximum defrag and time filefrag again with fewer extents.
/Patrik
Quoted 82 lines
Marc
Quoted 58 lines · Marc MERLIN
Hi Chris,
After our quick chat, I gave it a shot on 3.19.6, and things are better than last time I tried.
legolas:/var/local/nobck/VirtualBox VMs# lsattr Win7/ ---------------C Win7/Logs
---------------C Win7/Snapshots ---------------C Win7/Win7.vdi ---------------C Win7/Win7.png ---------------C Win7/autotune1.png ---------------C Win7/new_autotune2.png ---------------C Win7/Win7.vbox-prev ---------------C Win7/Win7.vboxBut I have snapshots of that subvolume, so obviously that gets in the way of disabling COW.
I had a look, and I have 100K fragments. That took 10mn to figure out:
legolas:/var/local/nobck/VirtualBox VMs/Win7# filefrag Win7.vdi Win7.vdi: 104306 extents found
This first filefrag run took about 10mn to count all the fragments on my SSD. That feels a bit slow, but maybe the userland tool is doing things in suboptimal ways.
Defrag actually worked (mostly) and wasn't too slow. It used to take hours not to finish, and now it worked in 3mn:
legolas:/var/local/nobck/VirtualBox VMs/Win7# time btrfs fi defrag Win7.vdireal 3m43.807s user 0m0.000s sys 0m44.044sThis is defintely better than before.
Note that it's not fully defragged, but close enough. Each subsequent run, filefrag is faster, and defrag is still faster than filefrag:legolas:/var/local/nobck/VirtualBox VMs/Win7# time filefrag Win7.vdi Win7.vdi: 11428 extents found
real 2m42.090s user 0m0.000s sys 2m37.308slegolas:/var/local/nobck/VirtualBox VMs/Win7# time btrfs fi defrag Win7.vdi
real 0m7.483s user 0m0.000s sys 0m2.672slegolas:/var/local/nobck/VirtualBox VMs/Win7# time filefrag Win7.vdi Win7.vdi: 11132 extents found
real 0m22.525s user 0m0.000s sys 0m22.264sIt's a bit unexpected that I still have 10k fragments after 2 defrag runs, but it's better than 100k :)
Quoted 11 lines
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Patrik Lundquist posted on Wed, 24 Jun 2015 10:28:09 +0200 as excerpted:
Quoted 13 lines
But what doesn't make sense to me is btrfs fi defrag; the -t option says
-t <size>
defragment only files at least <size> bytes bigThe -t value goes into struct btrfs_ioctl_defrag_range_args.extent_thresh which is documented as
/*
* any extent bigger than this will be considered * already defragged. Use 0 to take the kernel default * Use 1 to say every single extent must be rewritten */
Quoted 6 lines
Default
extent_threshis 256K. I can't see how 1 would say every single extent must be rewritten. On the contrary; 1 skips every extent. The compress flag even sets extent_thresh=(u32)-1 to force a rewrite.Marc, try btrfs fi defrag -t 4294967295 Win7.vdi for maximum defrag and time filefrag again with fewer extents.
The manpage wording for btrfs fi defrag -t has been debated on-list several times, and I believe remains (as of btrfs-progs v4.1) confusing still.
First, under the general defragment description, before the individual options, it says:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Any extent bigger than threshold given by -t option, will be considered already defragged. Use 0 to take the kernel default.
So according to that, an extent BIGGER than -t is treated as already defragged (it defrags SMALLER extents)
But, under the -t option, it says:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
-t <size>[kKmMgGtTpPeE]
defragment only files at least <size> bytes big
So according to that, only extents BIGGER than -t are defragged (smaller is ignored).
Again, that's the btrfs-filesystem (8) manpage as of -progs 4.1.
So which is it? The manpage itself can't make up its mind.
AFAIK, it's set huge to defrag everything, but last time I posted on this I got it wrong, and I don't remember for sure what I said then, so... try it and see to be sure, which is what I'd do.
Meanwhile, it's worth noting that btrfs data chunks are normally 1 GiB (tho apparently they can be bigger under certain circumstances). 1 extent per chunk is the best btrfs normally does, which means 1 GiB per extent is nominally the best that can be done, with the first and last extent possibly less than a gig (the first taking up the remainder of a partially used chunk and the last finishing up the file, which probably won't end on an even chunk boundary).
Assuming "set a huge -t to defrag to the maximum extent possible" is correct, that means -t 1G should be exactly as effective as -t 1T...
Regardless of whether 1 or huge -t means maximum defrag, however, the nominal data chunk size of 1 GiB means that 30 GiB file you mentioned should be considered ideally defragged at 31 extents. This is a departure from ext4, which AFAIK in theory has no extent upper limit, so should be able to do that 30 GiB file in a single extent.
But btrfs or ext4, 31 extents ideal or a single extent ideal, 150 extents still indicates at least some remaining fragmentation.
Finally, last I remember, filefrag didn't understand btrfs compression (which is off for nocow, so this shouldn't apply there), which uses 128 KiB blocks IIRC. Until it does, large btrfs-compressed files will always show many extents 8/MiB, so thousands on anything even close to a GiB, tens of thousands on multiple GiBs). But I believe there had been some work to teach filefrag about btrfs compression, tho I don't know if it has made it into an e2fsprogs release, yet. If so, it'll be pretty close to the latest release. So anything but the latest filefrag won't be accurate with btrfs-compressed files, while the latest may now be accurate, or not, I'm not sure. I guess one could check e2fsprogs' release notes...
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 3 lines · Duncan
Patrik Lundquist posted on Wed, 24 Jun 2015 10:28:09 +0200 as excerpted:
AFAIK, it's set huge to defrag everything,
It's set to 256K by default.
Quoted 2 lines
Assuming "set a huge -t to defrag to the maximum extent possible" is correct, that means -t 1G should be exactly as effective as -t 1T...
1G is actually more effective because 1T overflows the uint32 extent_thresh field, so 1T, 0, and 256K are currently the same.
3G is the largest value that works with -t as expected (disregarding the man page) and is easy to type.
Quoted 2 lines
But btrfs or ext4, 31 extents ideal or a single extent ideal, 150 extents still indicates at least some remaining fragmentation.
I gave it another shot but I've now got 154 extents instead. :-)
Patrik Lundquist posted on Wed, 24 Jun 2015 14:05:57 +0200 as excerpted:
Quoted 7 lines · Duncan
Quoted 4 lines
Patrik Lundquist posted on Wed, 24 Jun 2015 10:28:09 +0200 as excerpted:
AFAIK, it's set huge to defrag everything,
It's set to 256K by default.
What I meant is that AFAIK, set it huge to defrag everything...
Quoted 5 lines
Quoted 2 lines
Assuming "set a huge -t to defrag to the maximum extent possible" is correct, that means -t 1G should be exactly as effective as -t 1T...
1G is actually more effective because 1T overflows the uint32
extent_threshfield, so 1T, 0, and 256K are currently the same.
Then the manpage needs some work (in addition to the more serious ambiguity over whether 1 or 1G means defrag everything), since it mentions upto petabyte (P), without any indication that setting anything that large won't work as expected.
If it's uint32 limited, either kill everything above that in both the documentation and code, or alias everything above that to 3G (your next paragraph) or whatever.
Quoted 8 lines
3G is the largest value that works with -t as expected (disregarding the man page) and is easy to type.
Quoted 2 lines
But btrfs or ext4, 31 extents ideal or a single extent ideal, 150 extents still indicates at least some remaining fragmentation.
I gave it another shot but I've now got 154 extents instead. :-)
Is it possible there's simply no gig-size free-space holes in the filesystem allocation, so it simply /can't/ defrag further than that, because there's no place to allocate whole-gig data chunks at a time?
Which brings up a more general defrag functionality question. For multi- gig files, does btrfs fi defrag allocate fresh data chunks in ordered to create the largest extents possible (possibly after filling the remainder of the original first chunk), thereby increasing data chunk allocation before fully using currently allocated chunks, or does it try to find the biggest extents possible in currently allocated chunks, first, and only allocate new chunks when all current allocation is full?
Obviously if it uses up current allocations first, that could explain your problem. OTOH, if either defrag or general allocation strategy favors new chunks for large extents when necessary, that would explain the "deoptimization" some people report from running balance.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 8 lines · Duncan
Patrik Lundquist posted on Wed, 24 Jun 2015 14:05:57 +0200 as excerpted:
Quoted 1 line · Duncan
If it's uint32 limited, either kill everything above that in both the documentation and code, or alias everything above that to 3G (your next paragraph) or whatever.
My simple overflow patch yesterday fixes the problem, so 4G or larger is max instead of 0.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
But btrfs or ext4, 31 extents ideal or a single extent ideal, 150 extents still indicates at least some remaining fragmentation.
I gave it another shot but I've now got 154 extents instead. :-)
Is it possible there's simply no gig-size free-space holes in the filesystem allocation, so it simply /can't/ defrag further than that, because there's no place to allocate whole-gig data chunks at a time?
I would guess so, without allocating new chunks. Defrag can probably be smarter and avoid rewriting extents if it means splitting them (unless the compression flag is set and it must rewrite everything).
Quoted 9 lines · Duncan
Regardless of whether 1 or huge -t means maximum defrag, however, the nominal data chunk size of 1 GiB means that 30 GiB file you mentioned should be considered ideally defragged at 31 extents. This is a departure from ext4, which AFAIK in theory has no extent upper limit, so should be able to do that 30 GiB file in a single extent.
But btrfs or ext4, 31 extents ideal or a single extent ideal, 150 extents still indicates at least some remaining fragmentation.
So I converted the VMware VMDK file to a VirtualBox VDI file:
-rw------- 1 plu plu 28845539328 jul 13 13:36 Windows7-disk1.vmdk -rw------- 1 plu plu 28993126400 jul 13 14:04 Windows7.vdi
$ filefrag Windows7.vdi Windows7.vdi: 15 extents found
$ btrfs filesystem defragment -t 3g Windows7.vdi $ filefrag Windows7.vdi Windows7.vdi: 24 extents found
How can it be less than 28 extents with a chunk size of 1 GiB?
E2fsprogs version 1.42.12
Patrik Lundquist posted on Tue, 14 Jul 2015 13:57:07 +0200 as excerpted:
Quoted 16 lines · Duncan
Quoted 9 lines
Regardless of whether 1 or huge -t means maximum defrag, however, the nominal data chunk size of 1 GiB means that 30 GiB file you mentioned should be considered ideally defragged at 31 extents. This is a departure from ext4, which AFAIK in theory has no extent upper limit, so should be able to do that 30 GiB file in a single extent.
But btrfs or ext4, 31 extents ideal or a single extent ideal, 150 extents still indicates at least some remaining fragmentation.
So I converted the VMware VMDK file to a VirtualBox VDI file:
(quoted diff) +0
-2-rw------- 1 plu plu 28845539328 jul 13 13:36 Windows7-disk1.vmdk -rw------- 1 plu plu 28993126400 jul 13 14:04 Windows7.vdi
Quoted 8 lines
$ filefrag Windows7.vdi Windows7.vdi: 15 extents found$ btrfs filesystem defragment -t 3g Windows7.vdi $ filefrag Windows7.vdi Windows7.vdi: 24 extents foundHow can it be less than 28 extents with a chunk size of 1 GiB?
E2fsprogs version 1.42.12
That's why I said "nominal"[1] 1 GiB. I'm just a list and filesystem user, not a dev, and I don't know the details, but someone (a dev or at least someone that can actually read code, but not a btrfs dev) mentioned in reply to a post of mine a few months ago, that under the right conditions, btrfs can allocate larger-than 1 GiB data chunks.
I /believe/ data chunk allocation size has something to do with the amount of unallocated space on the filesystem; that on large (TiB plus, perhaps) btrfs some of the initial allocations will be multiple GiB, which of course would allow greater-than 1 GiB extents as well. But I really don't know the conditions under which that can happen and I've not seen an actual btrfs dev comment on it, and AFAIK the "base" data chunk size remains 1 GiB under most conditions. Meanwhile, I tend to partition up my storage here, and while I have multiple separate btrfs, the partitions are all under 50 GiB, so I'm unlikely to see that sort of > 1 GiB data chunk allocations at all, here.
So rather than go to the complexity of explaining all this detail that I'm not sure of anyway, I deliberately blurred out a bit as not necessary to the primary point, which was that for files over a GiB, don't expect to see or be able to defrag to a single extent, as 1 GiB data chunks and thus extents are nominal/normal.
If it does happen, I'd consider it due to those data "superchunks" and wouldn't be entirely surprised, but the point remains that you're unlikely to get the number of extents much below the file size number in GiB using defrag, even when everything is working "perfectly as designed".
---
[1] Nominal: In the sense of normal or standard as-designed value, see wiktionary's English adjective sense 6 and 10, as well as the wikipedia writeups on real vs. nominal values and nominal size:
https://en.wiktionary.org/wiki/nominal#Adjective
https://en.wikipedia.org/wiki/Real_versus_nominal_value
https://en.wikipedia.org/wiki/Nominal_size
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 16 lines · Patrik Lundquist
Quoted 9 lines · Duncan
Regardless of whether 1 or huge -t means maximum defrag, however, the nominal data chunk size of 1 GiB means that 30 GiB file you mentioned should be considered ideally defragged at 31 extents. This is a departure from ext4, which AFAIK in theory has no extent upper limit, so should be able to do that 30 GiB file in a single extent.
But btrfs or ext4, 31 extents ideal or a single extent ideal, 150 extents still indicates at least some remaining fragmentation.
So I converted the VMware VMDK file to a VirtualBox VDI file:
(quoted diff) +0
-2-rw------- 1 plu plu 28845539328 jul 13 13:36 Windows7-disk1.vmdk -rw------- 1 plu plu 28993126400 jul 13 14:04 Windows7.vdi
Quoted 8 lines
$ filefrag Windows7.vdi Windows7.vdi: 15 extents found$ btrfs filesystem defragment -t 3g Windows7.vdi $ filefrag Windows7.vdi Windows7.vdi: 24 extents foundHow can it be less than 28 extents with a chunk size of 1 GiB?
I _think_ the fragment size will be limited by the block group
size. This is not the same as the chunk size for some RAID levels -- for example, RAID-0, a block group can be anything from 2 to n chunks (across the same number of devices), where each chunk is 1 GiB, so potentially you could have arbitrary-sized block groups. The same would apply to RAID-10, -5 and -6.
(Note, I haven't verified this, but it makes sense based on what I
know of the internal data structures).
Hugo.
Signature · 4 lines
Hugo Mills | Go not to the elves for counsel, for they will say hugo@... carfax.org.uk | both no and yes. http://carfax.org.uk/ | PGP: E2AB1DE4 |
Quoted 33 lines · Hugo Mills
Quoted 16 lines · Patrik Lundquist
Quoted 9 lines · Duncan
Regardless of whether 1 or huge -t means maximum defrag, however, the nominal data chunk size of 1 GiB means that 30 GiB file you mentioned should be considered ideally defragged at 31 extents. This is a departure from ext4, which AFAIK in theory has no extent upper limit, so should be able to do that 30 GiB file in a single extent.
But btrfs or ext4, 31 extents ideal or a single extent ideal, 150 extents still indicates at least some remaining fragmentation.
So I converted the VMware VMDK file to a VirtualBox VDI file:
(quoted diff) +0
-2-rw------- 1 plu plu 28845539328 jul 13 13:36 Windows7-disk1.vmdk -rw------- 1 plu plu 28993126400 jul 13 14:04 Windows7.vdiQuoted 8 lines
$ filefrag Windows7.vdi Windows7.vdi: 15 extents found$ btrfs filesystem defragment -t 3g Windows7.vdi $ filefrag Windows7.vdi Windows7.vdi: 24 extents foundHow can it be less than 28 extents with a chunk size of 1 GiB?
I _think_ the fragment size will be limited by the block group
size. This is not the same as the chunk size for some RAID levels -- for example, RAID-0, a block group can be anything from 2 to n chunks (across the same number of devices), where each chunk is 1 GiB, so potentially you could have arbitrary-sized block groups. The same would apply to RAID-10, -5 and -6.
Quoted 2 lines
(Note, I haven't verified this, but it makes sense based on what I
know of the internal data structures).
It's a raid1 filesystem, so the block group ought to be the same size as the chunk, right?
A 2GiB block group would suffice to explain it though.
Quoted 34 lines · Patrik Lundquist
Quoted 33 lines · Hugo Mills
Quoted 16 lines · Patrik Lundquist
Quoted 9 lines · Duncan
Regardless of whether 1 or huge -t means maximum defrag, however, the nominal data chunk size of 1 GiB means that 30 GiB file you mentioned should be considered ideally defragged at 31 extents. This is a departure from ext4, which AFAIK in theory has no extent upper limit, so should be able to do that 30 GiB file in a single extent.
But btrfs or ext4, 31 extents ideal or a single extent ideal, 150 extents still indicates at least some remaining fragmentation.
So I converted the VMware VMDK file to a VirtualBox VDI file:
(quoted diff) +0
-2-rw------- 1 plu plu 28845539328 jul 13 13:36 Windows7-disk1.vmdk -rw------- 1 plu plu 28993126400 jul 13 14:04 Windows7.vdiQuoted 8 lines
$ filefrag Windows7.vdi Windows7.vdi: 15 extents found$ btrfs filesystem defragment -t 3g Windows7.vdi $ filefrag Windows7.vdi Windows7.vdi: 24 extents foundHow can it be less than 28 extents with a chunk size of 1 GiB?
I _think_ the fragment size will be limited by the block group
size. This is not the same as the chunk size for some RAID levels -- for example, RAID-0, a block group can be anything from 2 to n chunks (across the same number of devices), where each chunk is 1 GiB, so potentially you could have arbitrary-sized block groups. The same would apply to RAID-10, -5 and -6.
Quoted 5 lines
Quoted 2 lines
(Note, I haven't verified this, but it makes sense based on what I
know of the internal data structures).It's a raid1 filesystem, so the block group ought to be the same size as the chunk, right?
Yes.
Quoted 1 line
A 2GiB block group would suffice to explain it though.
Not with RAID-1 -- I'd expect the block group size to be 1 GiB.
Hugo.
Signature · 4 lines
Hugo Mills | There isn't a noun that can't be verbed. hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 |
Quoted 38 lines · Hugo Mills
Quoted 37 lines · Patrik Lundquist
Quoted 35 lines · Hugo Mills
Quoted 16 lines · Patrik Lundquist
Quoted 9 lines · Duncan
Regardless of whether 1 or huge -t means maximum defrag, however, the nominal data chunk size of 1 GiB means that 30 GiB file you mentioned should be considered ideally defragged at 31 extents. This is a departure from ext4, which AFAIK in theory has no extent upper limit, so should be able to do that 30 GiB file in a single extent.
But btrfs or ext4, 31 extents ideal or a single extent ideal, 150 extents still indicates at least some remaining fragmentation.
So I converted the VMware VMDK file to a VirtualBox VDI file:
(quoted diff) +0
-2-rw------- 1 plu plu 28845539328 jul 13 13:36 Windows7-disk1.vmdk -rw------- 1 plu plu 28993126400 jul 13 14:04 Windows7.vdiQuoted 8 lines
$ filefrag Windows7.vdi Windows7.vdi: 15 extents found$ btrfs filesystem defragment -t 3g Windows7.vdi $ filefrag Windows7.vdi Windows7.vdi: 24 extents foundHow can it be less than 28 extents with a chunk size of 1 GiB?
I _think_ the fragment size will be limited by the block group
size. This is not the same as the chunk size for some RAID levels -- for example, RAID-0, a block group can be anything from 2 to n chunks (across the same number of devices), where each chunk is 1 GiB, so potentially you could have arbitrary-sized block groups. The same would apply to RAID-10, -5 and -6.(Note, I haven't verified this, but it makes sense based on what I
know of the internal data structures).
Quoted 8 lines
Quoted 2 lines
It's a raid1 filesystem, so the block group ought to be the same size as the chunk, right?
Yes.
Quoted 1 line
A 2GiB block group would suffice to explain it though.
Not with RAID-1 -- I'd expect the block group size to be 1 GiB.
So I had a look at the filefrag source and filefrag actually doesn't print the number of extents but the number of disk fragments. Contiguously allocated extents counts as one fragment.
"Windows7.vdi: 47 extents found" is really 213 extents over 47 disk fragments.
But I have one 2GiB extent, according to filefrag -v, so the question remains. :-)