enquiry about autodefrag option (resent)
I am testing btrfs for long-term storage and backup, and i would like to know more about "autodefrag" option:
1. Will "autodefrag" option benefit ssd?
My understanding is:
autodrag -> number of extent decrease -> metadata decrease -> a
"healthier" filesystem in the long run
(P.S. I am aware that autodefrag will introduce extra write I/O)
2. AFAIK, "autodefrag" detects small random writes into files and queues them up for an automatic defrag process, so the filesystem will defragment itself while it's used.
If the system reboot/crash/remount-ro, will the autodefrag process
continue after resume?
Re: enquiry about autodefrag option
can anybody helps?
Quoted 20 lines · ching
I am testing btrfs for long-term storage and backup, and i would like to know more about "autodefrag" option:
1. Will "autodefrag" option benefit ssd?
My understanding is:
autodrag -> number of extent decrease -> metadata decrease -> a
"healthier" filesystem in the long run(
P.S. I am aware that autodefrag will introduce extra write I/O)2. AFAIK, "autodefrag" detects small random writes into files and queues them up for an automatic defrag process, so the filesystem will defragment itself while it's used.
If the system reboot/crash/remount-ro, will the autodefrag process
continue after resume?
Re: enquiry about autodefrag option
Quoted 15 lines · ching
can anybody helps?
Quoted 12 lines · ching
I am testing btrfs for long-term storage and backup, and i would like to know more about "autodefrag" option:
1. Will "autodefrag" option benefit ssd?
My understanding is:
autodrag -> number of extent decrease -> metadata decrease -> a
"healthier" filesystem in the long run(
P.S. I am aware that autodefrag will introduce extra write I/O)
Yes, your understanding is right, random write workloads will benefit from it.
Quoted 7 lines
Quoted 7 lines
2. AFAIK, "autodefrag" detects small random writes into files and queues them up for an automatic defrag process, so the filesystem will defragment itself while it's used.
If the system reboot/crash/remount-ro, will the autodefrag process
continue after resume?
For reboot, autodefrag will be waited to finish during umounting btrfs.
For crash and remount-ro, it won't resume since it is not that necessary and we're all COWed so that the data is ok.
And autodefrag will only take effect when taking the 'autodefrag' mount option.
thanks,
liubo
Quoted 7 lines
Quoted 1 line
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: enquiry about autodefrag option
Am Mittwoch, 19. September 2012 schrieb Liu Bo:
Quoted 1 line · ching
[…]
Quoted 11 lines
Quoted 11 lines · ching
Quoted 10 lines
I am testing btrfs for long-term storage and backup, and i would like to know more about "autodefrag" option:
1. Will "autodefrag" option benefit ssd?
My understanding is:
autodrag -> number of extent decrease -> metadata decrease ->
a
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
"healthier" filesystem in the long run
(
P.S. I am aware that autodefrag will introduce extra write I/O)Yes, your understanding is right, random write workloads will benefit from it.
What about the extra I/O? And the greatly reduced seek times on SSDs?
Upto now I kept away from defragmenting on SSD.
I wonder about a good way to decide whether autodefrag makes things better or worse for a specific workload. What are the criteria on rotating media and what are they on SSD?
[… informational part about a BTRFS on SSD that should have an age of at least 8 months with almost daily upgrades …]
I only have / running on SSD, but this since quite some time. And it does not seem to have gotten much worse – however this is only subjective feeling of performance.
Except for fstrim times. fstrim take way more time than in the beginning¹. So there seems to be free space fragmentation. Which makes sense for a root filesystem on a Debian Sid machine with lots of upgrade activity and way over 50% usage.
merkaba:~> df -hT /
Dateisystem Typ Größe Benutzt Verf. Verw% Eingehängt auf/dev/dm-0 btrfs 19G 13G 3,6G 79% /
merkaba:~> btrfs fi sh
failed to read /dev/sr0
Label: 'debian' uuid: […]
Total devices 1 FS bytes used 12.25GB
devid 1 size 18.62GB used 18.62GB path /dev/dm-0
merkaba:~> btrfs fi df /
Data: total=15.10GB, used=11.58GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.75GB, used=688.96MB
Metadata: total=8.00MB, used=0.00
I think I get rid of that duplicate metadata once I redo the fs with 8 or 16 KiB metadata blocks.
I thought about rebalancing it, but last time boot time doubled after a complete rebalance. The effect of a rebalance of just the metadata tree to one instead of two might be different tough.
Intel SSD 320 300GB on Kernel 3.6-rc5.
[1] It has been a second or two in the beginning I think. Then it grew over time.
merkaba:~> time fstrim -v /
/: 5877809152 bytes were trimmed
fstrim -v / 0,00s user 5,74s system 14% cpu 39,920 total merkaba:~> time fstrim -v /
/: 5875712000 bytes were trimmed
fstrim -v / 0,00s user 5,55s system 14% cpu 39,095 total merkaba:~> time fstrim -v /
/: 5875712000 bytes were trimmed
fstrim -v / 0,00s user 5,62s system 14% cpu 38,538 total
Signature · 2 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
Re: enquiry about autodefrag option
In reply to Liu BoQuoted 10 lines
Quoted 8 lines
Quoted 8 lines
2. AFAIK, "autodefrag" detects small random writes into files and queues them up for an automatic defrag process, so the filesystem will defragment itself while it's used.
If the system reboot/crash/remount-ro, will the autodefrag process
continue after resume?For reboot, autodefrag will be waited to finish during umounting btrfs.
Quoted 9 lines
For crash and remount-ro, it won't resume since it is not that necessary and we're all COWed so that the data is ok.
And autodefrag will only take effect when taking the 'autodefrag' mount option.
thanks,
liubo
Can the loophole be fixed by tuning the autodefrag algorithm? e.g. also queue up small extents detected during read operation for the autodefrag process.
Is there any io niceness control for autodefrag process too? it will be nice if the idle class is used.
Actually, i try to write shell script to defrag entire FS monthy. but i soon find out it is hard to keep track of progress across reboot and avoid redefragging defragged file
For me, autodefrag is a killer feature. :)
Re: enquiry about autodefrag option
Quoted 15 lines · ching
Quoted 15 lines
Quoted 8 lines
Quoted 8 lines
2. AFAIK, "autodefrag" detects small random writes into files and queues them up for an automatic defrag process, so the filesystem will defragment itself while it's used.
If the system reboot/crash/remount-ro, will the autodefrag process
continue after resume?For reboot, autodefrag will be waited to finish during umounting btrfs.
For crash and remount-ro, it won't resume since it is not that necessary and we're all COWed so that the data is ok.
And autodefrag will only take effect when taking the 'autodefrag' mount option.
Quoted 8 lines
Quoted 4 lines
thanks,
liuboCan the loophole be fixed by tuning the autodefrag algorithm? e.g. also queue up small extents detected during read operation for the autodefrag process.
Sounds interesting, though this would increase write load for a pure read but may improve fragmentation situation and is obviously done only on the first read. The write side of autodefrag could be extended to defrag also small extents that would fit into some threshold, not only newly written data.
Quoted 2 lines
Is there any io niceness control for autodefrag process too? it will be nice if the idle class is used.
No. Autodefrag will mark file data dirty and they'll be written back to the storage in the same way as any other write through the worker threads.
Quoted 3 lines
Actually, i try to write shell script to defrag entire FS monthy. but i soon find out it is hard to keep track of progress across reboot and avoid redefragging defragged file
The defrag process will skip already deframented files (in the meaning of "defragmented" as it sees fit, implementation dependent currently, not exactly that the file has 1 extent).
Quoted 1 line
For me, autodefrag is a killer feature. :)
Yeah, it improves the inherent problem of fragmentation for a COW based filesystem.
david
Re: enquiry about autodefrag option
In reply to Martin SteigerwaldQuoted 5 lines · Martin Steigerwald
Quoted 4 lines
Quoted 1 line
Quoted 1 line
(
P.S. I am aware that autodefrag will introduce extra write I/O)Yes, your understanding is right, random write workloads will benefit from it.
Quoted 7 lines
What about the extra I/O? And the greatly reduced seek times on SSDs?
Upto now I kept away from defragmenting on SSD.
I wonder about a good way to decide whether autodefrag makes things better or worse for a specific workload. What are the criteria on rotating media and what are they on SSD?
Reducing data block fragmentation is accompanied by decreased metadata usage that's needed to maintan the data blocks, and this means less cpu processing and less memory usage.
Blocks in larger contiguous chunks are also more friendly to the SSD garbage collection. A full filesystem defrag & fstrim once in a while may improve performance and device lifetime.
Quoted 11 lines
[??? informational part about a BTRFS on SSD that should have an age of at least 8 months with almost daily upgrades ???]
I only have / running on SSD, but this since quite some time. And it does not seem to have gotten much worse ??? however this is only subjective feeling of performance.
Except for fstrim times. fstrim take way more time than in the beginning¹. So there seems to be free space fragmentation. Which makes sense for a root filesystem on a Debian Sid machine with lots of upgrade activity and way over 50% usage.
I'm roughly counting:
used: 11 + 2*0.688 = 12
touched blocks: 15 + 2* 1.75 = 18.5
of 19, ~90%, this may be problematic even on a SSD regarding the erase block related specifics.
Quoted 30 lines
merkaba:~> df -hT /
Dateisystem Typ Größe Benutzt Verf. Verw% Eingehängt auf/dev/dm-0btrfs 19G 13G 3,6G 79% /merkaba:~> btrfs fi sh
failed to read/dev/sr0
Label: 'debian' uuid: [???]
Total devices 1 FS bytes used 12.25GB
devid 1 size 18.62GB used 18.62GB path/dev/dm-0merkaba:~> btrfs fi df /
Data: total=15.10GB, used=11.58GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.75GB, used=688.96MB
Metadata: total=8.00MB, used=0.00I think I get rid of that duplicate metadata once I redo the fs with 8 or 16 KiB metadata blocks.
I thought about rebalancing it, but last time boot time doubled after a complete rebalance. The effect of a rebalance of just the metadata tree to one instead of two might be different tough.
Intel SSD 320 300GB on Kernel 3.6-rc5.
[1] It has been a second or two in the beginning I think. Then it grew over time.
Quoted 9 lines
merkaba:~> time fstrim -v /
/: 5877809152 bytes were trimmed
fstrim -v / 0,00s user 5,74s system 14% cpu 39,920 total merkaba:~> time fstrim -v /
/: 5875712000 bytes were trimmed
fstrim -v / 0,00s user 5,55s system 14% cpu 39,095 total merkaba:~> time fstrim -v /
/: 5875712000 bytes were trimmed
fstrim -v / 0,00s user 5,62s system 14% cpu 38,538 total
Looks like the issued TRIM commands actually did no work.
david
Re: enquiry about autodefrag option
In reply to David SterbaQuoted 8 lines
Quoted 3 lines
Is there any io niceness control for autodefrag process too? it will be nice if the idle class is used.
No. Autodefrag will mark file data dirty and they'll be written back to the storage in the same way as any other write through the worker threads.
AFAIK, the autodefrag will read related extents and mark them dirty, io niceness should be applicable to the read operation
Re: enquiry about autodefrag option
Quoted 2 lines · ching
AFAIK, the autodefrag will read related extents and mark them dirty, io niceness should be applicable to the read operation
I was hesitant to believe that, but tried it and it's right:
$ dd if=/dev/zero bs=4k count=1 seek=15 of=file $ dd if=/dev/zero bs=4k count=1 seek=13 of=file conv=notrunc $ dd if=/dev/zero bs=4k count=1 seek=14 of=file conv=notrunc
Filesystem type is: 9123683e
File size of file is 65536 (16 blocks, blocksize 4096)
ext logical physical expected length flags
0 13 1027 1 1 14 1050 1027 1 2 15 1025 1050 1 eof
file: 4 extents found
remount with autodefrag:
$ dd if=/dev/zero bs=4k count=1 seek=16 of=file conv=notrunc
Filesystem type is: 9123683e
File size of file is 69632 (17 blocks, blocksize 4096)
ext logical physical expected length flags
0 13 1033 4 eof
file: 2 extents found
david